Analyzing Bird Audio Cover Photo

Authors: Adithya Balaji, Malika Khurana

This report details our process for analyzing bird audio, with some snippets of code. You can find the full project repo on github.

import IPython.display as ipd
import requests

Introduction

We aim to accurately classify bird sounds as songs or calls. We used 3 different approaches and models based on recording metadata, the audio data itself, and spectrogram images of the recording to perform this classification task.

Pipeline Overview

Motivation

The primary motivation to address this problem is to make it easier for scientists to collect data on bird populations and verify community-sourced labels.

The other motivation is more open-ended: to understand the "hidden" insights in bird sounds. Bird calls reveal regional dialects, a sense of humor, information about predators in the area, indicators of ecosystem health—and inevitably also the threat on their ecosystems posed by human activity. Through the process of exploring bird call audio data, we hope we can build towards better understanding the impacts of the sounds produced by humans and become better listeners.

Songs vs Calls

Bird sounds have a variety of different dimensions, but one of the first levels of categorizing bird sounds is classifying them as a song or a call, as each have distinct functions and reveal different aspects of the birds’ ecology (1, 2).

Songs

Songs tend to be longer, more melodic, and used for marking territory and attracting mates. Birds' song repertoire and song rate can indicate their health and the quality of their habitat, including pollutant levels and plant diversity (3, 4, 5).

ipd.Audio(
    requests.get(
        "https://github.com/adithyabsk/bird_audio/blob/main/notebooks/assets/574080.mp3?raw=true"
    ).content
)

Calls

Calls are shorter than songs, and perform a wider range of functions like signalling food, maintaining social cohesion and contact, coordinating flight, resolving conflicts, and sounding alarms (distress, mobbing, hawk alarms) (6). Bird alarm calls can be understood and passed along across species, and have been found to encode information about the size and threat of a potential predator, so birds can respond accordingly - i.e. more intense mobbing for a higher threat (7, 8). Alarm calls can also give scientists an estimate of the number of predators in an area.

ipd.Audio(requests.get("https://github.com/adithyabsk/bird_audio/blob/main/notebooks/assets/585148.mp3?raw=true".content)

Allometry of Alarm Calls: Black-Capped Chickadees Encode Information About Predator Size (8) The number of D-notes in chickadee alarm mobbing calls varies indirectly with the size of predator.

Gender identification using acoustic analysis in birds without external sexual dimorphism (9) Bird sounds were analyzed to classify gender. Important acoustic features were: fundamental frequency (mean, max, count), note duration, syllable count and spacing, and amplitude modulation.

Regional dialects have been discovered among many bird species, and the Yellowhammer is a great example (10, 11) Yellowhammer bird sounds in the Czech Republic and UK were studied to identify regional dialects, which differed in frequency and length of final syllables.

DVC

Data Version Control (DVC) is a useful tool for data science projects. You can think of it like git but for data. We built out our pipeline first in jupyter notebooks, and then in DVC, making it easy to change parameters and run the full pipeline from one place.

Note: Due to the size of the datasets, we chose not to include inline Jupyter snippets of code processing real data and instead opted to present only the outputs of the DVC scripts. (Python files, not notebooks)

Collecting Data

For our analysis, we used audio files and metadata from [xeno-canto.org]. Xeno-canto (XC) is a website for collecting and sharing audio recordings of birds. Recordings and identifications on XC are sourced from the community (anyone can join).

Xeno Canto API Page

XC has a straightforward API that allows us to make RESTful queries, and specify a number of filter parameters including country, species, recording quality, and duration. We used the XC API to get metadata and IDs for all recordings in the United States, and saved the JSON payload as a dataframe and csv. Below we see the main snippet of code from the DVC step that parallelizes data collection from XC.

def search_recordings(**query_params) -> List[JSON]:
    """Search for recordings using the Xeno Canto search API

    The keys in the return dictionaries are specified in the API docs

    https://www.xeno-canto.org/explore/api


    Args:
        **query_params: A dictionary with query and/or page as keys with values as specified in the
            API docs.

    Returns:
        A list of recording information (list of dictionaries)

    """
    url = (BASE_URL / "recordings").with_query(query_params)
    resp = requests.get(str(url))
    if resp.status_code == 200:
        resp_json = resp.json()
        num_pages = resp_json["numPages"]
        recordings = resp_json["recordings"]
        if num_pages > 1:
            page_urls = [url.update_query(page=p) for p in range(2, num_pages + 1)]
            with ProcessPoolExecutor(max_workers=10) as ppe:
                recordings.extend(
                    itertools.chain(
                        *tqdm(
                            ppe.map(get_page_recordings, page_urls), total=num_pages - 1
                        )
                    )
                )
    else:
        raise Exception(f"Request failed with status code: {resp.status_code}")

    return recordings

Filtering & Labeling

Through our DVC pipeline, we further filtered by the top 220 unique species, recordings under 20 seconds, recording quality A or B, and recordings with spectrograms available on XC. This reduced our dataset size from ~60,000 to get a dataframe of 5,800 recordings. We created labels (1 for call, 0 for song) by parsing the 'type' column of the df.

The following scripts handle that process:

  1. build_filter.py
  2. build_song_vs_call.py
  3. proc_svc_meta.py

Exploring & Visualizing Data

With our dataset assembled, we began exploring it visually. A distribution of recordings by genus, with song-call splits shows that the genus most represented in the dataset are warblers (Setophaga) with many more songs than call recordings. We can also see that, as expected, woodpeckers (Melanerpes), jays, magpies, and crows (Cyanocitta, Corvus) have almost no song recordings in the dataset.

Count vs Genus for the Top 20 Largest Genus

A map of recording density shows the regions most represented in the dataset which are, unsurprisingly, bird watching hot spots.

Observation Count KDE Plot

Given our domain knowledge that songs serve an important function in mating, we expected to see a higher proportion of songs in the spring, which is confirmed by the data.

Song and Call Percent vs Month

Metadata Classification Model

In our first model, we used the tabular metadata from XC entries to train a Gradient Boosted Decision Tree (GBDT) model using XGBoost. XGBoost, is a particular Python implementation of GBDTs that is designed to work on large amounts of data.

We used the genus, species, English name, and location (latitude and longitude) from XC metadata. These features were then all mapped and imputed using sklearn transformers to one-hot encoded form apart from latitude, longitude, and time (all mapped using standard or min-max scaling, and time features transformed with a sin function). We can see 10 rows of unprocessed data in the HTML table below.

df_indexidgenspsspenreccntloclatlngalttypeurlfilefile-namesonolicqlengthtimedateuploadedalsormkbird-seenplayback-usedpredgenderagemonthdayhourminute
096454911BrantacanadensisNaNCanada GooseBruce LagerquistUnited StatesSedro-Woolley, Skagit County, Washington48.5237-122.018530call//www.xeno-canto.org/454911//www.xeno-canto.org/454911/downloadXC454911-190202_02 Canadian Geese.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/JHFICMRVUX/ffts/XC454911-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/JHFICMRVUX/ffts/XC454911-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/JHFICMRVUX/ffts/XC454911-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/JHFICMRVUX/ffts/XC454911-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/A0:171900-01-01 11:30:002019-02-022019-02-04['Cygnus buccinator']Mixed flock of Trumpeter Swans and Canada Geese feeding in an agricultural field. Recording of Swan's here XC454910yesno1NaNNaN2.02.011.030.0
197418340BrantacanadensisNaNCanada GooseSue RiffeUnited StatesAu Sable SF - Big Creek Rd, Michigan44.0185-83.7560180song//www.xeno-canto.org/418340//www.xeno-canto.org/418340/downloadXC418340-Canada Goose on 5.11.18 at Au Sable SF MI at 11.20 for .14 _0908 .mp3{'small': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC418340-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC418340-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC418340-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC418340-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/A0:141900-01-01 11:20:002018-05-112018-06-03['Agelaius phoeniceus']Natural vocalizationyesno0NaNNaN5.011.011.020.0
2107291051BrantacanadensisNaNCanada GooseEric HoughUnited StatesSan Juan River, Cottonwood Day-Use Area, Navajo Lake State Park, San Juan County, New Mexico36.8068-107.67891800call//www.xeno-canto.org/291051//www.xeno-canto.org/291051/downloadXC291051-CANG_11515_1730_SanJuanRiver-NavajoDam.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/BCFUZDOSJZ/ffts/XC291051-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/BCFUZDOSJZ/ffts/XC291051-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/BCFUZDOSJZ/ffts/XC291051-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/BCFUZDOSJZ/ffts/XC291051-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/A0:151900-01-01 17:30:002015-11-152015-11-18['']Flock calling while flying over at dusk. Amplification, low and high pass filters used in Audacity.yesno1NaNNaN11.015.017.030.0
3108283618BrantacanadensisNaNCanada GooseGarrett MacDonaldUnited StatesBeluga--North Bog, Kenai Peninsula Borough, Alaska61.2089-151.010340call, flight call//www.xeno-canto.org/283618//www.xeno-canto.org/283618/downloadXC283618-LS100466.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/CDHIAMGTRT/ffts/XC283618-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/CDHIAMGTRT/ffts/XC283618-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/CDHIAMGTRT/ffts/XC283618-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/CDHIAMGTRT/ffts/XC283618-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/A0:101900-01-01 11:00:002015-05-202015-10-03['']Natural vocalizations from a pair of birds in flight. Recording not modified.yesno1NaNNaN5.020.011.00.0
4110209702BrantacanadensisNaNCanada GooseAlbert @ Max lastukhinUnited StatesOyster Bay (near Lattingtown), Nassau, New York40.8881-73.585110call//www.xeno-canto.org/209702//www.xeno-canto.org/209702/downloadXC209702-Poecile atricapillus Dec_27,_2014,_4_05_PM,C1.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/LELYWQKUZX/ffts/XC209702-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/LELYWQKUZX/ffts/XC209702-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/LELYWQKUZX/ffts/XC209702-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/LELYWQKUZX/ffts/XC209702-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/A0:111900-01-01 16:00:002014-12-272015-01-09['Poecile atricapillus']NaNyesno1NaNNaN12.027.016.00.0
5118165398BrantacanadensisparvipesCanada GooseTed FloydUnited StatesBoulder, Colorado40.0160-105.27651600call//www.xeno-canto.org/165398//www.xeno-canto.org/165398/downloadXC165398-CanG for Xeno-Canto.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/KADPGEQPZI/ffts/XC165398-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/KADPGEQPZI/ffts/XC165398-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/KADPGEQPZI/ffts/XC165398-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/KADPGEQPZI/ffts/XC165398-full.png'}//creativecommons.org/licenses/by-nc-sa/3.0/A0:191900-01-01 09:30:002014-01-242014-01-25['']A large flock of Canada Geese taking off. I believe most of the birds in this flock were parvipes ("Lesser") Canada Geese, but there were also larger (subspecies moffitti?) Canada Geese and a few Cackling Geese (several of the subspecies hutchinsii and possibly one of the subspecies minima) in the general vicinity. In the old days this would have been an "obvious" or "easy" flock of "Canada Geese." Now we're dealing with perhaps two species and probably two or three subspecies in the recording. Again, I believe most of the birds audible here are parvipes ("Lesser") Canada Geese.yesno1NaNNaN1.024.09.030.0
61291136BrantacanadensisNaNCanada GooseDon JonesUnited StatesBrace Road, Southampton, NJ39.9337-74.7170?song//www.xeno-canto.org/1136//www.xeno-canto.org/1136/downloadbird034.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/BCWZQTGMSO/ffts/XC1136-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/BCWZQTGMSO/ffts/XC1136-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/BCWZQTGMSO/ffts/XC1136-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/BCWZQTGMSO/ffts/XC1136-full.png'}//creativecommons.org/licenses/by-nc-nd/2.5/A0:10NaT1997-10-172008-11-20['']NaNunknownunknown0NaNNaN10.017.0NaNNaN
7132536877BrantacanadensisNaNCanada GooseSue RiffeUnited StatesS Cape May Meadows, Cape May Cty, New Jersey38.9381-74.94460adult, call, sex uncertain//www.xeno-canto.org/536877//www.xeno-canto.org/536877/downloadXC536877-Canada Goose on 10.18.19 at S Cape May Meadows NJ at 18.52 for .19.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC536877-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC536877-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC536877-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/PVQOLRXXWL/ffts/XC536877-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/B0:191900-01-01 18:52:002019-10-182020-03-21['Charadrius vociferus']Natural vocalization of a flock of geese landing on the water near sunset. Windyyesno1NaNadult10.018.018.052.0
8133511453BrantacanadensisNaNCanada GoosePhoenix BirderUnited StatesGilbert, Maricopa County, Arizona33.3634-111.7341380adult, call, female, male//www.xeno-canto.org/511453//www.xeno-canto.org/511453/downloadXC511453-CaGo.2019.12.10.AZ.Maricopa.RiparianPreserve.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/UKNISVRBBF/ffts/XC511453-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/UKNISVRBBF/ffts/XC511453-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/UKNISVRBBF/ffts/XC511453-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/UKNISVRBBF/ffts/XC511453-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/B0:191900-01-01 08:52:002019-12-102019-12-10['Toxostoma curvirostre']Sound Devices MixPre-3 Wildtronics Stereo Model #WTPMMSA 22” Parabolic Reflector, phoenixbirder@gmail.comyesno1maleadult12.010.08.052.0
9134504983BrantacanadensiscanadensisCanada Goosenick talbotUnited StatesCentral Park, New York city,USA40.7740-73.971020call//www.xeno-canto.org/504983//www.xeno-canto.org/504983/downloadXC504983-2019_10_21 Branta canadensis2.mp3{'small': '//www.xeno-canto.org/sounds/uploaded/CCUCXWCPSW/ffts/XC504983-small.png', 'med': '//www.xeno-canto.org/sounds/uploaded/CCUCXWCPSW/ffts/XC504983-med.png', 'large': '//www.xeno-canto.org/sounds/uploaded/CCUCXWCPSW/ffts/XC504983-large.png', 'full': '//www.xeno-canto.org/sounds/uploaded/CCUCXWCPSW/ffts/XC504983-full.png'}//creativecommons.org/licenses/by-nc-sa/4.0/B0:131900-01-01 13:00:002019-10-212019-10-30['']A pair of birds calling from a lakeyesno1NaNNaN10.021.013.00.0

Here we also see a snippet of the data transformation pipeline and model training code which was done in the following jupyter notebook.

feature_mapper = DataFrameMapper(
    [
        ("id", None),
        (["gen"], OneHotEncoder(drop_invariant=True, use_cat_names=True)),
        (["sp"], OneHotEncoder(drop_invariant=True, use_cat_names=True)),
        (["en"], OneHotEncoder(drop_invariant=True, use_cat_names=True)),
        (["lat"], [SimpleImputer(), StandardScaler()]),  # gaussian
        (["lng"], [SimpleImputer(), MinMaxScaler()]),  # bi-modal --> MinMaxScaler
        # TODO: maybe later look into converting month / day into days since start of year
        (
            ["month"],
            [
                SimpleImputer(),
                FunctionTransformer(lambda X: np.sin((X - 1) * 2 * np.pi / 12)),
                StandardScaler(),  # gaussian
            ],
        ),
        (
            ["day"],
            [
                SimpleImputer(),
                FunctionTransformer(lambda X: np.sin(X * 2 * np.pi / 31)),
                MinMaxScaler(),  # uniform
            ],
        ),
        # TODO: maybe later look into converting hour / minute into seconds since start of day
        (
            ["hour"],
            [
                SimpleImputer(),
                FunctionTransformer(lambda X: np.sin(X * 2 * np.pi / 24)),
                StandardScaler(),  # gaussian
            ],
        ),
        (
            ["minute"],
            [
                SimpleImputer(),
                FunctionTransformer(lambda X: np.sin(X * 2 * np.pi / 60)),
                MinMaxScaler(),  # uniform
            ],
        ),
    ],
    df_out=True,
)

X_feat_df = feature_mapper.fit_transform(X_df, y_df["pred"])
X_train, X_test = (
    X_feat_df[X_feat_df.id.isin(train_ids)].drop(columns=["id"]),
    X_feat_df[X_feat_df.id.isin(test_ids)].drop(columns=["id"]),
)
y_train, y_test = (
    y_df[y_df.id.isin(train_ids)].drop(columns=["id"]).squeeze(),
    y_df[y_df.id.isin(test_ids)].drop(columns=["id"]).squeeze(),
)

xgb_clf = xgb.XGBClassifier()
eval_set = [(X_train, y_train), (X_test, y_test)]
xgb_clf.fit(
    X_train, y_train, eval_metric=["error", "logloss"], eval_set=eval_set, verbose=False
)

print(xgb_clf.score(X_test, y_test))

Audio Classification Model

In one model we used the bird audio recordings themselves (mp3 and wav files), converted into time series arrays using librosa and processed with tsfresh to extract features, which we used to train a Gradient Boosted Tree model.

Building Audio Features

We ran audio data through a high-pass Butterworth filter to take out background noise. We tested different parameters for Butterworth and Firwin filters, then examined resulting spectrograms and audio to determine which best reduced background noise without clipping bird sound frequencies.

Filter Comparisons

The below code snippet shows the process of loading the .mp3 file and performing the above filtering steps before saving as a pd.DataFrame which is what ts-fresh expects.

def unpack_audio(recordings_path: Path, id, filter_order, cutoff_freq):
    """Load mp3 or wav into a floating point time series, then run a high-pass filter.

    Args:
        recordings_path: The path to the recordings dir
        id: The id for the audio recording to unpack
        filter_order: The order for the Butter filter
        cutoff_freq: The critical frequency for the Butter filter (below this is filtered out)

    Returns:
        A df of filtered time series data for the given id, with 'id', 'time', and 'val' columns
    """
    try:
        audio_path = FILE_PATH / ("data/raw/recordings/" + str(id) + ".mp3")
        # load mp3 as audio timeseries arr
        timeseries, sr = librosa.load(audio_path)
    except FileNotFoundError:
        audio_path = FILE_PATH / ("data/raw/recordings/" + str(id) + ".wav")
        timeseries, sr = librosa.load(audio_path)

    # high-pass filter on audio timeseries
    timeseries_filt = highpass_filter(timeseries, sr, filter_order, cutoff_freq)

    df = pd.DataFrame(timeseries_filt, columns=["val"])
    df.reset_index(inplace=True)
    df["id"] = id  # fill col with id
    df = df.reindex(columns=["id", "index", "val"])
    df.columns = ["id", "time", "val"]
    return df

Feature Selection & Extraction

We used ts-fresh to featurize each audio array after unpacking and filtering to avoid running out of memory. ts-fresh takes in dataframes with an id column, time column, and value column.

Time Series Input DF for a Single ID

ts-fresh provides feature calculator presets, but due to their and librosa.load's long runtimes (13+ hours for 5% of the dataset), we manually specified the following small set of features based on our domain understanding of bird audio analysis.

Lastly, we passed this "static" time series feature dataframe into a similar XGBoost model (from above) to predict the output class.

manual_fc_params = {
    "abs_energy": None,
    "fft_aggregated": [{"aggtype": "centroid"}, {"aggtype": "kurtosis"}],
    "root_mean_square": None,
    "spkt_welch_density": [{"coeff": 2}, {"coeff": 5}, {"coeff": 8}],
}

# select features to calculate
# features can be found here: https://tsfresh.readthedocs.io/en/latest/api/tsfresh.feature_extraction.html#tsfresh.feature_extraction.feature_calculators.fft_aggregated
def featurize_audio(id, fc_params):
    return extract_features(
        unpack_audio(id),
        column_id="id",
        column_sort="time",
        default_fc_parameters=fc_params,
        disable_progressbar=True,
        # we impute = remove all NaN features automatically
        impute_function=impute,
        # turn off parallelization
        n_jobs=0,
    )


# featurize dataset
# returns df of all combined
def featurize_set(ids, fc_params=None):
    if fc_params is None:
        fc_params = EfficientFCParameters()
    X_df = pd.DataFrame()
    for id in tqdm(ids):
        X_df = pd.concat([X_df, featurize_audio(id, fc_params)])
    return X_df
Feat Output for all IDs

Spectrogram Classification Model

Training Notebook Link

We used a computer vision approach to analyze spectrograms using fast.ai pre-trained model. We use an xresnet18 architecture pre-trained on ImageNet.

We load the data using fast.ai's ImageDataLoader. The model is then cut at the pooling layer (frozen weights) and then trained on its last layers to utilize transfer learning on our spectrogram images. A diagram of the architecture pulled directly from the original resnet paper is included below.

ResNet50 Architecture

The model itself was trained on a Tesla K80 using Google Colab to speed up the training process. Additionally, we used Weights and Biases to track the training and improve the model tuning. We've listed the main snippets of code below that handle the training process.

bs = 128  # Batch size
kwargs = {}
if IS_COLAB:
    kwargs["num_workers"] = 0
data = (
    # convert_mode is passed on intern|ally to the relevant function that will handle converting the images;
    # 'L' results in one color channel
    ImageDataLoaders.from_df(
        image_df,
        folder=ROOT_PATH / "data/raw/sonograms",
        valid_col="is_valid",
        bs=bs,
        # num_works needs to be set to 0 for local evaluation to turn off multiprocessing
        **kwargs,
    )
)
learn = cnn_learner(data, xresnet.xresnet18, pretrained=True)

# Make sure this path exists on colab
fname = "sono_model.pth"
model_path = (ROOT_PATH / f"models/{fname}").resolve().absolute()
if IS_COLAB and TRAIN:
    # Fine tune model
    wandb.init(project="sono-model")
    learn.fit_one_cycle(1, cbs=WandbCallback())
    # GDrive fails when you try to use mkdir
    # so we manually call `save_model`
    save_path = f"/home/{fname}"
    save_model(save_path, learn.model, getattr(learn, "opt", None))
    %ls -al /home
    from google.colab import files

    files.download(save_path)
else:
    load_model(model_path, learn.model, learn.opt)

Results

Across our three models, we achieved scores in a range of 64-77%. This is above the baseline score of 55% (mean of labels), and we believe with more time to tune and ensemble the models, one could achieve an even more accurate classifier. We are encouraged by the amount of room both the time series based and sonogram based models have for improvement given that the metadata model wipes the floor in terms of accuracy.

| Model | Train Log Loss | Test Log Loss | Train Accuracy | Test Accuracy | |-|-|-|-|-| | Metadata Model | 0.331 | 0.507 | 0.879 | 0.773 | | Audio Model | 0.255 | 0.694 | 0.957 | 0.639 | | Spectrogram Model | 0.661 | 0.675 | 0.682 | 0.682 | | Baseline | 0.69 | 0.55 | 0.55 | 0.54 |

Plots

Metadata Model

We note a plateau in the XGBoost validation accuracy which tends to suggest that further improvements in early stopping may be achieved.

XGBoost Log Loss

Additionally, due to the nature of the decision tree based model we are able to compute feature importance. The most important features include the genera - this is not so surprising when we recall our genus-count distribution and see that the genera here are mostly those with recordings that are almost entirely songs or calls. The other important feature is month - again, we recall that in the spring the ratio of songs to calls goes up, so time of year is a "good" feature.

XGBoost Feature Importance

Time Series Model

We can see that the test loss increases due to over-fitting, also evidenced by the very high training accuracy. This is a potential area of improvement in further research.

LogReg Log Loss

Spectrogram Model

This is the direct output from WandB which depicts the training process for the fine-tuned xresnet model. It is important to note that the X axis is steps and not epochs as this model was only trained for a single epoch (to save time and memory).

Wandb Train Ouput

Future Work

We would like to note that there are a couple of immediate next steps that the project could take to dramatically improve the model performance

  • Ensembling the 3 models using a VotingClassifier
  • More training time for the Spectrogram model (only 30 minutes was provided for fine-tuning)
    • Additional epochs (only 1 epoch was provided)
  • Filtering features in the audio classification model (ts-fresh likely generates more features than are needed)

Long term: integrate model with Xeno Canto to provide tag suggestions based on the audio clip

Conclusion

The classification of song vs call is the first distinction one can make in bird audio data across species, and on its own can give insights into the number of predators in an ecosystem, the timing of mating season, and other behaviors. It could also be valuable when part of a larger system of models. This report presents a promising start to tackle this problem with three separate machine learning models with reasonable accuracy. These models will likely prove quite handy in downstream classification tasks that look to find species, gender, location, and other parameters from the bird audio sample.

References

  1. "A Beginner’s Guide to Common Bird Sounds and What They Mean." Audubon.org.
  2. "Two Types of Communication Between Birds: Understanding Bird Language Songs And Calls." Youtube.
  3. "Bird Vocalization." Wikipedia.
  4. Gorissen, Leen, et al. “Heavy Metal Pollution Affects Dawn Singing Behaviour in a Small Passerine Bird.” Oecologia, vol. 145, no. 3, 2005, pp. 504–509. JSTOR
  5. Ortega, Yvette K.; Benson, Aubree; Greene, Erick. 2014. Invasive plant erodes local song diversity in a migratory passerine. Ecology. 95(2): 458-465. Ecological Society of America
  6. Marler, P. (2004), Bird Calls: Their Potential for Behavioral Neurobiology. Annals of the New York Academy of Sciences, 1016: 31-44. https://doi.org/10.1196/annals.1298.034
  7. "These birds 'retweet' alarm calls—but are careful about spreading rumors." National Geographic.
  8. Templeton, Christopher N., et al. “Allometry of Alarm Calls: Black-Capped Chickadees Encode Information About Predator Size.” Science, vol. 308, no. 5730, American Association for the Advancement of Science, 2005, pp. 1934–37, doi:10.1126/science.1108841.
  9. Volodin, I.A., Volodina, E.V., Klenova, A.V. et al. Gender identification using acoustic analysis in birds without external sexual dimorphism. Avian Res 6, 20 (2015). https://doi.org/10.1186/s40657-015-0033-y
  10. "About yellowhammers." Yellowhammer Dialects.
  11. Harry R Harding, Timothy A C Gordon, Emma Eastcott, Stephen D Simpson, Andrew N Radford, Causes and consequences of intraspecific variation in animal responses to anthropogenic noise, Behavioral Ecology, Volume 30, Issue 6, November/December 2019, Pages 1501–1511, https://doi.org/10.1093/beheco/arz114
  12. "Open-source Version Control System for Machine Learning Projects." DVC.
  13. xeno-canto.
  14. scikit-learn.
  15. xgboost.
  16. fast.ai.

Metrics

Word Count

1753 words

Code Line count

We used CLOC to generate the code line counts

Language Files Code
Jupyter Notebook 9 1195
Python 8 397
Sum 17 1592