New automated nest monitoring method using video footage

Fitz HRA Rita Covas and colleagues have published an exciting new method for analysing thousands of hours of video footage from long-term nest-monitoring projects to classify the birds' behaviours, saving researchers many hours of hard work and plenty of expensive data storage space as well. 

Here is an excerpt from a blog post by the first author, Liliana Silva, about the new tool:

Watching animal behaviour is one of the most widely used methods in ecology. But anyone who has spent hours viewing video footage knows how quickly behavioural analysis becomes overwhelming. A single nest camera can generate hundreds of hours of recordings, and turning those videos into behavioural data often means endless manual annotation. As a former video technician myself, I have spent more than 2000 hours analysing footage and I, together with André Ferreira, often wondered whether there could be an easier way. 

This challenge started us on a 5-year path to develop an automated framework for recognising nest behaviours from video recordings of the sociable weaver. Once we had a successful workflow, we brought together collaborators from other long-term projects on blue tits (Arlette Fauteux, France) and great tits (Irene Martínez-Baquero, United Kingdom) to make sure our results were useful for other systems. This resulted in our recent paper 'From video to behaviour: An LSTM-based approach for automated nest behaviour recognition in birds', where we describe a practical workflow that helps researchers move from video footage to behavioural data. 

For our long-term monitoring projects of species in the wild, we needed robust automated systems that could cope with environmental variation over time. Our goal was not simply to build a proof-of-concept modelling approach, but to create a deployment-focused framework that researchers could adapt to their own study systems and real-world field data.

The framework combines stored footage, previously annotated behaviours and deep learning using ‘long short-term memory’ models to recognise behavioural sequences from video data. The system learns patterns of behaviour over time rather than relying on single images alone. This is especially important for behaviours that unfold over time, as the model can better distinguish similar-looking behaviours, such as differentiating a bird entering a specific nest from a bird simply passing by.

Using this approach, we built three automated systems: one for each species. These models provide second-by-second behavioural classifications, including nest entering, leaving, building, aggression and sanitation, depending on the species. Importantly, model performance was comparable to human observer performance, although much faster.

Automated behavioural recognition has enormous potential for ecology and evolution. Tools like this can help researchers process much larger datasets than would be possible manually, opening the door to large-scale behavioural studies. For example, the Sociable Weaver project collects more than 2,000 hours of video per year. Our models increased analysis speed eightfold, from 40 to over 300 videos per week. Additionally, by keeping only footage containing behaviours of interest, the system reduced storage requirements by over 90%. 

Importantly, by sharing a practical open-source workflow, we hope more researchers will feel able to explore these approaches. One of the key strengths of the workflow is its flexibility. Researchers can train the system using their own annotated videos, making it easily adaptable to different species and ecological contexts. We also focused heavily on practical implementation by walking through data preparation, annotation strategies, model training and deployment considerations, and providing a roadmap that other researchers can realistically follow.

Paper link: https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.70325

Websites: https://sociableweaverproject.com , https://mesangecefe1.wixsite.com/mesangecefe, https://www.wythamwoods.ox.ac.uk/wytham-tit-project