BRC-BIO: Optimizing Snake Identification by Understanding the Interplay of Computer Vision, Crowdsourcing, and Expert Verification

Species identification is fundamental to biology, and in the case of snakes not only helps understanding their diversity but could also be critical in the case of snake bites. Recently, computer systems that can help people with classification tasks have become a lot more common. Computer vision algorithms are computer programs that have been trained to perform specific classification tasks, such as Shazam, Google Lens, iNaturalist’s Seek, and eBird’s Merlin app. This research aims to understand the influence that conflicting information can have on the accuracy of classification tasks. This is important because it could lead us to an optimal pipeline that allows humans and computers to work together to perform these tasks while making the minimal number of mistakes. This project involves the identification of snake species from photos, a task that is difficult for both humans and computers because there are over 4,000 different snake species and some of them look extremely similar. In the project, people will be shown images of snakes and asked to identify them to species. Sometimes, they will also receive information about what other people or computer vision algorithms called these snakes. The researchers will measure any changes in identification accuracy that are caused by the additional information. The researchers will also gather images of rare species of snakes from preserved specimens in natural history museums to determine whether these types of images can help train computer vision algorithms to better identify these snakes in the wild. This project will provide training opportunities for undergraduate and graduate students, as well as invite participation from the public in identification and data collection. This project will explore the intersection of Convolutional Neural Network (CNN) and human species identification. It will also study effects of accuracy of CNNs from different training sets, using field and collection data. Using a gamified platform, researchers will pre-test the snake ID skills of participants, then randomly assign participants snake ID tasks with or without prior labels. Some prior labels will be correct and others will be incorrect. Some prior labels will appear to come from other participants and others from computer vision algorithms. Researchers will measure how frequently players give an incorrect ID that matches competing or confirming prior information. Whether and how susceptible players are to join the “bandwagon” of incorrect IDs from algorithms or other humans will help determine how and when to add a “human-in-the-loop” while maintaining or increasing identification accuracy. In the second project, the performance of computer vision algorithms trained on images of museum specimens, wild snakes, or both will be compared in order to better understand the potential of using photos of museum specimens to nourish training datasets for computer vision algorithms. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.