Real-time Computer Vision predictions in Seek by iNaturalist version 2.0

On April 5, in conjunction with the release of the Our Planet series by Netflix, iNaturalist released a new version of Seek by iNaturalist. To heed David Attenborough’s call to action in Our Planet to protect biodiversity, we need to understand what’s here and what we might lose. Seek by iNaturalist unlocks curiosity about the natural world by giving speedy identifications with computer vision and challenging you to earn badges for finding species new-to-you.

How it works

When you open the Seek camera and point it at a living thing, the app immediately tells you what you’re looking at, even before you take a picture. This on-screen identification is tied to the tree of life, and guides you towards taking a more identifiable photo by getting more specific as you fill the frame and get the right angle or features. When the app narrows it down to species, it prompts you to take a picture, which earns you a badge and unlocks more information about the species. This “augmented reality” view of the world makes it easy to explore and interpret the natural world around you all while guiding you to take more identifiable photos.  Seek can’t always identify things to species (it’s still learning…), but it aims to provide the most precise correct name it can. Here’s some footage of it in action!

In Seek v1, which launched in March 2018 with support from HHMI Tangled Bank Studios, you needed to take a photo before you had a sense of whether or not it was even identifiable by Seek. This caused frustration when people repeatedly experienced the lack of a computer vision match. Now, since you can see predictions overlayed on the camera you get immediate feedback about what you see and the specificity with which Seek can identify it.

The species included in Seek are based entirely on photos and identifications made by the global iNaturalist community, so the Seek camera will work best in places where there is already an active community of iNaturalist users, and for species that are easily identified from photos. Seek also uses data submitted to iNaturalist to show suggestions for “species nearby,” but unlike iNaturalist, findings made with Seek will not be shared publicly, making it safe for children to use.

Seek is geared to encourage outdoor exploration of wild biodiversity (rather than pets, zoo animals, or garden plants). We hope kids, families, educators, and anyone into games will start exploring their natural surroundings with Seek, and we want this to inspire the next generation of biodiversity stewards by encouraging exploration and unlocking the names of species as a way to learn more about them. We want to make it easier for curious people who may not consider themselves naturalists to learn more about nature.

Innovations in computer vision came from research collaborations with Grant Van Horn, an Adjunct Scientist with iNaturalist. Van Horn’s dissertation research at CalTech, advised by Pietro Perona, made it possible to produce a refined dataset of iNaturalist observations, train the classification model, and export it for efficient inference on mobile devices. The computer vision model includes 15,798 species and 12,524 broader taxonomic groups (such as kingdom, phylum, class, order, family, and genus). The accuracy of predictions and breadth of species included will continue to improve as the iNaturalist community and dataset grows.

How to download it

Seek is freely available on both iOS and Android. Unlike the iNaturalist mobile apps which have separate code bases for iOS and Android, we used React Native for Seek, which is a relatively new technology that allowed us to build Seek for both Android and iOS with a single codebase. It’s still nascent — there’s not even a stable 1.0.0. release yet — but we liked the benefits:

  • Saves us time in development, since we’re only using one coding language.
  • Brings Seek to a wider, more varied audience, since we’re able to support 7,000+ devices on Android alone — this also allows for a more international audience, since Android tends to be more popular anywhere outside of North America, Australia, and western parts of Europe.

Seek is currently translated into 7 languages: English, Spanish, Hindi, Chinese, Portuguese, German, and French.

This major update to Seek was made possible with support from WWF and the Our Planet series on Netflix. Seek is created by iNaturalist, which is a joint initiative of the California Academy of Sciences and the National Geographic Society.