How much reliance can be placed on the identification of a species using an online image? In a newly published paper, researchers document the results of a study looking at accuracy (and agreement) of newt identification between experts…
As technology and equipment becomes cheaper and more accessible to a wider audience, it is increasingly being used as part of surveys and research to record parameters such as species presence, number, distribution and behaviour. Products such as camera traps, smartphones and drones are being used more and more frequently by professionals and wildlife enthusiasts alike. These types of equipment produce a plethora of images and associated metadata – but how accurately can information regarding the identification of ‘less obvious’ species be extrapolated from this data?
A recent study undertaken by Austen et al. (2018), looked at the accuracy of newt identification by holders of great crested newt (GCN) (Triturus cristatus) licenses, when looking at online images. The study notes that:
“…these images permit identification and validation to take place at a later date, and can be shared relatively easily…Whether as part of a citizen science project, through a local recording group, or simply ad-hoc observations, the process involves images being uploaded electronically for identification or verification by enthusiasts of varying expertise”.
In areas of uncertainty, ‘expert’ judgement is often deferred to in order to verify a species identity. But this in itself has its own inherent uncertainties. Defining who an expert is can be tricky and is dependant on a whole range of factors. With regards to obtaining a GCN license, the researchers comment:
“…instead of being required to demonstrate the relevant identification skills, applicants are required to supply a written reference from another licence holder. Therefore, licences are issued according to the perceptions and opinions of other practitioners within the domain, demonstrating that both individuals, and the community as a whole, are considered to be competent in newt identification”.
In order to test how accurately license holders are able to correctly identify newts, the researchers used images that did not display a geographical or environmental context that could potentially influence the participants answers. They sought to determine the following factors:
- How experts sort and name images of different newt species.
- Whether this was linked to (1) self-assessed identification ability; (2) perceived identification ability in comparison with peers; and (3) experience.
- Accuracy between professional and volunteer surveyors.
- Whether the presence of certain diagnostic features in the images was linked to higher accuracy of identification.
A total of seventeen licensed volunteers took part in the research. The study chose four species of newt found in the UK (smooth (Lissotriton vulgaris), palmate (L. helveticus), great crested and alpine (Ichthyosaura alpestris)) and through means of a Google search, pulled out twenty images of each species. The identification label of the online image was assumed to be correct and a preference towards images from sources perceived to be most reliable in terms of correct labelling was made, such as The Wildlife Trusts. The participants were asked to group the images as they saw fit and then to label each group according to species.
The Results
The project found that nine out of the seventeen participants correctly sorted the images into four groups and assigned the names of each of the study species to their groups. However, these groups were different for each participant, and none of the participants sorted their images into groups that agreed with the species named in the downloaded images. In addition to grouping the images in accordance with the study groups, seven participants created and named further groups outside of the study groups.
The mean overall accuracy was found to be 83%. This percentage increased to 87% when only counting images that were mistaken for one of the other study species. Judgements on species identification were found to be inconsistent, and approximately half of the participants named species that were not present, or opted for ‘don’t know’ decisions.
Participants differed in their level of experience (from four to twenty-six years), in their perception of their own identification abilities, and the identification abilities of their peers, but none of these were found to correlate to individual performance.
Although approximately half of the participants grouped and named the four study species only, the images included in the species groups that they made, differed between participants. However, the mean accuracy of participants naming only the four study species was 91%, compared with an average of 74% accuracy by those who designated one of their groups as ‘don’t know’. These findings suggest that accurate identification from images alone is limited. This could be as a result of factors such as image quality, image composition, or even the species itself.
The study reported that as well as identification varying between participants, it was also highly variable among species, ranging from 96% for GCN to 69% for palmate newt.
A few of the more pertinent conclusions and comments are detailed below:
- Agreement between experts can be inconsistent and have limitations.
- Participation was restricted to those in possession of a European Protected Species license, the issuing of which relates to competency in field identification rather than image identification. However, this restriction allowed the study of differences between individuals, all of whom had demonstrated a certain level of expertise within their community.
- Experience in species identification can increase the knowledge associated with certain features being ‘typical’ of a species, rather than being absolute, defining characteristics.
- Some participants noted that images were no substitute for handling the animal. Conversely, not everyone favours handling, or are permitted to handle specimens, and these restrictions can limit the observation of diagnostic characteristics – the research also found that agreement on identification was higher in images showing views of the underside of the newt, but to obtain this view, the newt generally needs to be handled.
- Individuals who are not licensed and therefore are restricted in their opportunities to look for diagnostic features, may find it difficult to accurately identify the species. Add into that, differences in site conditions (e.g. light levels), and time of year influencing the appearance of the newt, confusion can be encountered by an inexperienced surveyor.
- Whilst the findings indicate that it is best to consult with as many experts as possible in order to identify the species correctly, participants agreed on the species name for less than one-in-four of all images, demonstrating the difficulty of naming an organism from a single image.
Reference:
About the Author: Kate Priestman (CEnv, MCIEEM) has over sixteen years experience as an ecologist. Prior to setting up her own consultancy business in 2012, Kate worked in London for over a decade, providing the lead ecology role for a number of high profile projects. Kate works as an artist, author, writer and editor.
