taxastand is one of several R packages that match scientific names against a taxonomy. This vignette compares it with three others that do a similar job, on names with known errors, so that you can judge which suits your data:
Three other packages resolve names by calling a web service instead of matching against a local reference:
name_backbone()
matches names against the GBIF backbone.One of these may be a good choice when you want the current taxonomy of a particular service and can work online. However, because they are API-based, the same query can give a different answer once the service updates its taxonomy or its matching, and the time taken depends on the network and the server, not on the package. Results can therefore not be reproduced reliably or timed fairly, so they are left out of this benchmark.
The numbers come from a benchmark that was run once, on the maintainer’s machine, and saved with the package (the packages compared are not dependencies of taxastand, and taxify needs a large download). All the code that produced them is shown below. Please read the limits before drawing conclusions.
The same reference for every package. Each package is given exactly the same reference table: the World Flora Online backbone as installed by taxify (1,595,937 species and infraspecific names). taxify can only use its own backbones, WorldFlora reads WFO, and taxastand and U.Taxonstand accept any table, so using WFO lets all four run on identical data and makes the numbers directly comparable.
Queries with known answers. Each query is made by changing a name taken from the reference, so the correct record is known. There are 100 queries for each kind of damage (fewer where the damage did not change the name), drawn at random. The query code is in The code.
| kind | what changes | example |
|---|---|---|
exact |
Nothing: the full name with its author | Mahonia paxii Fedde → Mahonia paxii Fedde |
epithet_substitution |
One letter of the epithet is replaced | Gilia anbigua Rattan → Gilia aqbigua Rattan |
epithet_deletion |
One letter of the epithet is deleted | Piper oviedoi Urb. → Piper ovidoi Urb. |
epithet_swap |
Two neighboring letters of the epithet are swapped | Erica triceps Link → Erica tricesp Link |
two_edits |
Two letters of the epithet are changed | Lobelia inflata L. → Lobelia nqlata L. |
short_epithet |
One letter deleted from an epithet of five letters or fewer | Quercus robur Asso → Quercus obur Asso |
genus_deletion |
One letter of the genus is deleted | Vepris inermis Comm. → Veprs inermis Comm. |
author_punctuation |
Periods removed from the author | Croton spinosus L. → Croton spinosus L |
author_typo |
One letter deleted from the author | Poa polyphylla Hack. → Poa polyphylla Hck. |
no_author_exact |
The name without its author | Pyrus albicans Kit. → Pyrus albicans |
no_author_epithet_deletion |
The name without its author, with a letter of the epithet deleted | Elvira martyni Cass. → Elvira martyi |
basionym_missing |
The basionym author (in parentheses) is missing | Carex tremula (Ohwi) Ohwi → Carex tremula Ohwi |
ex_dropped |
The ex element of the author is dropped | Rosa remensis Desf. ex Bosc → Rosa remensis Desf. |
rank_changed |
The infraspecific rank is changed (var. or subsp.) | Phalaris minor var. integra Trab. → Phalaris minor subsp. integra Trab. |
spelling_variant |
A standard Latin spelling variant of the epithet (y/i, ph/f, th/t, rh/r, ae/e) | Poa nephelophila Bor → Poa nefelophila Bor |
The last four kinds change how the name is written, not how it is
spelled. They are differences that taxastand checks for explicitly (see
the “Match types” section of ?ts_match_names). A fifth,
dropping an in element (Tagawa in Morton queried
as Tagawa), could not be made, because WFO has almost no
names with an in element. The queries above are drawn at random
from the reference, so most of them are for names that only one record
has. A separate section looks at names that
several records share.
What is measured. Every query is scored by record ID:
WFO.one(),
U.Taxonstand through its best match. taxastand does not guess: it
returns every record that fits, and its single answer is the record when
only one fits. The tables give the share of queries that received a
single answer, then the share of those answers that were right
(“answered / right”).
knitr::kable(
wide("recall"),
caption = "Share of queries for which the right record was returned"
)| kind | taxastand | taxify | U.Taxonstand | WorldFlora |
|---|---|---|---|---|
| exact | 100% | 100% | 94% | 100% |
| epithet_substitution | 100% | 99% | 100% | 100% |
| epithet_deletion | 100% | 99% | 100% | 99% |
| epithet_swap | 100% | 100% | 100% | 100% |
| two_edits | 100% | 99% | 100% | 100% |
| short_epithet | 100% | 99% | 100% | 88% |
| genus_deletion | 95% | 82% | 99% | 99% |
| author_punctuation | 100% | 100% | 97% | 100% |
| author_typo | 100% | 93% | 95% | 100% |
| no_author_exact | 100% | 100% | 100% | 100% |
| no_author_epithet_deletion | 100% | 98% | 100% | 100% |
| basionym_missing | 100% | 100% | 95% | 100% |
| ex_dropped | 100% | 97% | 80% | 100% |
| rank_changed | 100% | 71% | 85% | 100% |
| spelling_variant | 100% | 99% | 99% | 98% |
single <- function(kinds = names(what_main)) {
out <- data.frame(kind = kinds, stringsAsFactors = FALSE)
for (tool in shown) {
a <- res$accuracy[[tool]]
i <- match(kinds, a$kind)
out[[tool_labels[[tool]]]] <- paste0(
pct(a$pick_answered[i]), " / ",
ifelse(is.na(a$pick_precision[i]), "-", pct(a$pick_precision[i]))
)
}
out
}
knitr::kable(
single(),
caption = paste0(
"Share of queries that got a single answer / share of those answers ",
"that were right"
)
)| kind | taxastand | taxify | U.Taxonstand | WorldFlora |
|---|---|---|---|---|
| exact | 100% / 100% | 100% / 100% | 100% / 94% | 100% / 100% |
| epithet_substitution | 91% / 100% | 100% / 99% | 100% / 99% | 100% / 99% |
| epithet_deletion | 94% / 100% | 100% / 99% | 100% / 100% | 99% / 100% |
| epithet_swap | 100% / 100% | 100% / 100% | 100% / 100% | 100% / 100% |
| two_edits | 93% / 100% | 100% / 99% | 100% / 99% | 100% / 99% |
| short_epithet | 82% / 100% | 100% / 99% | 100% / 100% | 88% / 100% |
| genus_deletion | 89% / 100% | 86% / 95% | 100% / 99% | 99% / 100% |
| author_punctuation | 100% / 100% | 100% / 100% | 100% / 97% | 100% / 100% |
| author_typo | 99% / 100% | 100% / 93% | 100% / 95% | 100% / 100% |
| no_author_exact | 94% / 100% | 100% / 100% | 100% / 100% | 100% / 100% |
| no_author_epithet_deletion | 63% / 100% | 99% / 99% | 100% / 99% | 100% / 98% |
| basionym_missing | 100% / 100% | 100% / 100% | 100% / 94% | 100% / 96% |
| ex_dropped | 97% / 100% | 100% / 97% | 100% / 80% | 100% / 79% |
| rank_changed | 100% / 100% | 100% / 71% | 100% / 84% | 100% / 100% |
| spelling_variant | 100% / 100% | 100% / 99% | 100% / 99% | 99% / 99% |
Where a package returns several records, taxastand’s result tells you
which queries need a decision. On the author-less typo kind
(no_author_epithet_deletion) taxastand returned 3.3 records
per answered query on average: without an author, many names have
several records that fit equally well, and taxastand reports all of
them.
This section is different from the ones above, so read it separately. Each query here is made from a record whose genus and epithet are also used by at least one other record with a different author (a homonym, or a name that differs only in its rank). Such records were chosen on purpose, because it is where the author, and so the nomenclatural rules, has to decide which record is meant. The results say how each package behaves when a name is shared. They do not say how often that happens: roughly one record in ten in WFO shares its genus and epithet with another. For the other nine in ten, the genus and epithet already point to one record, and the tables above apply.
| kind | what changes | example |
|---|---|---|
homonym |
A name that several records share, queried with its own author | Linum perenne Lam. → Linum perenne Lam. |
homonym_typo |
The same, with a letter of the epithet deleted | Oenothera alata Raf. → Oenothera lata Raf. |
homonym_basionym_missing |
A shared name, with the basionym author missing | Urtica hirta (Sw.) Sw. → Urtica hirta Sw. |
homonym_ex_dropped |
A shared name, with the ex element of the author dropped | Lantana dubia Wall. ex Walp. → Lantana dubia Wall. |
homonym_rank_changed |
A shared name, with the infraspecific rank changed | Cornus mas var. alba Weston → Cornus mas subsp. alba Weston |
homonym_spelling_variant |
A shared name, with a Latin spelling variant of the epithet | Thymus zygis Pall. → Thymus zigis Pall. |
knitr::kable(
wide("recall", names(what_shared)),
caption = paste0(
"Names shared by several records: share of queries for which the right ",
"record was returned"
)
)| kind | taxastand | taxify | U.Taxonstand | WorldFlora |
|---|---|---|---|---|
| homonym | 100% | 100% | 81% | 100% |
| homonym_typo | 100% | 99% | 99% | 99% |
| homonym_basionym_missing | 100% | 95% | 52% | 100% |
| homonym_ex_dropped | 99% | 88% | 36% | 100% |
| homonym_rank_changed | 96% | 34% | 47% | 100% |
| homonym_spelling_variant | 100% | 100% | 99% | 99% |
knitr::kable(
wide("unambiguous", names(what_shared)),
caption = paste0(
"Names shared by several records: share of queries for which only the ",
"right record was returned"
)
)| kind | taxastand | taxify | U.Taxonstand | WorldFlora |
|---|---|---|---|---|
| homonym | 100% | 100% | 81% | 0% |
| homonym_typo | 88% | 99% | 0% | 0% |
| homonym_basionym_missing | 97% | 95% | 46% | 0% |
| homonym_ex_dropped | 87% | 88% | 33% | 0% |
| homonym_rank_changed | 92% | 34% | 0% | 0% |
| homonym_spelling_variant | 100% | 100% | 0% | 0% |
knitr::kable(
single(names(what_shared)),
caption = paste0(
"Names shared by several records: share of queries that got a single ",
"answer / share of those answers that were right"
)
)| kind | taxastand | taxify | U.Taxonstand | WorldFlora |
|---|---|---|---|---|
| homonym | 100% / 100% | 100% / 100% | 100% / 81% | 100% / 100% |
| homonym_typo | 88% / 100% | 100% / 99% | 100% / 99% | 100% / 99% |
| homonym_basionym_missing | 97% / 100% | 100% / 95% | 100% / 48% | 100% / 59% |
| homonym_ex_dropped | 88% / 98% | 100% / 88% | 100% / 34% | 100% / 28% |
| homonym_rank_changed | 96% / 96% | 100% / 34% | 100% / 45% | 100% / 98% |
| homonym_spelling_variant | 100% / 100% | 100% / 100% | 100% / 99% | 100% / 99% |
When a name is shared, the packages differ much more than above:
The comparison is again about the records returned, not about which package is “best”: a package that returns every record that fits gives the user the decision, and one that returns a single record decides for them.
speed_table <- function() {
s <- do.call(rbind, res$speed)
s$tool <- speed_labels[s$tool]
w <- reshape(s, idvar = "tool", timevar = "n_queries", direction = "wide")
names(w) <- sub("seconds.", "", names(w), fixed = TRUE)
rownames(w) <- NULL
w
}
knitr::kable(
speed_table(), na = "not run",
caption = "Seconds to match n queries (best of 3, after warm-up)"
)| tool | 1 | 10 | 100 | 1000 |
|---|---|---|---|---|
| taxastand (optimized) | 0.02 | 0.15 | 0.37 | 3.09 |
| taxastand (not optimized) | 0.66 | 1.06 | 4.70 | 24.72 |
| taxify | 0.01 | 0.43 | 0.74 | 2.50 |
| U.Taxonstand | 0.93 | 2.56 | 17.36 | NA |
| WorldFlora | 0.74 | 11.15 | 126.32 | NA |
U.Taxonstand and WorldFlora were timed only up to 100 queries, because larger runs on a reference of this size take many minutes for those packages.
The “taxastand (optimized)” row are results that used a reference
optimized once with ts_optimize_ref() and reused for every
call. The “taxastand (not optimized)” row are results that passed the
reference names (a character vector) to ts_match_names(),
so the reference is parsed again on every call.
The timings are made after a warm-up: before timing, each package was run once on the whole set of queries, so one-time costs are not counted. An optimized reference, for example, is parsed lazily, one genus at a time, so the first time a genus is used costs more than later calls.
acceptedNameUsageID.match = FALSE and
WFO.one(Auth.dist = ...) aside; see the code below).
Settings such as the thresholds for fuzzy matching can be changed, and a
package tuned to your own data could give different results. The
defaults differ in kind: taxify allows a fraction of the name to differ
and counts a swap of two neighboring letters as one edit, while
taxastand allows a fixed number of edits (max_dist) and, by
default, counts a swap as two (transpositions = FALSE). How
a name is passed in also matters: WorldFlora was given infraspecific
names as separate genus, species and infraspecific epithet columns, the
way to ask for a match that ignores the rank. Given as one string, a
variety that is queried as a subspecies is not found.This is the code that produced the results above, shown unchanged. It
is also installed with taxastand
(system.file("benchmark", package = "taxastand")). The
scripts that run it, data-raw/benchmark-vignette.R
(including the kinds and speed arguments used
to run some kinds separately) and
data-raw/benchmark-combine.R (which merges the runs), are
in the source repository.
The reference is read from taxify’s installed backbone, reduced to species and infraspecific names, and given one row per distinct authored name.
# One row per distinct authored name, with columns:
# taxonID, canonical, authorship (NA if none), full (canonical + authorship),
# taxonRank (lower case), taxonomicStatus, acceptedNameUsageID (NA if
# accepted)
load_reference <- function(source = c("ppg", "wfo"), ppg_path = NULL,
ppg_version = "v2.0.0", wfo_backbone = "wfo") {
source <- match.arg(source)
if (source == "ppg") {
if (is.null(ppg_path)) ppg_path <- file.path(tempdir(), "ppg.csv")
if (!file.exists(ppg_path)) {
utils::download.file(
sprintf(
"https://raw.githubusercontent.com/pteridogroup/ppg/%s/data/ppg.csv",
ppg_version
),
ppg_path,
quiet = TRUE
)
}
x <- utils::read.csv(ppg_path, stringsAsFactors = FALSE, na.strings = "NA")
x <- x[!is.na(x$scientificName), ]
} else {
# taxify keeps each backbone as a vectra file in its data directory
f <- file.path(
taxify::taxify_data_dir(), wfo_backbone, "latest",
paste0(wfo_backbone, ".vtr")
)
if (!file.exists(f)) stop("taxify backbone not found: ", f)
b <- as.data.frame(vectra::collect(vectra::tbl(f)))
x <- data.frame(
taxonID = b$taxon_id, scientificName = b$canonical_name,
scientificNameAuthorship = b$authorship,
taxonRank = tolower(b$taxon_rank),
taxonomicStatus = tolower(b$taxonomic_status),
acceptedNameUsageID = ifelse(
b$accepted_taxon_id == b$taxon_id, NA_character_, b$accepted_taxon_id
),
stringsAsFactors = FALSE
)
}
x <- x[x$taxonRank %in% c("species", "subspecies", "variety", "form"), ]
auth <- x$scientificNameAuthorship
has_auth <- !is.na(auth) & auth != ""
full <- x$scientificName
full[has_auth] <- paste(x$scientificName[has_auth], auth[has_auth])
out <- data.frame(
taxonID = x$taxonID, canonical = x$scientificName,
authorship = ifelse(has_auth, auth, NA_character_), full = full,
taxonRank = x$taxonRank, taxonomicStatus = x$taxonomicStatus,
acceptedNameUsageID = x$acceptedNameUsageID,
stringsAsFactors = FALSE
)
out[!duplicated(out$full), ]
}damage() changes one letter; make_queries()
builds each kind of query and records the correct answer.
# Damage one letter of `x`: substitution, deletion, or swap of two neighbors.
# `from` is the first position that may be changed.
damage <- function(x, kind, from = 1L) {
n <- nchar(x)
if (n < from + 1L) {
return(x)
}
i <- if (n - from > 0L) from - 1L + sample.int(n - from, 1L) else from
switch(kind,
substitution = `substr<-`(x, i, i, if (substr(x, i,
i) == "q") "z" else "q"),
deletion = paste0(substr(x, 1, i - 1), substr(x, i + 1, n)),
swap = {
i <- min(i, n - 1L)
a <- substr(x, i, i)
b <- substr(x, i + 1L, i + 1L)
if (a == b) x else `substr<-`(x, i, i + 1L, paste0(b, a))
}
)
}
# Queries made from random species-rank records that have an author, `n` per
# kind of error. Returns a data frame: query, kind, source (row of `ref`) and
# `truth` (a list of the record IDs that count as the right answer: the record
# itself, or, when the query has no author, every record sharing the canonical
# name).
# exact, no_author_exact the name as in the reference (without author)
# epithet_*, genus_deletion one letter changed, with the author
# two_edits two letters changed in the epithet
# short_epithet one deletion in an epithet of five letters or
# less
# author_punctuation periods removed from the author
# author_typo a letter deleted in the author
# homonym, homonym_typo a name shared by records with different authors,
# queried with its own author (as is, or with a
# typo in the epithet): needs the author to decide
# Nomenclatural differences (the form of the name changes, not its spelling).
# They come last, so that the other kinds are the same with or without them,
# and a query that is the same as another reference name is left out, so the
# right record stays unambiguous:
# basionym_missing "(Turcz.) Czerep." queried as "Czerep."
# ex_dropped, in_dropped "Willd. ex Schkuhr" as "Willd.", "A in B" as "A"
# rank_changed "var." queried as "subsp." or the reverse
# spelling_variant y/i, ph/f, th/t, rh/r or ae/e in the epithet
# The same four, with a homonym_ prefix, for records that share their genus and
# epithet with another record: here the nomenclatural rules decide which record
# is meant. They are picked because they have competitors, so they show how a
# package behaves when a name is shared, not how often that happens.
make_queries <- function(ref, n = 100, seed = 20261109) {
set.seed(seed)
species <- ref$taxonRank == "species" & !is.na(ref$authorship) &
grepl("^[A-Z][a-z]+ [a-z-]+$", ref$canonical)
epithet <- sub("^\\S+ ", "", ref$canonical)
dup_canon <- ref$canonical %in% ref$canonical[duplicated(ref$canonical)]
auth <- ref$authorship
not_basio <- !is.na(auth) & !grepl("^[(]", auth)
infra <- ref$taxonRank %in% c("subspecies", "variety") & !is.na(auth) &
grepl("^[A-Z][a-z]+ [a-z-]+ (var[.]|subsp[.]) [a-z-]+$", ref$canonical)
pools <- list(
default = which(species & nchar(ref$canonical) > 12),
short = which(species & nchar(epithet) <= 5 & nchar(epithet) >= 3),
homonym = which(species & dup_canon),
basionym = which(species & grepl("^[(][^)]*[)]\\s*\\S", auth)),
ex = which(species & not_basio & grepl(" ex ", auth, fixed = TRUE)),
`in` = which(species & not_basio & grepl(" in [A-Z]", auth)),
infra = which(infra),
foldable = which(species & grepl("y|ph|th|rh|ae", epithet))
)
# The same records, but only those that share their genus and epithet
# (ignoring the rank) with another record, so that the author, and with it the
# nomenclatural rules, has to decide which record is meant
key <- sub(" (var[.]|subsp[.]|f[.]) ", " ", ref$canonical)
shared <- key %in% key[duplicated(key)]
for (k in c("basionym", "ex", "infra", "foldable")) {
pools[[paste0(k, "_shared")]] <- pools[[k]][shared[pools[[k]]]]
}
by_canon <- split(ref$taxonID, ref$canonical)
# name parts of record i
parts <- function(src) {
list(
genus = sub(" .*", "", ref$canonical[src]), epi = epithet[src],
auth = ref$authorship[src]
)
}
dmg <- function(x, kind, from = 1L) vapply(x, damage, "", kind, from = from)
author_typo <- function(a) {
vapply(a, function(x) {
w <- strsplit(x, "(?<=[ .])", perl = TRUE)[[1]]
k <- which.max(nchar(w))
w[k] <- damage(w[k], "deletion", from = 2L)
paste(w, collapse = "")
}, "")
}
# swap var. and subsp.
swap_rank <- function(x) {
ifelse(
grepl(" var. ", x, fixed = TRUE), sub(" var. ", " subsp. ", x,
fixed = TRUE),
sub(" subsp. ", " var. ", x, fixed = TRUE)
)
}
# fold one Latin spelling variant (the first that applies) in an epithet
spelling_variant <- function(x) {
vapply(x, function(e) {
for (pat in list(c("y", "i"), c("ph", "f"), c("th", "t"), c("rh", "r"),
c("ae", "e"))) {
if (grepl(pat[1], e, fixed = TRUE)) {
return(sub(pat[1], pat[2], e, fixed = TRUE))
}
}
e
}, "")
}
f_basionym <- function(p) {
paste(p$genus, p$epi, sub("^[(][^)]*[)]\\s*", "", p$auth))
}
f_ex <- function(p) paste(p$genus, p$epi, sub(" ex .*$", "", p$auth))
f_in <- function(p) paste(p$genus, p$epi, sub(" in .*$", "", p$auth))
f_rank <- function(p) paste(p$genus, swap_rank(p$epi), p$auth)
f_spelling <- function(p) paste(p$genus, spelling_variant(p$epi), p$auth)
kinds <- list(
exact = list("default", function(p) paste(p$genus, p$epi, p$auth)),
epithet_substitution = list("default", function(p) {
paste(p$genus, dmg(p$epi, "substitution"), p$auth)
}),
epithet_swap = list("default", function(p) {
paste(p$genus, dmg(p$epi, "swap"), p$auth)
}),
epithet_deletion = list("default", function(p) {
paste(p$genus, dmg(p$epi, "deletion"), p$auth)
}),
two_edits = list("default", function(p) {
paste(p$genus, dmg(dmg(p$epi, "deletion"), "substitution"), p$auth)
}),
short_epithet = list("short", function(p) {
paste(p$genus, dmg(p$epi, "deletion"), p$auth)
}),
genus_deletion = list("default", function(p) {
paste(dmg(p$genus, "deletion", from = 2L), p$epi, p$auth)
}),
no_author_exact = list("default", function(p) paste(p$genus, p$epi)),
no_author_epithet_deletion = list("default", function(p) {
paste(p$genus, dmg(p$epi, "deletion"))
}),
author_punctuation = list("default", function(p) {
paste(p$genus, p$epi, gsub(".", "", p$auth, fixed = TRUE))
}),
author_typo = list("default", function(p) {
paste(p$genus, p$epi, author_typo(p$auth))
}),
homonym = list("homonym", function(p) paste(p$genus, p$epi, p$auth)),
homonym_typo = list("homonym", function(p) {
paste(p$genus, dmg(p$epi, "deletion"), p$auth)
}),
# a third element, TRUE, leaves out queries equal to another reference name
basionym_missing = list("basionym", f_basionym, TRUE),
ex_dropped = list("ex", f_ex, TRUE),
in_dropped = list("in", f_in, TRUE),
rank_changed = list("infra", f_rank, TRUE),
spelling_variant = list("foldable", f_spelling, TRUE),
# the same, for names that several records share
homonym_basionym_missing = list("basionym_shared", f_basionym, TRUE),
homonym_ex_dropped = list("ex_shared", f_ex, TRUE),
homonym_rank_changed = list("infra_shared", f_rank, TRUE),
homonym_spelling_variant = list("foldable_shared", f_spelling, TRUE)
)
out <- do.call(rbind, lapply(names(kinds), function(k) {
pool <- pools[[kinds[[k]][[1]]]]
if (!length(pool)) {
return(NULL) # the reference has no names for this kind of query
}
src <- pool[sample.int(length(pool), min(n, length(pool)))]
q <- trimws(kinds[[k]][[2]](parts(src)))
keep <- if (length(kinds[[k]]) > 2) !q %in% ref$full else TRUE
data.frame(
query = q, kind = k, source = src, stringsAsFactors = FALSE
)[keep, ]
}))
out$truth <- lapply(seq_len(nrow(out)), function(i) {
if (grepl("^no_author", out$kind[i])) {
by_canon[[ref$canonical[out$source[i]]]]
} else {
ref$taxonID[out$source[i]]
}
})
out$original <- ref$full[out$source]
# a query string can arise from different records or kinds; keep each once
out[!duplicated(out$query), ]
}
# `hits` has one row per record returned: query, id, and `pick` (TRUE for the
# record the tool itself chooses as its answer; at most one per query).
# One row per query: n_hits (records returned), correct (the right record is
# among them), unambiguous (it is the only one returned), picked (the tool chose
# a record) and pick_correct (the chosen record is right).
score_queries <- function(queries, hits) {
hits <- hits[!duplicated(hits[, c("query", "id")]), ]
lv <- queries$query
by_q <- split(hits$id, factor(hits$query, levels = lv))
picks <- split(
hits$id[hits$pick], factor(hits$query[hits$pick], levels = lv)
)
n_hits <- vapply(by_q, length, 0L)
correct <- mapply(function(h, t) any(h %in% t), by_q, queries$truth)
picked <- vapply(picks, length, 0L) > 0
pick_correct <- mapply(
function(p, t) length(p) > 0 && p[[1]] %in% t, picks, queries$truth
)
data.frame(
query = queries$query, kind = queries$kind, n_hits = unname(n_hits),
correct = unname(correct), unambiguous = unname(correct & n_hits == 1L),
picked = unname(picked), pick_correct = unname(pick_correct),
stringsAsFactors = FALSE
)
}
# Per kind of error:
# - recall: share of all queries for which the right record is among those
# returned; precision: the same share among queries that got an answer
# - unambiguous: share where the right record is the only one returned;
# mean_hits: average records returned per answered query
# - pick_answered: share of queries for which the tool chose one record;
# pick_precision: share of those choices that are right
summarize_scores <- function(scores) {
f <- function(s) {
answered <- s$n_hits > 0
data.frame(
n = nrow(s), answered = mean(answered), recall = mean(s$correct),
precision = if (any(answered)) mean(s$correct[answered]) else NA_real_,
unambiguous = mean(s$unambiguous),
mean_hits = if (any(answered)) mean(s$n_hits[answered]) else NA_real_,
pick_answered = mean(s$picked),
pick_precision = if (any(s$picked)) {
mean(s$pick_correct[s$picked])
} else {
NA_real_
}
)
}
out <- do.call(rbind, lapply(split(scores, scores$kind), f))
out$kind <- rownames(out)
rownames(out) <- NULL
out[, c("kind", setdiff(colnames(out), "kind"))]
}Each adapter builds its package’s reference from the shared table and returns the records matched for a set of queries, plus the package’s own single answer.
U.Taxonstand and WorldFlora take the name and the author separately, so this helper splits a query first.
# U.Taxonstand and WorldFlora take the name and the author separately. Split a
# query into the name (genus and epithet, plus the rank and infraspecific
# epithet of an infraspecific name) and the author.
split_query <- function(q) {
pattern <- "^(\\S+ \\S+ (?:var[.]|subsp[.]|f[.]) \\S+|\\S+ \\S+)\\s*(.*)$"
m <- regmatches(q, regexec(pattern, q, perl = TRUE))
name <- vapply(m, `[`, "", 2)
data.frame(
name = name, author = vapply(m, `[`, "", 3),
infraspecific = grepl(" (var[.]|subsp[.]|f[.]) ", name),
stringsAsFactors = FALSE
)
}
# Each make_*() takes the reference table and returns a list with `setup_s` (the
# time spent preparing the reference, once) and `run(queries)`.
make_taxastand <- function(ref, prepared = TRUE) {
tax <- data.frame(
taxonID = ref$taxonID, acceptedNameUsageID = ref$acceptedNameUsageID,
taxonomicStatus = ref$taxonomicStatus, scientificName = ref$full,
stringsAsFactors = FALSE
)
start <- proc.time()
reference <- if (prepared) {
taxastand::ts_optimize_ref(tax)
} else {
tax$scientificName
}
setup <- (proc.time() - start)[["elapsed"]]
list(
setup_s = setup,
run = function(q) {
m <- suppressWarnings(
taxastand::ts_match_names(q, reference, simple = TRUE)
)
m <- m[m$match_type != "no_match", ]
h <- data.frame(
query = m$query, id = ref$taxonID[match(m$reference, ref$full)],
stringsAsFactors = FALSE
)
# taxastand does not guess: its answer is the record when only one fits
n_ids <- tapply(h$id, h$query, function(x) length(unique(x)))
h$pick <- n_ids[h$query] == 1L
h
}
)
}
make_taxify <- function(ref, backbone = "wfo") {
setup <- system.time(
taxify::taxify("Quercus robur", backbone = backbone, verbose = FALSE)
)[["elapsed"]]
list(
setup_s = setup,
run = function(q) {
r <- as.data.frame(suppressWarnings(suppressMessages(
taxify::taxify(q, backbone = backbone, verbose = FALSE)
)))
ok <- !is.na(r$taxon_id)
# taxify returns one record per query: that is its pick
data.frame(
query = r$input_name[ok], id = r$taxon_id[ok], pick = TRUE,
stringsAsFactors = FALSE
)
}
)
}
make_utaxonstand <- function(ref) {
genus <- sub(" .*", "", ref$canonical)
setup <- system.time({
db <- data.frame(
ID = ref$taxonID, Name = ref$canonical,
Author = ifelse(is.na(ref$authorship), "", ref$authorship),
Genus = genus,
Rank = ifelse(ref$taxonRank == "species", 2, 3),
ACCEPTED_ID = ref$acceptedNameUsageID,
stringsAsFactors = FALSE
)
db <- U.Taxonstand::nameClean(db)
})[["elapsed"]]
list(
setup_s = setup,
run = function(q) {
parts <- split_query(q)
sp <- data.frame(
Name = parts$name, Author = parts$author,
Rank = ifelse(parts$infraspecific, 3, 2), stringsAsFactors = FALSE
)
# nameMatch() prints progress bars; discard them
utils::capture.output({
r <- suppressWarnings(suppressMessages(
U.Taxonstand::nameMatch(
spList = sp, spSource = db, author = TRUE, matchFirst = FALSE
)
))
})
# one row per matched record; Name_in_database may be empty on no match
ok <- !is.na(r$ID_in_database) & r$ID_in_database != ""
idx <- match(
paste(r$Submitted_Name, r$Submitted_Author),
paste(sp$Name, sp$Author)
)
# Name_set == 1 marks the best match of each query
h <- data.frame(
query = q[idx[ok]], id = r$ID_in_database[ok],
pick = r$Name_set[ok] == 1, stringsAsFactors = FALSE
)
h$pick <- h$pick & !duplicated(cbind(h$query, h$pick))
h
}
)
}
make_worldflora <- function(ref) {
parts <- strsplit(ref$canonical, " ", fixed = TRUE)
setup <- system.time({
wfo <- data.frame(
taxonID = ref$taxonID, scientificNameID = "",
scientificName = ref$canonical,
taxonRank = toupper(ref$taxonRank), parentNameUsageID = "",
scientificNameAuthorship = ifelse(is.na(ref$authorship), "",
ref$authorship),
family = "", genus = vapply(parts, `[`, "", 1),
specificEpithet = vapply(parts, `[`, "", 2),
infraspecificEpithet = vapply(parts, function(p) {
if (length(p) > 2) p[length(p)] else ""
}, ""),
verbatimTaxonRank = "", nomenclaturalStatus = "", namePublishedIn = "",
taxonomicStatus = tools::toTitleCase(ref$taxonomicStatus),
acceptedNameUsageID = ifelse(
is.na(ref$acceptedNameUsageID), "", ref$acceptedNameUsageID
),
nameAccordingToID = "", stringsAsFactors = FALSE
)
})[["elapsed"]]
list(
setup_s = setup,
run = function(q) {
parts <- split_query(q)
# Match one set of queries and pick one record for each. WFO.match() with
# acceptedNameUsageID.match = FALSE returns the record that matched, not
# the accepted record it points to, so that matching is scored alone.
match_wfo <- function(spec, ...) {
if (!nrow(spec)) {
return(NULL)
}
utils::capture.output({
r <- suppressWarnings(suppressMessages(
WorldFlora::WFO.match(
spec.data = spec, WFO.data = wfo, Authorship = "Authorship",
acceptedNameUsageID.match = FALSE, no.dates = TRUE,
verbose = FALSE, counter = 1e9, ...
)
))
})
r <- r[!is.na(r$taxonID) & r$taxonID != "", ]
# WFO.one() chooses one record per query, preferring the best author
# match (Auth.dist); it expects the columns that WFO.match() adds only
# when it replaces synonyms by accepted names
r$New.accepted <- FALSE
r$Old.status <- ""
r$Old.ID <- ""
utils::capture.output({
one <- WorldFlora::WFO.one(
r,
Auth.dist = "Auth.dist", verbose = FALSE, counter = 1e9
)
})
data.frame(
query = r$key, id = r$taxonID,
pick = paste(r$key, r$taxonID) %in% paste(one$key, one$taxonID),
stringsAsFactors = FALSE
)
}
sp <- !parts$infraspecific
# Species: one name string. Infraspecific names: genus, species and
# infraspecific epithet in separate columns and no rank, the way to ask
# WorldFlora for a match that ignores the rank (given as one string, or
# with the rank, a var. is not found when it is queried as subsp.).
words <- strsplit(parts$name[!sp], " ", fixed = TRUE)
rbind(
match_wfo(data.frame(
spec.name = parts$name[sp], Authorship = parts$author[sp],
key = q[sp], stringsAsFactors = FALSE
)),
match_wfo(
data.frame(
Genus = vapply(words, `[`, "", 1),
Species = vapply(words, `[`, "", 2),
Infraspecific = vapply(words, `[`, "", 4),
Authorship = parts$author[!sp], key = q[!sp],
stringsAsFactors = FALSE
),
Genus = "Genus", Species = "Species", Infraspecific = "Infraspecific"
)
)
}
)
}
# Elapsed seconds (best of `reps`) to match `n` queries, after a warm-up call.
# The queries are shuffled (fixed seed) so that every size is a mix of the
# kinds of error: the query set is ordered by kind, and exact names are much
# cheaper to match than damaged ones.
time_adapter <- function(adapter, queries, sizes = c(1, 10, 100, 1000),
reps = 3) {
set.seed(20261109)
queries <- sample(queries)
adapter$run(queries[seq_len(min(5, length(queries)))])
do.call(rbind, lapply(sizes, function(n) {
q <- rep_len(queries, n)
q <- unique(q)
if (length(q) < n) q <- c(q, paste0(q[seq_len(n - length(q))], "x"))
t <- min(vapply(
seq_len(reps), function(i) unname(system.time(adapter$run(q))[3]), 0
))
data.frame(n_queries = n, seconds = round(t, 2))
}))
}
ref <- load_reference("wfo")
queries <- make_queries(ref, n = 100)
adapters <- list(
taxastand = make_taxastand(ref),
taxify = make_taxify(ref),
utaxonstand = make_utaxonstand(ref),
worldflora = make_worldflora(ref)
)
for (tool in names(adapters)) {
hits <- adapters[[tool]]$run(queries$query)
scores <- score_queries(queries, hits)
print(summarize_scores(scores)) # found, single answer
print(time_adapter(adapters[[tool]], queries$query)) # speed
}The R session and package versions on the machine that ran the benchmark (not the session that built this vignette), recorded for reproducibility.
#> R version 4.6.0 (2026-04-24)
#> Platform: aarch64-apple-darwin23
#> Running under: macOS Tahoe 26.5.2
#>
#> Matrix products: default
#> BLAS: /Library/Frameworks/R.framework/Versions/4.6/Resources/lib/libRblas.0.dylib
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.6/Resources/lib/libRlapack.dylib; LAPACK version 3.12.1
#>
#> locale:
#> [1] C.UTF-8/UTF-8/C.UTF-8/C/C.UTF-8/C.UTF-8
#>
#> time zone: Asia/Tokyo
#> tzcode source: internal
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> loaded via a namespace (and not attached):
#> [1] compiler_4.6.0 WorldFlora_1.14-5 plyr_1.8.9 magrittr_2.0.5
#> [5] cli_3.6.6 U.Taxonstand_1.4.0 tools_4.6.0 Rcpp_1.1.2
#> [9] jsonlite_2.0.0 vectra_0.14.1 rlang_1.3.0 libgeos_3.11.1-3
#> [13] taxify_0.7.1