Methodology
How Talio measures speaking
Every conversation in Talio is recorded, transcribed, and scored the same way. This page explains what each number on your analytics means, what signal it comes from, and what it does not measure. Two rules apply everywhere: the goal is to be understood, not to sound native, and the most useful comparison is you against yourself on the same kind of scenario, because the scenario moves every measure more than anything else does.
Lexical diversity
What we measure
How varied your word choice is: the average stretch of speech you produce before you start repeating words (MTLD). Higher means a wider range of words, not more words.
From what signal
Your tokenized transcript from the speech recognizer. Each of your turns is split into words (as the tokenizer segments them, in Chinese too), and the index is computed over your side of the conversation only. A conversation needs about 50 words to show a value and about 100 for a reliable one; shorter ones show as gaps.
What it is not
Lexical diversity is not vocabulary size. A small vocabulary can score high by not repeating, and a large one can score low in a scenario that keeps calling for the same words. It is not a fluency score either. The scenario moves it more than anything else, so compare conversations of the same type.
Read more
Speaking speed
What we measure
Words you say per minute across your own speaking turns (characters per minute in Chinese, mora per minute in Japanese). Pauses within a turn count toward the time.
From what signal
Voice-activity detection on your audio. Each turn is timed from the start of your first stretch of speech to the end of your last. The time before you begin a turn, and time spent listening, are not counted.
What it is not
Speaking speed is not a score, and faster is not better. Very slow and very fast speech are both harder to follow, and listeners find learner speech easiest at a pace a little below native conversational rate. The shaded band on the chart shows a typical conversational pace for reference only; for Japanese it is a provisional target range. Rates also differ across languages, so figures are not comparable between them.
Read more
Mean length of run
What we measure
The average amount you say between pauses, in words (characters in Chinese, mora in Japanese). Longer runs mean you are retrieving words and building sentences with less searching.
From what signal
Voice-activity detection on your audio. A run is a stretch of speech with no pause of a quarter second or longer inside it; we divide the words you said by the number of runs.
What it is not
It is not a measure of how much you said overall, and it is not a grammar score. A long run can be a rehearsed phrase, and a short one can be a careful, correct answer. It rises as a scenario becomes familiar, so compare repeats of the same scenario.
Read more
Intelligibility
What we measure
How clearly a listener could make out what you said, from 0 to 100.
From what signal
Your audio, scored turn by turn. Where a word is unclear, the score for that turn drops; where every word comes through, the score is high.
What it is not
Intelligibility is about whether your words can be made out. It is not a comparison to any reference speaker, but like any listener it is somewhat harder on unfamiliar accents, so it is not accent-blind: accent is not what it measures, unclear words are. It is also not the same thing as pronunciation, which scores individual sounds. The bands on the chart are Talio's own; no cutoff on this scale is a standard from the research literature.
Read more
Pronunciation
What we measure
How closely the sounds you produced match the expected sounds for each word, from 0 to 100, scored by the speech assessment engine. It scores individual sounds, and tones in Chinese.
From what signal
The pronunciation assessment engine's per-sound accuracy scores, adjusted for how clearly each word came through. Each word in your turn is aligned to its expected sounds, each sound gets an accuracy score, and the word and turn scores are averaged from those. Strengths and focus areas are ranked by how often a sound occurs and how far it sits below your own average; for Chinese, each sound category also carries a weight for how much it tends to matter for being understood.
What it is not
Pronunciation scores individual sounds and tones, not stress, rhythm, or accent. The goal is to be understood, not to sound native. A low score on a sound does not automatically make it the sound to work on first, because some sound contrasts matter far more to a listener than others. The green band (75 to 100) is the range Talio treats as good to excellent; it is Talio's own descriptor, not a threshold from the literature.
Read more
Vocabulary
What we measure
Which words have come up in your conversations, and whether you have said them yourself. “Heard” means the tutor used the word in a conversation with you; “said” means you produced it. Words are grouped by their dictionary form, so run, ran, and running count once.
From what signal
The tutor's and your tokenized transcripts. Each word in each turn is attributed to the speaker who produced it, and the counts accumulate across your conversations.
What it is not
Neither heard nor said is, by itself, knowledge of the word. Hearing a word once does not mean you understood it, and saying it once does not mean you can use it again on demand; knowing a word builds up over repeated meetings and uses. The counts are a record of exposure and production in your conversations, not a vocabulary test.
Read more
Can-do statements
What we measure
Whether a conversation shows evidence of a specific speaking ability, such as ordering a meal or describing something that happened to you. The statements are Talio's own, written in the style of the CEFR's can-do descriptors and arranged by CEFR level.
From what signal
The tutor's and your transcripts, read together for each statement in scope for the scenario. Each conversation gives one verdict per statement: demonstrated, not demonstrated (a fair chance was there and was not taken), or no clear opportunity when the conversation never called for it. Across conversations those verdicts roll up into a tier: Demonstrated, Emerging (shown at least once), Attempted (tried, not yet shown), or No evidence yet. Demonstrated takes repeated evidence across conversations, not a single success.
What it is not
A statement marked demonstrated means the ability was shown in a conversation, not mastered. It is evidence from a conversation, not a placement, and a scenario's level caps what can be demonstrated in it: a scenario written at A2 cannot show a B2 ability, however well you spoke. Showing the same ability across several scenarios is stronger evidence than showing it once.
Read more
References
- Council of Europe (2020). Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Companion Volume. Strasbourg: Council of Europe Publishing.
- de Jong, N. H., Steinel, M. P., Florijn, A., Schoonen, R., & Hulstijn, J. H. (2013). Linguistic skills and speaking fluency in a second language. Applied Psycholinguistics, 34(5), 893–916.
- Koizumi, R., & In'nami, Y. (2012). Effects of text length on lexical diversity measures: Using short texts with less than 200 tokens. System, 40(4), 554–564.
- Levis, J. M. (2005). Changing contexts and shifting paradigms in pronunciation teaching. TESOL Quarterly, 39(3), 369–377.
- McCarthy, P. M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42(2), 381–392.
- Munro, M. J., & Derwing, T. M. (2001). Modeling perceptions of the accentedness and comprehensibility of L2 speech: The role of speaking rate. Studies in Second Language Acquisition, 23(4), 451–468.
- Munro, M. J., & Derwing, T. M. (2006). The functional load principle in ESL pronunciation instruction: An exploratory study. System, 34(4), 520–531.
- Webb, S. (2007). The effects of repetition on vocabulary knowledge. Applied Linguistics, 28(1), 46–65.