What Lexical Diversity Actually Measures (and When to Ignore It)

Lexical diversity is the ratio of different words to total words. Writing that reuses the same vocabulary scores low; writing that draws on a wider range scores high. That is the whole idea, and it is genuinely useful - repetitive vocabulary is one of the most common reasons a draft feels flat.

It is also one of the easiest metrics to misread, because the simplest version of it is broken in a way that is not obvious.

The type-token ratio, and why it lies

The basic measure is the type-token ratio (TTR): the number of unique words (types) divided by the total number of words (tokens). A passage of 100 words using 70 distinct words has a TTR of 0.70.

The problem is that TTR falls as documents get longer, always, regardless of how well they are written.

The reason is structural. English requires function words - the, of, and, to, is, that - and you cannot avoid repeating them. In the first 100 words, most of your content words are new. By word 2,000, you have used the perhaps 120 times, and each new sentence adds far more tokens than types. A well-written 5,000-word essay will score lower than a mediocre 300-word paragraph, every time.

So a raw TTR is only meaningful between texts of near-identical length. Compare a 400-word draft to a 4,000-word one and the number tells you about the lengths, not the writing.

The measures that fix it

Two length-robust alternatives do most of the real work.

MATTR (moving-average type-token ratio) slides a fixed window - typically 50 or 100 words - across the text, computes TTR in each window, and averages the results. Because every window is the same size, the length bias disappears. A 400-word text and a 4,000-word text can be compared directly.

MTLD (measure of textual lexical diversity) works differently: it reads forward until the running TTR drops below a threshold, records how many words that took, resets, and repeats. The score is the average length of those runs. Text that keeps introducing new vocabulary sustains a high TTR for longer, so its runs are longer. MTLD is generally the more sensitive of the two, and the harder to explain.

Both give you a number that means the same thing at any length, which is the only way a diversity score is useful for revision - you want to know whether this draft improved on the last draft, and those are rarely the same length.

You will also see hapax legomena (the proportion of words appearing exactly once) and Yule's K (a repetition measure designed to be length-independent). They correlate with the others and are mostly useful as cross-checks.

What a low score is actually telling you

Here is the part worth internalising: a low diversity score is rarely a vocabulary problem, and the instinctive fix makes it worse.

The instinct is to reach for a thesaurus and replace repeated words. That produces prose full of near-synonyms that do not quite mean what you meant, and readers notice immediately - it is the single most recognisable signature of over-edited writing.

In practice, a low score is usually caused by one of these:

• Repeated sentence openers. Six consecutive sentences starting This, The results, or It is will tank a diversity score and, more importantly, make a paragraph exhausting to read.
• Transition crutches. However, Moreover, Furthermore, Additionally at the head of every paragraph. Each is fine; a pattern of them is a tic.
• Uniform sentence length. Not a vocabulary measure at all, but it correlates strongly with feeling repetitive, because rhythm is doing as much work as word choice.
• Genuinely repeating a key term - which is often correct and should be left alone.

The fix for the first three is structural: vary how sentences begin, cut transitions that are not doing logical work, and let sentence lengths differ. None of that requires a single new word. We wrote about this at length in why your writing sounds repetitive.

When the score should be ignored

Technical and academic writing is supposed to repeat terms. If your paper is about mitochondrial density, the phrase mitochondrial density appears many times, and every substitution makes it worse. Consistent terminology is a virtue in technical prose; varying it introduces ambiguity about whether you mean the same thing. A methods section should score badly and that is correct.

Short texts give unstable numbers. Under roughly 100 words there is not enough material for any of these measures to be reliable, including the length-robust ones. Treat scores on an abstract as noise.

Quoted material distorts the result. A passage that is 30% block quotation is measuring someone else's vocabulary alongside yours.

Lists and data are not prose. A page of tabulated results will produce a score that means nothing about your writing.

Using it well

The number is most useful as a relative signal. Run a draft, revise it, run it again, and the direction of change tells you something real. Chasing an absolute target - "I need to be above 0.7" - produces worse writing, because the fastest way to raise the number is to introduce words you do not mean.

The more useful output is not the score at all but the specific findings: which words you have leaned on, which openers repeat, where sentence lengths flatten out. Those point at particular sentences you can fix. A single number points at nothing.

That is why our Lexical Diversity tool highlights the actual offenders in your text rather than just scoring it - click any finding and it shows you where the problem is, which is the only form in which this information is actionable. You can also check length, reading level and sentence statistics with the Word Counter, which is often enough to spot the uniform-sentence-length problem on its own.