Language identification from character frequency works on a simple principle — every written language has a characteristic statistical fingerprint of which characters appear most often — and this fingerprint is reliable enough that modern language detection libraries can correctly identify a language from as few as 20-30 characters with high confidence
The previous articles on this site covered Shannon entropy and information theory, Zipf's law and NLP foundations, frequency analysis and cipher breaking, Huffman coding, and language identification from character frequency. This article addresses character frequency in forensic linguistics and authorship analysis — the specific statistical techniques used to identify authorship, detect plagiarism, and distinguish human-written from machine-generated text based on stylometric patterns.
Stylometry: authorship identification from writing statistics
Stylometry is the study of linguistic style — specifically the quantitative analysis of writing patterns to attribute texts to authors, identify forgeries, or distinguish between writers. Character frequency is one component of a stylometric fingerprint.
The Federalist Papers attribution: the most famous stylometry case. Of the 85 Federalist Papers (1787-1788), 51 were known to be written by Alexander Hamilton, 29 by James Madison, and 5 disputed. In 1964, Mosteller and Wallace applied statistical analysis to word frequency patterns — finding that function word usage (the, of, while, upon) differed systematically between Hamilton and Madison. Their conclusion (Madison authored the disputed papers) is now the academic consensus.
Function words vs content words: stylometry relies primarily on function words (prepositions, articles, conjunctions: the, a, in, of, to, and) rather than content words (nouns, verbs, adjectives). Content words are determined by what you're writing about; function words are largely unconscious stylistic choices that remain consistent regardless of topic. An author who habitually uses "upon" instead of "on" does so in academic papers and casual emails alike.
Character n-grams: going below the word level
Character n-grams — sequences of n consecutive characters — provide a finer-grained stylometric signal than word frequencies:
2-grams (bigrams): "th", "he", "in", "er", "an" — the most common English character bigrams 3-grams (trigrams): "the", "ing", "ion", "tio" — the most common English trigrams 4-grams: "tion", "ing ", " the" — still informative, higher specificity
Character n-gram language models have several advantages over word-level models for authorship analysis:
- Handle out-of-vocabulary words naturally (unknown words still produce known character sequences)
- Capture morphological patterns (suffixes like "-tion", "-ing" are characteristic of English)
- Robust to spelling variations and typos
- Useful for short texts where word-level models lack sufficient data
The forensic application: comparing character n-gram distributions between a known text and an unknown text. If the cosine similarity between their bigram frequency vectors exceeds a threshold, the texts are more likely from the same author.
AI-generated text detection: what frequency analysis reveals
AI-generated text (from models like GPT-4, Claude, Gemini) has characteristic statistical patterns that differ from human writing — some of which are visible in character and word frequency distributions:
Vocabulary distribution: LLMs tend toward more even vocabulary distribution than human writers. Human writers have highly idiosyncratic word choices (favourite phrases, domain-specific terms, personal quirks); LLM output is more "averaged" — pulled toward common usage patterns in training data.
Punctuation frequency: LLMs use certain punctuation patterns differently from human writers — less frequent use of em dashes, ellipses, and sentence fragments; more consistent paragraph structure; different comma frequency.
Perplexity: "perplexity" is a measure of how "surprised" a language model is by a text. Text generated by a language model tends to have low perplexity (the model assigned high probability to each token) compared to human-written text, which is more varied and sometimes uses unexpected word choices.
The detection limitation: character frequency analysis alone is insufficient for reliable AI text detection. State-of-the-art detectors combine perplexity analysis, burstiness (humans write in "bursts" of connected ideas with more consistent entropy variation), and other statistical measures. No current publicly available detector is reliable enough for formal attribution without human review.
Byte Order Marks and encoding detection via character frequency
Character frequency has a practical use beyond linguistics — encoding detection. When a file's encoding is unknown, the distribution of byte values can identify the encoding:
UTF-8 encoded English text: high frequency of bytes in the 0x20-0x7E range (ASCII printable characters); low frequency of bytes above 0x7F (multi-byte sequences for non-ASCII characters)
UTF-8 encoded Chinese text: very high frequency of bytes in the 0xE4-0xE9 range (CJK characters in UTF-8 three-byte sequences)
UTF-16 encoded text: alternating pattern of null bytes if Latin text (e.g., "A" in UTF-16 LE is 0x41 0x00)
Latin-1 (ISO-8859-1) encoded French text: bytes in the 0xC0-0xFF range for accented characters like é, à, ç
The Byte Order Mark (BOM): UTF-8 files sometimes begin with EF BB BF (the UTF-8 BOM), UTF-16 LE files with FF FE, UTF-16 BE with FE FF. These magic bytes identify encoding without frequency analysis — but for files without BOMs, character frequency is the fallback.
How to use the Character Frequency Counter on sadiqbd.com
- For language identification: paste a text sample and examine the high-frequency characters — languages with non-Latin scripts (Arabic, Chinese, Korean) will show characters outside the Latin range; Romance languages will show accented characters; German will show ä, ö, ü, ß
- For stylometric baseline: compare character frequency distributions between two texts (paste each separately and compare the frequency tables) — significant differences in punctuation frequency, vowel-to-consonant ratio, or function word proxies can indicate different authors
- For encoding diagnosis: if a file contains unexpected characters or displays garbled text, the character frequency output reveals which byte ranges are present — helping identify encoding mismatches
Frequently Asked Questions
Can stylometry reliably identify anonymous online authors — for example, to match a Reddit post to a known blogger? With sufficient text, stylometry has demonstrated accuracy in research settings — but it's far from reliable in adversarial conditions. Studies have shown stylometric techniques achieving 80-90% accuracy in controlled conditions when comparing known authors with similar training text volumes. In practice: short texts (under 500 words) significantly reduce accuracy; authors can intentionally modify their style; and the base rate problem means false positives are high when the suspect pool is large. Courts have generally not accepted stylometry as definitive evidence (more as corroborating evidence). In practice it's used in combination with other investigative techniques rather than as standalone attribution.
Is the Character Frequency Counter free? Yes — completely free, no sign-up required.
Try the Character Frequency Counter free at sadiqbd.com — analyse character distribution in any text instantly.
