Formatted Error Rate (FER) vs Word Error Rate: Caption Quality Metrics Explained
Procurement officers and accessibility coordinators evaluating captioning vendors often see claims of "99% accuracy" or "WER under 1%." But caption quality involves more than word-level accuracy. Formatted Error Rate (FER) and Word Error Rate (WER) measure different aspects of caption quality. Understanding both metrics helps you specify the right quality requirements in RFPs and evaluate vendor performance. This guide explains the technical definitions, when each metric applies, and what to request from vendors.
Word Error Rate (WER): The Foundation
Word Error Rate (WER) is the industry-standard metric for measuring transcription accuracy. It counts word-level errors in the caption text compared to a reference (ground truth) transcript.
The Formula
WER = (Substitutions + Insertions + Deletions) / Total Words in Reference
- Substitutions: A spoken word is replaced with a different word (e.g., "their" instead of "there," "principle" instead of "principal")
- Insertions: Extra words appear in the caption that were not spoken
- Deletions: Spoken words are missing from the caption
Example
- Reference (spoken): "The professor explained the theory clearly" (6 words)
- Hypothesis (caption): "The professor explained their theory clearly" (6 words)
- Error: 1 substitution ("their" for "the")
- WER: 1/6 = 16.7% error rate, or 83.3% accuracy
WER Characteristics
- Can exceed 100%: When insertions are numerous, WER can exceed 100% because the denominator is based on reference words, not hypothesis words.
- Treats all errors equally: A substitution that changes meaning ("not" vs "now") counts the same as a harmless homophone ("their" vs "there").
- No credit for partial correctness: If "government" is transcribed as "governments," it counts as a substitution; the reader might still understand.
- Widely used in speech recognition: The US National Institute of Standards and Technology (NIST) and research communities use WER to evaluate automatic speech recognition (ASR) systems.
WER answers the question: "How many words are wrong?" It does not address punctuation, speaker labels, capitalization, or formatting.
Formatted Error Rate (FER): Beyond Words
Formatted Error Rate (FER) extends the concept of accuracy to include formatting elements that matter for professional closed captioning. Captions are not just raw text; they include:
- Punctuation (periods, commas, question marks)
- Capitalization (proper nouns, sentence beginnings)
- Speaker identification (who said what)
- Non-speech elements (e.g., [laughter], [applause], [music])
- Grammar and sentence structure
FER measures the percentage of errors when these formatting requirements are considered in addition to word accuracy.
What FER Counts as Errors
- Word errors: Same as WER (substitutions, insertions, deletions)
- Punctuation errors: Missing or incorrect punctuation
- Capitalization errors: Incorrect capitalization of names, acronyms, or sentence starts
- Speaker label errors: Wrong or missing speaker identification
- Non-speech element errors: Missing or incorrect descriptions of sounds, music, or other audio
Why FER Matters for Accessibility
Professional captioning standards (e.g., DCMP, broadcast guidelines, many RFP requirements) expect captions to be not only accurate in words but also properly formatted. Poor punctuation or missing speaker labels can reduce comprehension and accessibility, especially for viewers who rely heavily on captions.
For example:
- WER might be low (1%) if all words are correct.
- FER might be higher (3%) if speaker labels are wrong, punctuation is missing, or [laughter] and [applause] are omitted.
FER gives a more complete picture of caption quality for compliance and user experience.
WER vs FER: Quick Comparison
| Factor | WER | FER |
|---|---|---|
| What it measures | Word accuracy only | Word accuracy + formatting |
| Includes | Substitutions, insertions, deletions | All WER errors + punctuation, caps, speaker ID, non-speech |
| Typical use | ASR evaluation, research | Professional captioning, compliance |
| Can exceed 100%? | Yes (due to insertions) | Depends on implementation |
| Vendor reporting | Common | Less common; ask for it |
When to Use Each Metric
Use WER when:
- Evaluating raw ASR output before human review
- Comparing different speech recognition engines
- Benchmarking automated caption quality
- Focusing on word-level accuracy for technical or scientific content
Use FER when:
- Evaluating final caption deliverables for compliance
- Specifying RFP quality requirements for professional captioning
- Assessing human-reviewed or human-created captions
- Requiring speaker identification, punctuation, and non-speech elements
Use both when:
- You want a full quality picture: WER for word accuracy, FER for formatting
- Procurement requires transparency on both dimensions
- You are comparing vendors with different workflows (e.g., AI-only vs AI+review)
Accuracy Targets by Service Tier
Many RFPs specify minimum accuracy thresholds. Here is how WER typically maps to service tiers:
| Service Tier | WER | Accuracy | Typical FER Expectation |
|---|---|---|---|
| Automated AI | 3-5% | 95-97% | Often not measured; formatting varies |
| AI + Human Review | 1-1.5% | 98.5-99% | Should include proper formatting |
| Professional Human | 0.1-0.5% | 99.5-99.9% | Full formatting compliance |
For compliance-critical captions, both WER and FER should meet your institution's thresholds. A vendor might report 99% WER but have higher FER if formatting is neglected.
What to Specify in RFPs
When drafting RFP language for caption quality:
- Define accuracy metric: Specify WER, FER, or both. Example: "Vendor shall report Word Error Rate (WER) and Formatted Error Rate (FER) on sample content."
- Set thresholds: Example: "Captions shall meet minimum 98.5% accuracy (WER ≤ 1.5%) and 99% formatted accuracy (FER ≤ 1%) for AI+review tier."
- Require methodology: Ask how the vendor measures WER/FER (e.g., reference transcripts, sampling, tool used).
- Request regular reporting: Specify frequency of quality audits (e.g., quarterly, on sample batches).
Frequently Asked Questions
Can WER and FER be the same?
Yes. If a caption has no formatting requirements or if formatting is perfect, WER and FER can match. In practice, FER is often slightly higher than WER because it counts additional error types.
How do vendors typically report these metrics?
Many vendors report WER or a generic "accuracy" percentage. FER is less commonly reported. Ask vendors explicitly for both metrics and their methodology.
Does Recap Innovations report WER and FER?
Recap Innovations uses quality assurance processes designed to meet RFP accuracy requirements. We provide transparency on our accuracy methodology and can discuss WER and FER reporting for your procurement needs. Every transcript generated on our platform can be tested for WER directly with a 100% accurate ground-truth transcript supplied by your team. Contact us to discuss quality metrics and sampling approaches.
What is a "good" WER or FER?
For ADA Title II and similar compliance, many institutions require 98.5%+ accuracy (WER ≤ 1.5%). For FER, 99% formatted accuracy is a common target for professional captioning. Requirements vary by content type and risk level.
How is reference (ground truth) established for WER?
WER requires a reference transcript against which the caption is compared. The reference is typically a human-verified transcript. For live captioning, the reference may be created after the event.
Get Expert Support for Quality Requirements
Recap Innovations offers AI-powered captioning with human review, custom dictionaries, and quality assurance processes aligned with institutional and RFP requirements. We help procurement teams define and meet accuracy targets.
Schedule a demo to discuss quality metrics, sampling, and reporting. Or explore our AI vs human transcription buyer's guide and RFP evaluation criteria for more on vendor selection.