What Is Handwritten Text Recognition and Why Historical Archives Still Need Human Review

Asit Dubey

Handwritten text recognition (HTR) is a machine learning technology that converts images of handwritten documents into searchable, editable text. Traditional OCR also uses neural networks today, but it is trained on printed fonts that generalize well across documents. HTR uses neural networks trained on transcribed handwriting, which varies so much between writers that models usually need examples from the collection itself. It has become a core part of modern content digitization, especially for registers, letters, court records, and manuscripts that were never typed.

HTR has improved dramatically, and it can now make collections searchable that would have taken decades to transcribe by hand. But the accuracy numbers quoted for modern benchmarks rarely match what archives see on real historical material, where many writers, faded ink, old abbreviations, and damaged pages are the norm. That gap is why human review remains essential, not as a replacement for HTR, but as the part of the workflow that makes its output trustworthy.

This guide explains what HTR is and what it makes possible, how it differs from OCR, how accurate it is on benchmarks and real archives, where large vision-language models fit, what error rates are good enough, how a human-in-the-loop workflow works, how effort compares with manual transcription, and what to ask a vendor.

Key Takeaways

• HTR reads handwriting with neural networks trained on transcribed examples: It works at line level and learns specific hands or scripts, which is why training data matters so much for accuracy.
• Benchmark accuracy and archive accuracy are very different: Modern English benchmarks reach character error rates below 3 percent, while in published case studies multi-writer collections have ranged from about 12 to 24 percent.
• Error rate determines use: Below about 5 percent supports transcription with light proofreading, while above about 10 percent correction can take longer than transcribing manually, although search can still work.
• Large AI models raise the stakes for review: Vision-language models and AI post-correction can produce fluent text that is not on the page, so human checks against the original image matter more, not less.
• Quality must be measured per collection: Character error rate, word error rate, and field accuracy on sampled pages show whether a model is fit for its intended use.

What Is Handwritten Text Recognition?

HTR systems read images of handwriting and produce text. Modern systems first run layout analysis to find text regions and lines on a page, then pass each line to a neural network that predicts the most likely sequence of characters. The network learns from ground truth: page images paired with accurate human transcriptions.

Because handwriting varies so much between writers, HTR models are usually trained or fine-tuned for a particular hand, script, period, or collection. A model trained on nineteenth-century English clerks will not automatically read medieval Latin or twentieth-century German shorthand, which is why archives often build several models for different series.

What Does HTR Make Possible for Archives?

• Full-text search across handwritten collections: Researchers can find a name or phrase across thousands of pages instead of relying on finding aids or reading volumes page by page.
• Entity extraction and linking: Names, places, and dates pulled from records can be linked across series, making it possible to follow people and events through an archive.
• Faster transcription projects: Correcting a good automated transcript is quicker than transcribing from scratch, which lets volunteers and staff cover far more material.
• Keyword spotting at scale: Even where transcripts are too rough to publish, search can still locate relevant pages, opening collections that would otherwise stay unread.

How Does HTR Differ From OCR?

The real difference is not neural versus non-neural. Widely used OCR engines moved to neural models years ago: the Tesseract documentation notes that version 4.0 added an engine based on LSTM neural networks. The difference is training needs and input variability. Printed fonts are consistent enough for general models to transfer across documents, while handwriting changes with every writer.

Even printed historical material can be harder than it looks. In a study of digitized historic newspapers, Holley (2009) describes 98 percent character accuracy as the threshold practitioners treat as good, yet found that raw OCR accuracy at the National Library of Australia ranged from 71 to 98.02 percent across a representative sample. Condition and typography affect recognition before handwriting is even involved.

How Accurate Is HTR Today?

Accuracy on Modern Benchmarks

HTR accuracy is usually reported as character error rate (CER), the percentage of characters the system gets wrong. On modern benchmarks, results are strong. Li et al. (2023), at Microsoft, introduced TrOCR, a transformer-based recognition model, and reported a CER of 2.89 percent on the IAM handwriting database, a widely used benchmark of modern English handwriting.

Accuracy on Real Historical Archives

Historical collections are much harder. Muehlberger et al. (2019), reviewing the use of HTR across European archives, report that in the best cases models produce handwritten transcripts with a CER below 5 percent, and printed material can reach 1 to 2 percent. But real projects varied widely. Passau Diocesan Archives achieved a CER of 17 to 19 percent on 1,200 images of nineteenth-century death registers written by 40 different scribes. The National Archives of Finland reached around 12 percent on court records using 75,000 words of training data, but around 24 percent on estate inventories written by many hands.

These figures come from a handful of published case studies rather than a broad survey, so they illustrate a pattern rather than set an expected range. The lesson is not that HTR fails on archives. It is that accuracy depends heavily on the number of writers, the amount of training data, and the condition of the originals, so every collection needs its own measurement.

Where Do Large Vision-Language Models Fit?

Since those case studies, the field has moved quickly. HTR platforms now offer large pre-trained models built from many hands and scripts, which reduce, but do not remove, the need for collection-specific training. Multimodal large language models, which read an image and generate text in one step, can also transcribe handwriting without any task-specific training.

Recent research shows both the promise and the limits. Crosilla, Klic, and Colavizza (2025), in a study published in the Journal of Documentation, benchmarked proprietary and open-source multimodal models against Transkribus models, including its newer general supermodel, on modern and historical datasets in English, French, German, and Italian. They found strong performance on English and modern handwriting, weaker performance on other languages, no significant ability to correct their own output, and inconsistent results against Transkribus on historical documents.

AI Post-Correction

Another approach is to run traditional HTR or OCR first, then ask a large language model to fix the errors. The evidence so far calls for caution. Boros et al. (2024) tested fourteen language models across eight historical datasets and found that LLM-based post-correction performed very poorly overall, considerably degrading the quality of the original transcriptions across all models, prompts, and datasets.

None of this means archives should ignore these tools. Large models are improving fast and can be useful for drafts, difficult layouts, and modern material. It means their output needs the same, or stricter, measurement and review as any other recognition method.

What Error Rate Is Good Enough?

The right target depends on how the output will be used. The thresholds of 5, 10, and about 30 percent come from the guidance in Muehlberger et al. (2019), and the descriptions reflect how teams typically use output within each band.

This matters for planning. A collection that only needs to be searchable can accept much higher error rates than one being published as a scholarly edition or used as evidence in a legal or administrative process.

Why Do Historical Archives Still Need Human Review?

Many Writers in One Collection

Registers, ledgers, and case files were often written by dozens of clerks over many years. Each new hand increases error rates, as the Passau and Finland examples show, and models rarely perform evenly across all of them.

Old Scripts, Abbreviations, and Spelling

Historical writing uses abbreviations, ligatures, archaic letterforms, and inconsistent spelling. A model can transcribe characters correctly and still produce text that misreads an abbreviation or a place name, which only an informed reviewer will catch.

Damaged and Faded Originals

Faded ink, bleed-through, stains, and torn pages create errors that no amount of model training fully removes. Reviewers can compare the image with the output and correct what the model could not see.

Tables, Forms, and Marginal Notes

Many archival records are laid out in tables and forms, and layout analysis on these remains difficult. If columns are misread, every value in a row can end up attached to the wrong person or date.

Errors Look Plausible

HTR errors are often fluent. A misread surname, date, or number looks like a real value, so it passes unnoticed unless someone checks it. Large generative models make this risk sharper: because they produce text rather than simply recognizing characters, they can output fluent words, names, or dates that do not appear on the page at all, or replace historical terms with modern ones. For records used in genealogy, land claims, or legal processes, a plausible error can be worse than an obvious one.

How Does a Human-in-the-Loop HTR Workflow Work?

1. Capture images to standard: High-quality scans at consistent resolution give the model the best chance, because poor images raise error rates before recognition even starts.
2. Run layout analysis: Detect text regions, lines, tables, and marginal notes, and correct the segmentation on complex pages, since recognition depends on accurate line detection.
3. Create ground truth: Transcribe a representative sample of pages accurately. Muehlberger et al. note that around 15,000 transcribed words, or about 75 pages, is generally enough to train a model for one hand, with more needed for many writers.
4. Train or fine-tune the model: Train on the ground truth, hold back a test set, and measure CER on pages the model has not seen, so accuracy estimates are honest.
5. Recognize and score confidence: Run the model across the collection and record a confidence score for each line or word, which makes it possible to target review effort.
6. Route low-confidence content to reviewers: Send uncertain lines, key fields, and complex layouts to trained reviewers, while high-confidence text passes through with sample checks.
7. Feed corrections back into training: Use reviewed pages as new ground truth and retrain periodically, so accuracy improves as the project progresses.

How Does Effort Compare With Manual Transcription?

The business case for HTR rests on how much human effort each approach needs per page. The comparison below is qualitative, because effort varies widely with the script, the condition of the pages, and the quality bar.

The main trade-off sits around the 10 percent error threshold. Below it, correcting HTR output is usually faster than typing from scratch. Above it, correction can take longer than manual transcription, so either the model needs more training data or the workflow should shift toward search and targeted review rather than full correction.

Where Should Human Review Focus?

• Names, dates, and places: These are the fields users search for most and the ones where errors cause the most harm, so they deserve the closest review in every collection.
• Numbers and amounts: Ledgers, land records, and financial documents depend on exact figures, and a single misread digit can change the meaning of a record entirely.
• Low-confidence lines: Model confidence scores show where the system is unsure, which lets reviewers spend their time where it adds the most value.
• Tables and forms: Reviewers should confirm that values are attached to the right rows and columns, since layout errors can corrupt whole records rather than single characters.
• Abbreviations and specialist terms: Legal, ecclesiastical, and administrative records use conventions that need domain knowledge to interpret correctly and consistently, so reviewers with relevant experience catch errors that general reviewers would miss.

How Do You Measure HTR Quality?

Three measures cover most needs. Character error rate shows overall recognition accuracy. Word error rate shows how readable and searchable the text is, since one wrong character can make a whole word unsearchable. Field accuracy measures whether key values, such as names and dates, are correct, which is often what matters most to users.

Quality should be checked through sample-based review with acceptance levels agreed in advance. For example, a program might require a CER below 5 percent for transcripts intended for publication and 99 percent accuracy on indexed names and dates, with batches that miss the target reworked and resampled.

What Are the Risk and Compliance Considerations?

When transcribed records support genealogy, land claims, benefits decisions, or legal proceedings, an error is not just an inconvenience. It can affect people’s rights and expose the institution to complaints or liability. Institutional buyers should treat transcription accuracy as something to document, not assume. Requirements vary by jurisdiction and use, so confirm specifics with your records officer or legal counsel.

• Document accuracy per collection: Record the measured error rates, sample sizes, and acceptance levels for each batch, so the institution can show how reliable a transcript is if it is ever questioned.
• Keep an audit trail: Log which model version produced each transcript, which pages were reviewed, who reviewed them, and what was changed, so any record can be traced back to its source.
• Label machine-generated text: Make clear to users which transcripts are automated, which are reviewed, and which are verified, and always provide access to the original image as the authoritative record.
• Restrict use of unreviewed output: Avoid relying on unreviewed transcripts for decisions about individuals, and route records used in legal or administrative processes through full human verification.

What Should You Ask an HTR Vendor?

1. How will the pilot be designed: Ask which pages will be sampled, how many writers and conditions they cover, and how the vendor will measure results on pages the model has not seen during training.
2. What are the acceptance criteria: Agree on target character error rates and field accuracy levels in advance, along with how batches are sampled and what happens when a batch misses the target.
3. How is our data secured: Ask where images and transcripts are stored, who can access them, whether data is encrypted, and whether any of it is used to train models for other clients.
4. Who owns the trained models and ground truth: Confirm that ground truth transcriptions and any models trained on your collections belong to you, and that you can take them with you if the engagement ends.
5. Are large language models used, and how: Ask whether generative models are used for recognition or correction, how their output is checked against the original image, and how hallucinated text is detected.

What Are the Most Common Mistakes?

• Assuming benchmark accuracy applies to your archive: Results on modern handwriting rarely transfer to multi-writer historical material, so every collection needs its own pilot and measurement.
• Skipping ground truth: Without enough accurately transcribed training pages, models perform poorly, and the project spends more time correcting output than it saves.
• Reviewing everything equally: Full manual review of every line wastes budget that would be better spent on key fields, low-confidence text, and difficult layouts.
• Publishing unreviewed transcripts: Fluent but wrong names and dates can mislead researchers and damage trust in the collection if they are presented as accurate.
• Ignoring layout: Recognition can be strong while table and column errors scramble the meaning of records, so layout needs its own quality checks.

How Digital Divide Data Can Help

Digital Divide Data (DDD) has delivered content digitization for 25 years. Our teams combine HTR with trained human reviewers, so institutions get the speed of automation and the reliability that historical records require.

Through our content digitization services, we capture images to standard, create ground truth, and train collection-specific models. Our OCR and document conversion teams then review names, dates, numbers, and low-confidence lines against the originals, with sample-based quality checks and documented accuracy for every batch.

A good first step is a sampled pilot on one representative series, which shows the error rate you can expect and how much review it needs. Talk to an expert to plan one.

Conclusion

Handwritten text recognition has made it realistic to search and transcribe handwritten archives at scale, and large vision-language models are pushing the field further. But accuracy on real historical material varies widely with the number of writers, the training data, and the condition of the originals, and both traditional and generative models can produce errors that look plausible. Human review, focused on the fields and lines that matter most, is what turns HTR output into records people can trust. The practical next step is to pilot HTR on one representative series, measure character error rate and field accuracy, and use the results to decide how much review the collection needs.

References

Boros, E., Ehrmann, M., Romanello, M., Najem-Meyer, S., & Kaplan, F. (2024). Post-correction of historical text transcripts with large language models: An exploratory study. Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024), 133–159. Association for Computational Linguistics. https://aclanthology.org/2024.latechclfl-1.14/

Crosilla, G., Klic, L., & Colavizza, G. (2025). Benchmarking large language models for handwritten text recognition. arXiv. https://doi.org/10.48550/arXiv.2503.15195

Holley, R. (2009). How good can it get? Analysing and improving OCR accuracy in large scale historic newspaper digitisation programs. D-Lib Magazine, 15(3/4). https://doi.org/10.1045/march2009-holley

Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C., Li, Z., & Wei, F. (2023). TrOCR: Transformer-based optical character recognition with pre-trained models. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11), 13094–13102. https://doi.org/10.1609/aaai.v37i11.26538

Muehlberger, G., Seaward, L., Terras, M., Ares Oliveira, S., Bosch, V., Bryan, M., Colutto, S., Déjean, H., Diem, M., Fiel, S., Gatos, B., Greinoecker, A., Grüning, T., Hackl, G., Haukkovaara, V., Heyer, G., Hirvonen, L., Hodel, T., Jokinen, M., . . . Zagoris, K. (2019). Transforming scholarship in the archives through handwritten text recognition: Transkribus as a case study. Journal of Documentation, 75(5), 954–976. https://doi.org/10.1108/JD-07-2018-0114

Tesseract OCR. (n.d.). Tesseract user manual. https://tesseract-ocr.github.io/tessdoc/

Frequently Asked Questions

What is handwritten text recognition?

HTR uses neural networks trained on transcribed examples to convert images of handwriting into searchable text, usually reading one line at a time.

Is HTR the same as OCR?

No. Both use neural networks, but OCR is trained on printed fonts that generalize well, while HTR learns handwriting and usually needs collection-specific training.

How accurate is HTR on historical documents?

It varies widely. Best cases reach below 5 percent character error, while published case studies of multi-writer collections have ranged from about 12 to 24 percent.

How much training data does an HTR model need?

Around 15,000 transcribed words, roughly 75 pages, is generally enough for one hand. Collections with many writers need more.

Can large language models replace HTR review?

No. Research shows they perform well on modern English but unevenly on historical material, and AI post-correction can degrade transcripts.

Which parts of an HTR transcript need human review?

Names, dates, places, numbers, tables, and low-confidence lines, because errors there matter most and look plausible.

asit dubey

Asit Dubey is a global operations leader with almost 30 years of experience across digitization, publishing, AI/ML, and LegalTech, currently serving as Executive Vice President at Digital Divide Data. He has led large-scale operations (3,500+ workforce) across APAC, EMEA, and North America, driving AI-led transformation and process excellence. A Six Sigma Black Belt, he specializes in automation, solutioning, and cost optimization, delivering productivity gains of over 300% and significant margin improvements. He has successfully scaled revenues from $750K to $3M+ monthly while turning around underperforming units. His expertise spans global delivery setup, GTM strategy, and client engagement. He is known for building resilient, multi-geo delivery models and enabling organizations to transition to AI-powered services.