Old OCR text cripples language model training, and FineBooks wants to fix that at scale
A new FineBooks project from Hugging Face and EleutherAI evaluated 14 OCR models on 2,000 historical book pages, finding the top model achieves 97.6% character accuracy at low cost.
- FineBooks, a joint project by Hugging Face and EleutherAI, evaluated 14 OCR models on 2,000+ historical book pages to assess their suitability for AI training data.
- The top model, dots.mocr, achieved 97.6% character accuracy at under $2 per 1,000 pages, making it cost-effective for large-scale AI training but not yet for scholarly use.
- Poor OCR quality in historical texts introduces errors that can degrade AI model performance, highlighting a critical data quality issue.
- The project aims to improve OCR tools for historical texts, addressing challenges like archaic fonts and faded ink that modern OCR systems struggle with.
Researchers from Hugging Face and EleutherAI have launched FineBooks, a project aimed at addressing a critical bottleneck in AI training data: the poor quality of digitized historical texts. The team tested 14 open-source OCR models on over 2,000 historical book pages, revealing significant variability in accuracy. The top-performing model, dots.mocr, achieved 97.6% character accuracy at a cost of under two dollars per thousand pages, making it viable for large-scale AI training but still insufficient for precise scholarly transcriptions.
The findings highlight how legacy OCR errors, such as misread characters or formatting issues, can propagate into AI datasets, degrading model performance. FineBooks proposes a scalable solution by benchmarking and improving OCR tools specifically for historical texts, where modern OCR systems often struggle due to archaic fonts, faded ink, or complex layouts. The project’s goal is to make high-quality digitized text accessible for AI training without the prohibitive costs of manual transcription.
While the current top model is a step forward, the team emphasizes that further refinement is needed before it meets the rigorous standards required for academic or archival use. The project also underscores the broader challenge of data quality in AI, where even minor errors in training data can have outsized impacts on model reliability.
Provides a benchmarked, cost-effective OCR solution for improving AI training data quality, especially for historical texts.
Offers a scalable way to enhance data pipelines for AI applications relying on digitized historical content.
Illustrates the importance of data quality in AI and the challenges of digitizing historical documents.
Highlights how legacy OCR flaws can impact AI systems and the need for better digitization tools.
- OCR
- Optical Character Recognition, technology that converts different types of documents into editable and searchable data.
- FineBooks
- A joint project by Hugging Face and EleutherAI to improve OCR quality for historical texts used in AI training.
AI ResearchAI professors are negotiating the new realities of academic research
With a feel for physics, AI models simulate a wider range of real-world scenarios - news.mit.edu
Artificial Intelligence in Dermoscopy: Why Expert Oversight Still Matters - Medscape
AI ResearchAnthropic Claude Riemann Hypothesis Claim Highlights the Need for Verifiable AI Math Research
AI ResearchMeta’s new Glimmer AI model offers a hint at Zuckerberg’s personal intelligence vision
NCCU opens nation’s first HBCU artificial intelligence insti... - QCity Metro
North Carolina Central University has opened the first artificial intelligence institute at an HBCU, aiming to advance AI education and research for underrepresented groups.
'AI is for everyone:' NC Central opens artificial intelligence institute - WUNC News
North Carolina Central University has opened a new artificial intelligence institute aimed at making AI education and research more accessible to all students.
Artificial Intelligence: Moving up the Investment Value Chain - ai-cio.com
Institutional investors are moving beyond pilot projects to full-scale AI allocations, reshaping the investment landscape.
SecurityOpenAI launches GPT-5.6-Cyber to help defenders find vulnerabilities before attackers do
OpenAI released GPT-5.6-Cyber, a specialized model for cybersecurity that resolves 98.5% of blocked security queries and has already identified two new Chrome vulnerabilities.
Where and How Artificial Intelligence Is Coming to Surfing - SURFER Magazine
AI is being integrated into surfing through tools that predict waves and analyze performance, with startups and surf schools adopting the technology.
How artificial intelligence is changing periodontal assessment: Enhancing diagnostic accuracy in dental hygiene practice - rdhmag.com
AI tools are now being integrated into dental hygiene workflows to improve the accuracy of periodontal disease detection and assessment.