OCR & Document Extraction

A Fine-Tuned OCR Pipeline for Typed and Handwritten Documents, Served as a Governed API

How Pfactorial Technologies fine-tuned and validated a custom Tesseract OCR model for typed and handwritten text, and served it through a token-metered production API.

August 21, 2026

OUTLINE

ENGAGEMENT SNAPSHOT

Figure 1 - Key figures from this engagement, at a glance.

EXECUTIVE SUMMARY

Our client needed OCR that worked reliably on both normal typed PDFs and handwritten documents, not just clean printed text - and needed it exposed as a governed API surface with authentication and per-request quotas, not a one-off script.

An off-the-shelf OCR model doesn't clear that bar: generic accuracy on the client's specific font mix, and especially on handwriting, isn't good enough without fine-tuning, which means assembling and annotating a bespoke ground-truth dataset for two very different data types, running a real training job against it, and then proving the fine-tuned model actually beats the base model rather than assuming it does.

Pfactorial built a fine-tuning and validation pipeline on top of Tesseract - a purpose-built ground-truth generator for typed PDFs, a manually annotated pipeline for handwritten line images from the IAM dataset, an edit-distance validation harness comparing custom against base model performance, and a token-gated Flask API for custom OCR, handwritten OCR, image-to-PDF conversion, and text prettification.

Why this engagement is representative This engagement demonstrates Pfactorial's ability to take an open-source OCR engine from generic accuracy to task-specific accuracy - building the ground-truth data pipeline, the fine-tuning run, and a quantitative validation harness, not just calling an existing model.

THE CHALLENGE

Fine-tuning Tesseract for real production use meant solving data, training, and validation problems that don't show up when you just call the base model.

1. Typed and handwritten text need entirely different ground-truth pipelines

Typed PDFs can be auto-segmented into line images and text pairs by code; handwritten line images from the IAM dataset have to be manually annotated, so the two data-collection paths share almost nothing but their output format.

2. A fine-tuned model is worthless without proof it actually improved

The validation harness has to score the base model and the custom model against the same held-out data using a real metric - edit distance - rather than assuming a fine-tuning run helped.

3. Line segmentation on handwritten pages is a preprocessing problem in its own right

Identifying true line breaks from whitespace, thresholding, and filtering out near-empty rows has to happen before any annotation can start, or the ground-truth data itself is unreliable.

4. OCR has to be served as a metered, authenticated API, not a notebook

Requests carry a token keyed to user ID and a specific request type - OCR, PDF conversion, or prettify - with a remaining-count check, so the same pipeline has to work as a governed production endpoint, not just a training script.

The real brief Not "fine-tune Tesseract once" but "build the data pipeline, the training run, and the proof that it worked, then serve it as a governed API."

THE SOLUTION

Pfactorial built an end-to-end OCR fine-tuning pipeline - separate ground-truth generation for typed and handwritten text, a Tesseract fine-tuning run, an edit-distance validation harness, and a token-authenticated Flask API in front of the resulting model.

Figure 1 - Ground truth and training run offline; validation proves the improvement before the model is ever served.

Architectural principles

Two ground-truth pipelines, one training path - Typed and handwritten data are collected completely differently - one auto-segmented, one manually annotated - but both land in the same {model_name}-ground-truth format so a single tesstrain fine-tuning path can consume either.
Validate against a real metric, not intuition - Edit distance between predictions and ground truth is computed for both the base and the custom model on the same held-out data, so a claimed improvement is a measured number, not an assumption.
Access is governed at the token, not the endpoint - Each request token carries a user ID and a specific capability key - OCR, PDF conversion, or prettify - checked against a remaining-count balance before any processing happens.
Preprocessing quality gates the whole pipeline - Grayscale thresholding and whitespace-based line-break detection run before any handwritten sample is annotated, because a bad line segmentation corrupts every ground-truth pair built on top of it.

CAPABILITIES DELIVERED

Each capability is exposed as its own token-gated API endpoint, sitting on top of the same underlying fine-tuned model.

CAPABILITY	WHAT IT DOES
Custom OCR (typed text)	/ocr_custom extracts text from typed PDFs or images using the fine-tuned Tesseract model.
Handwritten OCR	/ocr_handwritten runs the same request pattern against handwritten line images, using a model fine-tuned on manually annotated IAM-dataset samples.
Image-to-PDF conversion	/convert2pdf converts a base64-encoded image into PDF and returns the extracted text content, with the page count tracked in the response headers.
Text prettification	/prettify post-processes raw OCR output into cleaner, more readable text via a dedicated post-processing function.
Token-based request metering	Every request is authenticated against a token carrying user ID and per-capability request counts, so usage is governed centrally rather than per endpoint.

Figure 2 - The same validated model backs every endpoint, from raw OCR to prettified output.

Design note Keeping the base eng.traineddata model as the permanent validation baseline - not just a starting checkpoint discarded after fine-tuning - is what turns this into a measurable improvement rather than a fine-tuning run taken on faith; every future retrain is judged against the same reference point.

ENGINEERING FOR SCALE AND RELIABILITY

Five decisions kept the pipeline reproducible and the API defensible in production.

Edit distance as the validation metric, not accuracy alone

calculate_edit_distance() scores both the base and the custom model against the same held-out sequences, converted to sparse tensors for efficient batch scoring, giving a single comparable number for whether fine-tuning helped.

Separate ground-truth pipelines instead of forcing one process on two data types

Typed PDFs are segmented and labeled entirely by code; handwritten IAM lines go through grayscale thresholding, line-break detection, and manual annotation - forcing one pipeline onto both would have degraded whichever data type it wasn't designed for.

Token-based authorization with per-capability request counts

A single decoded token carries user ID and a key for OCR, PDF conversion, or prettify, checked against a remaining-count balance before any processing - usage governance lives at the auth layer, not scattered per endpoint.

tesstrain as the fine-tuning framework, not a from-scratch model

Fine-tuning eng.traineddata via the existing tesstrain Makefile pipeline gets a production-grade OCR base without training a recognition model from zero.

Padding sequences to a fixed max length before scoring

Ground truth, base-model, and custom-model predictions are all padded to the same max sequence length and converted to sparse tensors, so edit-distance comparisons are computed consistently across variable-length text lines.

DELIVERY APPROACH

The pipeline was built in the order data has to flow - ground truth first, then training, then proof, then the API in front of it.

1. Typed-text ground truth - building the PDF-to-line-image-and-text pipeline that auto-segments typed PDFs into ground-truth pairs.

2. Handwritten ground truth - preprocessing IAM-dataset page images (grayscale, thresholding, line-break detection) and manually annotating the resulting line images.

3. Model fine-tuning - cloning tesstrain, preparing language data, and running the fine-tuning job against the English base model for 10,000 iterations.

4. Validation harness - building the edit-distance scoring pipeline that compares base-model and custom-model predictions against held-out ground truth.

5. API layer - shipping the token-authenticated Flask endpoints - /ocr_custom, /ocr_handwritten, /convert2pdf, /prettify - in front of the validated model.

RESULTS AND IMPACT

- Key outcomes from this engagement.

The fine-tuned model was validated against the un-tuned base model using edit distance on held-out data, with the fine-tuning run itself completing in 5.5 hours over roughly 41,238 training samples at 10,000 iterations - giving the client a measured before/after rather than an assumed improvement.

Because typed and handwritten OCR, PDF conversion, and text prettification are all served from one token-authenticated Flask API with per-capability request counts, the client can expose OCR as a metered product capability rather than an internal script, with resource usage - 1.7 to 2.3 GB RAM and 40 to 60% CPU during training - already characterized for capacity planning.

What it enabled commercially

The client can now offer OCR - including handwritten documents most off-the-shelf services handle poorly - as a governed, usage-metered API product, with a repeatable pipeline for retraining and re-validating the model as new document types arrive.

WHY PFACTORIAL

This engagement reflects Pfactorial's applied machine-learning engineering practice - building the full path from ground-truth data to a validated, production-served model, rather than treating fine-tuning as a one-off notebook exercise.

- Service lines this engagement draws on.

Engagement enquiries Pfactorial Technologies works with organisations that need document intelligence - OCR, extraction, or classification - that's measurably better than an off-the-shelf model on their specific data, and served as a governed API. If you're evaluating a document-processing pipeline, we're happy to talk through what fine-tuning and validation would realistically take. · pfactorial.ai

APPENDIX A - TECHNOLOGY STACK

The technology stack underpinning the system, grouped by the layer it serves.

Result and Analysis

ENGAGEMENT SNAPSHOT

How Pfactorial Technologies fine-tuned and validated a custom Tesseract OCR model for typed and handwritten text, and served it through a token-metered production API.