Intelligent Document Processing System

The challenge

Document intake was the rate-limiting step in a high-volume legal workflow. Manual classification averaged 68% accuracy under volume pressure, creating compliance backlogs. The client needed a system that could scale to 3× volume within 18 months without proportional headcount growth.

Objectives

• Achieve classification accuracy above 90% at production volume
• Scale to 3× document throughput without headcount growth
• Maintain a compliance-grade audit trail for every classification decision
• Preserve human oversight for edge cases and low-confidence classifications

Discovery

We spent three weeks with the client's document review team, shadowing the manual workflow and cataloguing the 14 document categories, their distinguishing features, and the rules reviewers applied under time pressure. We also analysed 18 months of historical classifications to understand label distribution, inter-annotator agreement, and the document types driving most errors.

Strategy

A multi-stage pipeline rather than a single end-to-end model: OCR extraction, then classification, then rules-based routing. This separation allowed each stage to be independently optimised, evaluated, and replaced as models improved. Human-in-the-loop review was designed in from the start — the system routes low-confidence classifications to the queue rather than forcing a decision.

How we approached it

We built a multi-stage document pipeline: an OCR layer for extraction, a fine-tuned classification model for contract type and jurisdiction, a rules engine for routing based on risk flags, and a human-in-the-loop review queue for edge cases. The system was trained on 18 months of historical classifications and continuously improves from reviewer corrections.

Architecture

Documents enter via an S3 upload trigger. An AWS Lambda function runs OCR (Tesseract) and stores extracted text. A FastAPI inference service runs the classification model and returns a label with a confidence score. Documents below the confidence threshold are routed to the review queue. Reviewer decisions are logged and fed back into the training pipeline weekly.

Implementation

The first six weeks were spent on data preparation and OCR quality improvement — poor OCR output was the primary cause of misclassification. The classification model was fine-tuned in two stages: first on the full historical dataset, then on hard negatives identified from the first evaluation round. The rules engine was built iteratively with input from the compliance team over four two-week sprints.

Key decisions

What changed

Classification accuracy reached 97% at production volume. The team was redeployed to exception handling and client advisory work. Processing throughput scaled to 12,000 documents per day with no additional headcount.

Impact

The 15-person manual review team was redeployed to exception handling and client advisory work — higher-value, lower-volume tasks. The compliance backlog was eliminated within 6 weeks of go-live. The client was able to take on new business volume without hiring.

Lessons learned

1. 1OCR quality was underestimated as a bottleneck — 40% of the project time was spent on extraction quality before the classification model became the limiting factor.
2. 2Confidence calibration matters as much as accuracy — a model with 95% accuracy but poor confidence scores is harder to operate than a 92% accurate model with well-calibrated confidence.
3. 3Reviewer feedback loops need to be fast — weekly retraining cycles were more effective than monthly batch updates for keeping the model aligned with evolving document patterns.