6 min
Reading documents reliably, and when not to
Extraction is easy to demonstrate and hard to trust. The difference is what happens to the rows it is unsure about.
Taking figures out of a PDF demonstrates well. Drop in a file, watch the fields appear, and the problem looks solved. It is the easiest thing to show and one of the harder things to depend on.
The demonstration and the Tuesday
A demonstration uses a document somebody chose. Real input is whatever arrives: a scan at an angle, a form from two revisions ago, a figure written in the margin, a table that runs across a page break, a value that appears twice with different numbers.
Extraction that works on the chosen document and quietly produces a wrong number on the awkward one is worse than no extraction, because the wrong number looks exactly like a right one.
What makes it dependable
It says how sure it is. Not a score nobody can interpret — a split between values it is confident about and values it is not, with the second group handed to a person rather than written into a system.
It points back at the source. Every extracted value should be traceable to where it came from in the document. Checking a figure against a highlighted line takes seconds. Hunting for it takes as long as reading the document yourself.
It fails loudly. A document it cannot read should stop and say so. The failure mode to design against is the one where a field is silently left empty and a downstream process treats empty as zero.
When a person should decide
Anything where the document contradicts itself. Anything where the rule depends on context the document does not contain. Anything where being wrong is expensive and being slow is not.
Those are not gaps to be closed later. They are the design.
The shape that holds up
Read the document. Extract what can be extracted. Check it against whatever other record exists. Then produce two things: the values that are settled, and the short list of cases a person has to look at, each with the reason and the place in the document to look.
That last list is the point. A system that returns no exceptions is not more capable than one that returns a few; it is just not telling you.
You can see this shape running on synthetic data in the systems and demonstrations.