Improvement pass · 31 July 2026
One night, 106 questions, four ways PDFs lie to parsers
Last night this project ran its first automated hardening pass: an exam written against the agreements themselves, and a loop that only accepts a fix when the failure behind it can be named as a mechanism. This is a summary of what it found. The score moved from 69 to 77 out of 106 — but the score is the least interesting part.
The exam came first, and it was written by hand
106 questions across seven vendors' published agreements — OpenText, SAP, Salesforce, Okta, Adobe, Qlik, and a synthetic vendor with a full order form. Every expected answer was vetted by reading the deciding clauses in the source documents, never by asking the engine: the student does not get to write the answer key. The set was then frozen, with thirty questions held out as a surprise quiz the fixing process never saw. It also includes questions whose correct answer is "the agreements do not address this" — an engine that cannot say that will confidently invent rights no contract grants.
Four ways PDFs lie to parsers
The most valuable findings were not in the answering engine at all. They were in the gap between what a PDF shows a human and what it yields to a machine — and none of them are any vendor's fault in particular:
- Sentences that cross a page turn get cut. One family's order-of-precedence clause — the sentence that says which document wins a conflict — ends mid-sentence in extraction because its list starts at the bottom of a page. The engine had zero precedence records for a seven-document family and no idea anything was missing.
- Several sections can fuse into one. Five distinct mechanisms (beta terms, AI terms, free-trial terms, free-service terms, personnel) extracted as one clause with one label.
- Text can shatter into spaced glyphs — "i s d efined a s" — invisible to any word search while looking perfectly readable to a human.
- Two-column layouts can interleave, weaving two sentences into confident nonsense.
The generalization matters more than any instance: these are properties of vendor PDF styles at large. A parser that has not been made to fail on them simply has not met enough PDFs yet.
The measuring stick was bent, and the loop noticed
Eighteen of the original failures were not engine failures at all: the
scoring script counted grouped citations like [10, 11, 12]
as no citation. The overnight worker checked the instrument before
trusting the grades, fixed it, and re-scored. A benchmark that is not
itself audited will cheerfully report improvements that do not exist —
and failures that do not either.
The deepest finding: citations that cannot say "I'm not sure"
Three separate defects — a stale heading claimed by nineteen clauses, one section number naming two different sections, and the bent measuring stick above — are the same defect in different clothes. A citation is a claim about provenance, and this pipeline had no way for such a claim to be uncertain: every label is asserted with full confidence, right or wrong. Teaching the labelling layer to say "don't know" is the one architectural change this pass put on the roadmap.
Honesty section
The practice questions improved by six; the surprise quiz by one, then by one again in a follow-up pass. With a thirty-question holdout, that is statistically indistinguishable both from real generalization and from teaching to the test — which is itself the finding: the golden set needs to grow, and the next corpus expansion is already underway. Several failures were deliberately left standing: questions the current product cannot answer until it learns to model products as first-class graph citizens, and parser fixes that require re-extracting a family. Recorded, not rounded away.
Method note: fixes were only accepted when stated vendor-neutrally, and the same legal pattern was planted across multiple vendors — the purchase-order-void clause appears in three corpora in three phrasings, and merger-assignment rules that differ (consent required vs. notice sufficient) sit one golden apart — so a fix that memorizes one vendor's wording fails its sibling.