Where AI Fails at Document Review: Failure Modes From a Year of SOP Conflict Checks

Document conflicts are the quiet audit finding: the sanitation SOP says one concentration, the master cleaning schedule says another, and both were approved by people acting in good faith two revisions apart. This is exactly the cross-referencing work humans are worst at and machines should be best at. After a year of running AI conflict checks across our controlled documents, here is the honest scorecard.

Where it beats human review outright

Exhaustiveness. The model compares every document against every other document every time, which no human reviewer has ever done in the history of document control. Numeric mismatches, contradictory responsibilities, and orphaned references to retired procedures surface reliably, including conflicts that had survived multiple annual reviews.

The three failure modes that matter

• Implication blindness: two procedures that conflict only through their consequences, with no shared vocabulary to match on

• Authority confusion: flagging a deliberate site-specific deviation from a corporate template as if it were an error

• Confident scope drift: reviewing a document against a standard edition it was never written to meet

Design review around the failure modes

The pattern across all three: the model is weakest exactly where context lives outside the documents. So we changed what we feed it, edition bindings and deviation registers ride along with the documents, and we changed what reviewers do with the output: a flagged conflict is a finding to verify, and a clean pass on implication-heavy procedure pairs is treated as no information, not as clearance.

That last sentence is the one we would put on a poster. Knowing where the tool returns no information, and treating its silence there accordingly, is the difference between AI-assisted document control and AI-flavored false confidence.