IDP evaluation checklist

Choosing a document-AI platform is easier when you decide the criteria before the demos. This checklist is vendor neutral. Use it with any product, including ours.

1. Define the job

1. List the document types, volumes per month and languages.
2. Name the fields you need, which are mandatory, and the format your downstream system expects.
3. Decide what a wrong value costs. A mistyped middle name and a wrong claim amount are not the same risk.
4. Write down the target turnaround and who consumes the output.

2. Test accuracy on your own documents

• Build a sample that reflects reality: clean files, poor scans, handwriting, rotated pages, multi-document bundles.
• Label the correct answers yourself, before the vendor sees the files.
• Define “correct” per field: exact match, normalised match, or acceptable range.
• Run every shortlisted vendor on the same blind set, with no tuning that you would not repeat in production.
• Measure per field, not just overall. Note which fields fail and why.
• Check how the product reports confidence, and whether low confidence really lines up with the errors.
• Retest after a month with new documents. Vendors and models change.

Do not accept published accuracy numbers, or ours, as a substitute for this step.

3. Human review

• Can reviewers see the source next to the extracted value?
• Can you route only uncertain results to people, and pause a pipeline at chosen steps?
• Do corrections feed back, and where do those corrections stay?
• Can you assign review work by role, and see how long items wait?

4. Audit trail

• Is there a record of who viewed, changed, approved or deleted data, with timestamps?
• Can you export it? How long is it retained, and is the storage tamper-resistant?
• Are role-based access, last-admin protection and session revocation in place?

5. Deployment model

• Shared service, dedicated tenant, your own cloud account, or on premises? Each trades isolation against effort.
• Who operates it, patches it and responds to incidents?
• Can you use your own AI resources or keys if policy requires it?
• How does identity work: single sign-on, two-factor authentication, invitation flows?

6. Data handling and compliance

• Where is data stored and where is it processed by AI? Ask about both.
• Which subprocessors are used, and how much notice is given of changes?
• Is customer data used for model training? Get the answer in the contract.
• What compliance claims are made, and what is the real status? “In preparation” and “certified” are different statements.
• Is a DPA or BAA available, and on which plans?
• What are the deletion terms and timelines after termination?

7. Integration

• Is the API asynchronous or synchronous, and how are results delivered: polling, webhooks, both?
• How are keys managed, scoped and rotated?
• Which file types are supported and what are the size limits?
• What happens when a job fails or a document is unreadable?

8. Pricing units

• Is pricing per page, per document, per field or per user? Watch minimums, such as a minimum billable page count per document.
• How does overage work, and is anything blocked when you exceed the allowance?
• Are reprocessing, review seats, storage, search and support charged separately?
• Model your own volumes under each vendor’s rules, including a bad month.

9. Exit

• Can you export documents, extracted data, schemas and audit logs in open formats?
• How long after termination is data retained, and can you get a deletion attestation?
• Are there term commitments or auto-renewals?

Scoring your shortlist

Weight the criteria by your own risk, score each vendor in writing, and keep the evidence from your accuracy test. A sound decision is one you can explain to an auditor a year from now.