AI order automation should be evaluated on the decisions an order desk must make. A clean-looking table does not show whether a line was omitted, an invoice was mistaken for a new order or a customer reference lost its leading zeros. Those failures need different measurements and different responses.
Build the evaluation before relying on a demonstration. Give the system representative files, independently prepared expected values and examples it should refuse to process as orders. OpenAI's evaluation guidance recommends task-specific tests, realistic edge cases and human calibration. For purchase orders, that means testing the complete document and the resulting review work.
Choose documents that represent the real queue
Sample across customers, document layouts, file types, line counts and image quality. Include recurring clean orders, awkward scans, multiline descriptions, small decimal quantities, unusual units and amended documents. Record how often these categories occur so the final results can be read against the actual workload.
Keep a held-out set that is not used while tuning prompts or mappings. Otherwise the evaluation gradually becomes a demonstration of memorized examples. Synthetic orders help isolate specific problems and avoid exposing customer data during early development, but they should not be used to advertise a customer-document accuracy rate. Agree an authorized handling route before introducing real samples.
Test whether the document is an order at all
Add quotations, invoices, delivery notes, catalog pages and blank or unreadable files. Also include a file containing two separate orders if the proposed workflow accepts only one. A system that always returns an order-shaped object may score well on positive examples while creating work from documents that should have been rejected.
Measure valid orders rejected and non-orders accepted separately. Have the order team decide what happens when document type is uncertain: hold it for review, request a clearer source or route it elsewhere. Do not silently convert a quotation into a purchase commitment because it contains familiar columns and a total.
Report field, line and complete-document results
A field score needs a defined denominator. Count required values that are absent, extra values the system invented and lines that are missing or duplicated. Match the expected and returned lines using a documented rule so a shifted row cannot earn credit for a value belonging to another product.
Keep raw matches and normalized matches separate. Whitespace or an equivalent decimal representation may be harmless; dropping a leading zero from an identifier may be a different product. Agree these rules with the business owner before running the comparison. Then report how many complete orders met every required check, rather than relying on an average across thousands of easy cells.
| Measure | What it reveals |
|---|---|
| Document eligibility | Orders rejected and non-orders accepted |
| Required field matches | Missing, changed or invented values |
| Line completeness | Dropped, added or duplicated order lines |
| Complete-order success | Orders meeting every required extraction check |
| Reviewer effort | Work remaining before the ERP handoff |
Evaluate extraction and business validation separately
Correctly reading a customer part number does not prove it matches the right internal SKU. Likewise, a clearly printed price may disagree with the applicable agreement. Prepare separate expected outcomes for customer resolution, product matching, unit checks, duplicate detection and commercial review.
Use a case where two cable variants share most of their description. The extraction should preserve the distinguishing text; the matching stage should retain uncertainty until the evidence selects one variant. A reviewer choosing a candidate is a review outcome, not evidence that the model identified the product correctly without help.
Count recovery time and failed attempts
Time the journey from a submitted file to a reviewed, saved order. Include timeouts, retries, record searches and corrections. Report the slow cases as well as the typical case, particularly for long orders where an output limit can interrupt an otherwise readable document.
Track actual provider usage and all reprocessing attempts against the agreed operating envelope. Model context size does not guarantee that a complete long order will fit in the response. When the system stops, the operator should retain the original file and understand the next action. A failure that loses the selected file or creates a partial order adds avoidable work.
Tie the result to the version being released
Record the model, prompt, schema, extraction settings, dataset version and scoring rules. Rerun relevant cases after a change to document classification or output structure. If a run is interrupted, keep its partial observations separate from a completed evaluation; do not carry an earlier pass forward to a revised implementation.
Agree release criteria before seeing the scores. A wrong customer, missing line or accepted non-order may require holding the rollout even if most fields are correct. Keep human approval in the operating process and confirm that unresolved required issues cannot reach the handoff.
Use evidence to define the operating scope
Ordinon prepares order data for source-side review and human approval, then exports the approved values as CSV. A compatible catalog connection supports manual product pulls; it does not establish automatic ERP posting. Evaluate extraction quality and the customer-specific handoff independently, and expand the supported document mix only when the evidence supports it.