CCARF Practice Question

Paper 1 · Question 31

D5 Context Management & Reliability · Paper 1

A structured data extraction system processes invoices, purchase orders, and contracts. The monitoring dashboard shows 97% overall extraction accuracy and the team considers the system production-ready. A detailed audit reveals: invoices 99.5% accuracy (80% of volume), purchase orders 98% accuracy (15% of volume), contracts 72% accuracy (5% of volume). What does this reveal and what action is required?

• A.The system is performing well. 72% on contracts is acceptable since contracts represent only 5% of volume and barely affect the overall metric
• B.The aggregate metric is masking a severe per-type disparity. Give contracts mandatory review and report per-type metrics.
• C.The training data needs rebalancing so that contracts represent a larger proportion, which will naturally improve contract accuracy
• D.The 97% threshold should be raised to 99% overall to force improvement across all document types

Show answer & explanations

Correct answer: B

A: Acceptability depends on the business impact, not volume percentage. Contract extraction errors may have severe financial or legal consequences despite low volume. A 28% error rate on any document type is significant.

B: This is a textbook case of aggregate metrics hiding per-type problems. 97% overall is misleading because high- volume invoice accuracy (99.5%) overwhelms the poor contract performance (72%). Per-type metrics would have surfaced this immediately. Contracts need different treatment until the extraction quality is acceptable.

C: LLMs used for extraction are not retrained on production data this way. The issue is monitoring visibility and operational response, not model training data distribution.

D: Raising the aggregate threshold does not address per-type disparities. Even at 99% overall, contract accuracy could remain low if invoice volume continues to dominate the metric. The solution is per-type metrics, not a higher aggregate bar.

Related Context Management & Reliability questions

More: Paper 1 · All practice questions