CCARF Practice Question

Paper 1 · Question 55

D5 Context Management & Reliability · Paper 1

Your extraction pipeline reports 97% aggregate accuracy, and you want to automate high- confidence extractions. Which safeguards should be in place before you reduce human review? (Select 3)

• A.Calibrate field-level confidence thresholds against a labelled validation set.
• B.Rely on the aggregate accuracy figure once it has held steady for a full quarter.
• C.Verify accuracy separately by document type and field segment, not just in aggregate.
• D.Keep stratified random samples of high-confidence extractions under review to measure error rates and catch novel patterns.
• E.Expand the review team until every extraction can be checked by hand.

Show answer & explanations

Correct answer: A, C, D

A: Confidence scores only route review attention correctly once they are calibrated on labelled data.

B: Stability of the aggregate number does not reveal per-segment weaknesses, which is the risk the guide highlights.

C: A strong aggregate figure can mask poor performance on specific document types or fields.

D: Stratified sampling is the guide's mechanism for ongoing measurement after automation.

E: Reviewing everything ignores the point of calibration: routing limited reviewer capacity to low-confidence and ambiguous cases.

Related Context Management & Reliability questions

More: Paper 1 · All practice questions