CCARF Practice Question

Paper 3 · Question 60

D5 Context Management & Reliability · Paper 3

[Scenario 6: Structured Data Extraction] You are building a structured data extraction system: pulling information from unstructured documents, validating output against JSON schemas, handling edge cases gracefully, and integrating with downstream systems. Reviewer capacity covers only a fraction of extractions, so review attention must be routed where errors are likeliest. Which TWO practices form a sound routing design? (Select TWO.)

• A.Trust the model's raw self-reported confidence scores without validation
• B.Have the model output field-level confidence scores, then calibrate review thresholds against a labeled validation set
• C.Review a fixed 1% of extractions chosen by submission order
• D.Route to human review the extractions with low calibrated confidence or ambiguous/contradictory source documents, prioritizing limited reviewer capacity where errors concentrate
• E.Review only documents longer than ten pages

Show answer & explanations

Correct answer: B, D

A: uses uncalibrated confidence — the known-unreliable version of the right signal

B: Calibration then routing: field-level confidence scores become meaningful once thresholds are tuned on labeled data, and review capacity flows to the calibrated-low-confidence and ambiguous-source cases where errors actually cluster.

C: and E route by arrival order and page count, neither of which correlates with error risk. How did you go? If this helped your CCAR-F preparation, share it with someone else who’s preparing. For more, follow linkedin.com/in/purcellmatthew

D: Calibration then routing: field-level confidence scores become meaningful once thresholds are tuned on labeled data, and review capacity flows to the calibrated-low-confidence and ambiguous-source cases where errors actually cluster.

E: This option does not best satisfy the scenario requirements.

Related Context Management & Reliability questions

More: Paper 3 · All practice questions