CCARF Practice Question

Paper 3 · Question 59

D5 Context Management & Reliability · Paper 3

[Scenario 6: Structured Data Extraction] You are building a structured data extraction system: pulling information from unstructured documents, validating output against JSON schemas, handling edge cases gracefully, and integrating with downstream systems. Your extraction system reports 97% aggregate accuracy, and leadership wants to cut human review of high-confidence extractions. Which TWO practices must precede that decision? (Select TWO.)

• A.Segment accuracy by document type and field — verifying consistent performance across all segments, since aggregate metrics can mask systematic failures on specific types
• B.Accept the 97% aggregate as sufficient evidence on its own
• C.Implement stratified random sampling of high-confidence extractions for ongoing error-rate measurement and detection of novel error patterns
• D.Stop measuring once review is reduced, to save cost
• E.Review only extractions the model itself flags as uncertain

Show answer & explanations

Correct answer: A, C

A: Two safeguards make automation defensible: segmentation proves the average isn't hiding a failing document type or field, and stratified sampling of the “safe” population keeps measuring after the humans step back — catching drift and novel errors.

B: trusts exactly the number that masks segment failures

C: Two safeguards make automation defensible: segmentation proves the average isn't hiding a failing document type or field, and stratified sampling of the “safe” population keeps measuring after the humans step back — catching drift and novel errors.

D: removes the instrumentation precisely when risk increases

E: trusts self-flagged uncertainty, which misses the errors the model is confidently wrong about

Related Context Management & Reliability questions

More: Paper 3 · All practice questions