CCARF Practice Question

Paper 5 · Question 36

D4 Prompt Engineering & Structured Output · Paper 5

[Scenario: Customer Support Agent] Production metrics show that when resolving complex billing disputes or multi-order returns, customer satisfaction scores are 15% lower than for simple cases—even when the resolution is technically correct. Root-cause analysis shows the agent provides accurate solutions but inconsistently explains rationale: sometimes omitting relevant policy details, sometimes missing timeline info or next steps. The specific context gaps vary case by case. You want to improve solution quality without adding human oversight. What approach is most effective?

• A.Add a self-critique stage where the agent evaluates a draft response for completeness—ensuring it resolves the customer’s issue, includes relevant context, and anticipates follow-up questions.
• B.Add a confirmation stage where the agent asks “Does this fully resolve your issue?” before closing, allowing customers to request additional information if needed.
• C.Upgrade the model from Haiku to Sonnet for complex cases, routing based on a defined complexity metric.
• D.Implement few-shot examples in the system prompt showing complete explanations for five common complex case types, demonstrating how to include policy context, timelines, and next steps.

Show answer & explanations

Correct answer: A

A: A self-critique stage (the evaluator-optimizer pattern) directly addresses inconsistent explanation completeness by forcing the agent to assess its own draft against concrete criteria—such as policy context, timelines, and next steps—before presenting it. This catches case-specific gaps without human oversight.

B: Asking whether the reply fully resolves the issue pushes the completeness check onto the customer. Many will say yes and leave.

C: A larger model does not define which details a complete answer must include. The gaps change by case.

D: Five examples cover five case types. The missing timeline or next step on the next case is not in that set.

Related Prompt Engineering & Structured Output questions

More: Paper 5 · All practice questions