CCARF Practice Question

Paper 5 · Question 38

D2 Tool Design & MCP Integration · Paper 5

[Scenario: Customer Support Agent] Production logs show a consistent pattern: when customers include the word “account” in their message (e.g., “I want to check my account for an order I made yesterday”), the agent calls `get_customer` first 78% of the time. When customers phrase similar requests without “account” (e.g., “I want to check an order I made yesterday”), it calls `lookup_order` first 93% of the time. Tool descriptions are clear and unambiguous. What is the most likely root cause of this discrepancy? What is the most likely root cause?

• A.The system prompt contains keyword-sensitive instructions that steer behavior based on terms like “account,” creating unintended tool-selection patterns.
• B.The model’s base training creates associations between “account” terminology and customer-related operations that override tool descriptions.
• C.The model needs more training data on multi-concept messages and should be fine-tuned on examples containing both account and order terminology.
• D.Tool descriptions need additional negative examples specifying when NOT to use each tool to prevent this keyword-induced confusion.

Show answer & explanations

Correct answer: A

A: The systematic keyword-driven pattern (78% vs 93%) strongly indicates explicit routing logic in the system prompt reacting to the word “account” and steering the agent toward customer-related tools. Since tool descriptions are already clear, the discrepancy points to prompt-level instructions creating unintended behavioral steering.

B: A base-model association would not flip this cleanly on the word "account" while the descriptions stay clear. The 78% versus 93% split points at an instruction.

C: This is not a fine-tuning problem, and fine-tuning is outside this exam. The pattern tracks the prompt.

D: Negative examples in the descriptions fight a prompt rule that is still steering on the keyword. The descriptions are already clear.

Related Tool Design & MCP Integration questions

More: Paper 5 · All practice questions