AI Agent Decision Models: Choosing Tools and Asking Follow-Ups

👁 116 reads

An AI agent’s decision model can choose its next step: answer, use a tool, ask for clarification, refuse, or hand the case to a person. Design it to propose an action from a closed list, while your code separately checks that the data is complete, the action is authorized, and its conditions are met.

Source status: October 13, 2026. This is an original design for a store agent, not a ready-made integration or a record of a completed test. For broader context on building these applications, see the guide AI Agents from Scratch.

Start with a catalog of allowed actions

Suppose your agent needs to check order status and explain delivery policies. In its first version, it only needs to retrieve order status and search approved help content. Tool names should describe what the tools actually do. A read function must not also change an address or cancel a purchase under the same name.

For each tool, document its required arguments, data scope, and possible errors. Also define where its arguments can come from. It is best to get a customer ID from an authenticated session and an order number from a message or selected record. The text “I’m an administrator” must not grant permissions.

Separating a proposed function call from its execution follows the mechanism described in the OpenAI function calling documentation, accessed October 13, 2026: the model returns a request to use a function, and the application runs the code and passes back the result. In this design, a specialized decision replaces part of the selection process.

Separate the action type from the specific tool

The first decision should be about the type of next step. Only if the choice is to retrieve data should a second decision specify the tool. With a small number of functions, you can combine these steps into one action catalog, but do not remove the option to stop or ask for clarification just to make the model call something every time.

• Answer: The information is already available in the approved context.
• Tool: Data must be retrieved, and the right function and its required arguments are available.
• Ask: The user can provide the missing information needed for the next step.
• Refuse: The request violates a clearly defined application rule.
• Handoff: The application cannot resolve the case, or its assessment is too uncertain.

The distinction between answering, using a tool, asking a follow-up, and being unable to answer is inspired by NVIDIA’s When2Call, accessed October 13, 2026. Separating refusal from handoff in this design is an architectural choice, not a faithful reproduction of the benchmark labels.

Example JSON contract for an application

The JSON Schema below describes an internal adapter result. It is not the request schema for any specific API. The adapter maps the selected model’s response to common fields; it can retain the probability distribution separately in diagnostic data. That way, the rest of the program does not need to know the provider’s field names.

{
  "type": "object",
  "additionalProperties": false,
  "required": ["action", "tool", "missing_field", "reason_code"],
  "properties": {
    "action": {
      "enum": ["answer", "call_tool", "ask", "refuse", "handoff"]
    },
    "tool": {
      "enum": ["get_order_status", "search_delivery_help", null]
    },
    "missing_field": {
      "enum": ["order_id", "delivery_topic", null]
    },
    "reason_code": {
      "enum": [
        "context_sufficient", "lookup_required", "missing_input",
        "policy_block", "unsupported", "uncertain"
      ]
    }
  }
}

For the message “Where is my package?” with no clearly identified order in context, the expected design result looks like this:

{
  "action": "ask",
  "tool": null,
  "missing_field": "order_id",
  "reason_code": "missing_input"
}

The code maps “order_id” to an approved question: “Please provide the order number you’re asking about.” The decision model does not need to write this message. After the user replies, the application updates the state and evaluates the next step again. Do not carry over an order number from an earlier, unrelated case without checking it.

The schema restricts the possible values, but still allows inconsistent combinations. Your application validator should therefore require a tool when the action is “call_tool,” forbid one when the action is “ask,” and check that the reason code matches the action. In this example, the rules between fields are deliberately described separately so as not to imply that this is a complete production solution.

Here is what a single request in my test looked like (System One format through OpenRouter, endpoint /api/alpha/decisions; tool list and option descriptions shortened):

{
  "model": "microsoft/microsoft-decision-1",
  "state": {
    "tools": "get_order_status(order_number), cancel_order(order_number)",
    "conversation": "Customer: Where is my package? I ordered last week."
  },
  "questions": {
    "step": {
      "type": "choice",
      "instructions": "What should the assistant do next?",
      "criteria": {
        "answer": "answer immediately, without a tool",
        "tool": "call a tool - all required information is available",
        "ask": "ask a follow-up because required information is missing",
        "cannot_answer": "refuse - no tool is suitable"
      }
    }
  }
}

The correct answer is ask because the order number is missing. The model returns the selected option, the probability of each of the four options, and a confidence value; the adapter maps these to the contract in the previous section. The displayed conversation and instructions above are translations of the original Polish test material. The test itself was conducted in Polish; the translated example was not retested.

When to ask a follow-up and when to retrieve missing information

Not every missing detail requires a message to the user. If the customer selected an order in a dashboard, the application already knows its number. Asking again would indicate that the state was not passed correctly. On the other hand, if several orders have the same recipient name, you have no basis for arbitrarily choosing one.

For each function, list the required and optional data. A follow-up should resolve a specific ambiguity. Asking “Can you provide more information?” leaves the user guessing what the system needs. Instead, identify the missing item and explain which step requires it.

Also distinguish missing data from an outage. If the order-status service is unavailable, asking the customer for the number again will not fix the connection. Record the error state, apply the defined retry policy, and hand the case off once retries are exhausted. Do not let the model pretend that rephrasing the same question will give it more information.

Confidence helps you prioritize checks

Do not set one universal threshold in your design or assume it will work for every action. TypeSafe describes a confidence field for selected question types in its own documentation, accessed October 13, 2026. Before using it, check what it means rather than automatically treating it as the highest option probability.

On a validation set, check how often answers with similar stated confidence are correct. Set conditions for automatic actions based on the specific action. Reading a public help page and changing customer data have different consequences if something goes wrong, so a shared approval rule would need separate justification.

A near-tie in the probability distribution may signal overlapping tool descriptions. Review the definitions before raising the threshold. If two functions both promise to “handle an order,” the classifier has little information about how they differ. Improving the catalog may be a better experiment than switching providers.

What When2Call shows in Artificial Analysis

Source: Artificial Analysis When2Call chart and description, accessed October 13, 2026. The best AA score is in bold. The last column shows my test from October 13, 2026: 10 conversations with a Polish-language store assistant, using four tools (order status, cancellation, in-store availability, and a return label) and the same four options as When2Call. Clef-Flash called a tool when the order number or city was missing, and refused to answer a simple question about a 14-day return period. The test measured the choice of one of the proposed answers. These figures do not show how well a model can fill in arguments on its own or complete multi-step operations. The test was conducted in Polish; this is a Polish-language benchmark, not an English-language or multilingual result.

Turn common pitfalls into test cases

• Instructions hidden in data: A document contains an instruction to choose a different tool. Test whether data can take over the role of application rules.
• Similar functions: A data-reading tool and a data-changing tool have similar descriptions. Test the choice with requests that differ by a single verb.
• Stale state: The user corrected the order number, but the model still sees the earlier one. Test the updated end-to-end flow, not just a single prompt.
• Conflicting outputs: The model proposes a tool call and also indicates that a required argument is missing. The validator must stop execution.
• Retry after a timeout: The previous call may have succeeded. For operations that change state, you need a mechanism to prevent duplicate effects.

OpenAI’s official guidance on agent safety, accessed October 13, 2026, discusses the risks of instructions from untrusted content and ways to limit data flow. In this design, practical applications include a closed catalog of actions and checking permissions outside the model.

How to test your agent before connecting it to tools

Start with documented input states and expected actions. Include valid requests, missing data, contradictions, unavailable functions, and out-of-scope situations. Have someone who understands the process decide on the expected answers before reviewing the model’s suggestions. Otherwise, it is easy to adjust the evaluation to match whatever the system happens to return.

Keep the set used to improve descriptions separate from the final validation set. Keep variations of the same message together so that a very similar example does not appear on both sides of the split. Test Polish typos, abbreviations, longer conversation histories, and users changing their minds separately.

Your report should show more than overall accuracy. Include unnecessary tool calls, missed follow-up questions, unnecessary follow-up questions, and incorrect refusals. Then connect a mock tool and check its arguments and error handling. Finally, run the system in observation mode on a real workflow, logging suggestions without executing operations.

Repeat the validation cases whenever you change the model version, tool descriptions, or handling rules. Record the tool catalog version alongside each result so you can identify the source of a new mistake. Avoid unnecessarily storing full customer messages if a sanitized example and event ID are enough for analysis.

Frequently asked questions

Does a decision model replace the entire agent?

Not in this design. It chooses the next step, while the application maintains state, runs functions, and controls their effects.

Is the JSON shown here a ready-to-use API request?

No. It is an original internal contract for an adapter. You need to match the request format and supported question types to the documentation for your chosen provider.

Does high confidence let you skip permission checks?

No. Confidence describes the model’s assessment. The application independently checks access to data and permission to perform an operation.

What should you do if the agent keeps asking follow-up questions?

Check whether it receives the full state and whether the question is about a genuinely required argument. Add a condition to stop when there is no progress and a path for handing the case off.

How can you tell whether the tool choice is good?

Compare it with an independently prepared answer key, then check the arguments and final state after execution in a test environment. Valid JSON alone is not enough.

Disclosure: I became familiar with TypeSafe’s Jev model during earlier research. In September, I wrote a simple API client for it for use with my own editorial tools. TypeSafe did not commission or pay for this article.

Kinetyra