StepFun Tool Calling: Why Eval Loops Happen and How To Fix Them

Summary

Tool calling is not magic inside the model. A serving layer or benchmark harness sends text to the model, parses a tool request from the model output, runs the real tool, inserts the tool result back into the conversation, and asks the model to continue.

For StepFun Step 3.7 Flash, the continuation format matters. The model expects the tool result to come back as a user-side observation wrapped in <tool_response>. If the result is rendered as a separate custom role, or if the harness asks the model to make another required tool call after the tool result is already present, the model can loop, repeat tools, or fail multi-turn eval tasks that otherwise should be straightforward.

The Mental Model

A local LLM does not call your calculator, browser, shell, calendar, or email API directly. It emits a structured string that your wrapper interprets as a tool request. Your wrapper is responsible for the rest.

What StepFun Expects

The working StepFun pattern is: assistant emits <tool_call>, then the environment returns the result as a user-side <tool_response> observation, then the assistant continues.

Minimal Conversation

<|im_start|>user
What is 123 * 456?
<|im_end|>
<|im_start|>assistant
<tool_call>
<function=calculator>
<parameter=expression>
123 * 456
</parameter>
</function>
</tool_call>
<|im_end|>
<|im_start|>user
<tool_response>
{"result": 56088}
</tool_response>
<|im_end|>
<|im_start|>assistant
The answer is 56,088.
<|im_end|>

Equivalent Structured Messages

If your application works with JSON-style messages, the wrapper can represent the tool result as a user message containing the response block:

[
  {
    "role": "user",
    "content": "What is 123 * 456?"
  },
  {
    "role": "assistant",
    "content": "<tool_call>...</tool_call>"
  },
  {
    "role": "user",
    "content": "<tool_response>\n{\"result\": 56088}\n</tool_response>"
  }
]

Frameworks may use {"role": "tool"} internally. That is acceptable only if the active chat template converts that tool role into the StepFun-compatible user-side <tool_response> form before the prompt reaches the model.

Why This Breaks Evals

Many tool benchmarks use OpenAI-compatible client objects and expect the server template to translate those objects into the model's native format. This often works for models trained to tolerate OpenAI-style tool roles. StepFun is stricter: it expects the tool result to appear in the dialogue grammar it learned.

The Fix

{% elif message.role == "tool" %}
<|im_start|>user
<tool_response>
{{ message.content }}
</tool_response>
<|im_end|>
{% endif %}

first_model_call:
  tool_choice = "required"

after_tool_result:
  tool_choice = "auto"

How To Apply It

Step 1: Find the active chat template

In llama.cpp-style servers, the template may come from the GGUF metadata, a server flag, or a launch profile. You are looking for the logic that handles message.role == "tool", tool_calls, or custom tool-response roles.

The target behavior is simple: tool calls are produced by the assistant, and tool results are rendered as a user message with a <tool_response> block.

Step 2: Patch tool-result rendering

If the template currently emits a custom role, change it to the user-observation form.

<|im_start|>tool_response
{"result": 56088}
<|im_end|>

<|im_start|>user
<tool_response>
{"result": 56088}
</tool_response>
<|im_end|>

Step 3: Patch the client or harness loop

The tool loop should append the assistant's tool call, run the tool, append the result, and call the model again with automatic tool choice.

messages = [
    {"role": "user", "content": "What is 123 * 456?"}
]

response = call_model(
    messages=messages,
    tools=tools,
    tool_choice="required",
)

tool_call = parse_tool_call(response)
tool_result = run_tool(tool_call)

messages.append({
    "role": "assistant",
    "content": response.text,
})

messages.append({
    "role": "user",
    "content": "<tool_response>\n" + tool_result + "\n</tool_response>",
})

final = call_model(
    messages=messages,
    tools=tools,
    tool_choice="auto",
)

Step 4: Keep StepFun thinking mode compatible

For StepFun Step 3.7 Flash, use the normal thinking-compatible format unless the model provider explicitly documents a no-thinking mode for that exact model. Changing the reasoning grammar and the tool grammar at the same time makes debugging much harder.

Verification Checks

After applying the fix, test a small deterministic tool case before running a full benchmark.

In the local StepFun Step 3.7 Flash ROCmFP4 MTP evaluation, this protocol fix cleared the repeated calculator-call guard case and the full Tool Eval run completed at 88/100 with no detected overgeneration scenarios.