GeistBench category
Tool reliability
Whether an agent uses tools correctly: picks the right tool, calls it validly, recovers from errors, and declines what the tools cannot do.
This page defines a category, not the evaluated scope of a release. See the release coverage for capabilities actually assessed. All categories
Systems and records
- The tool interfaces of an enterprise workspace, in-app or through an MCP server
- A knowledge base describing what the workspace can and cannot do
- Navigation that depends on the acting role's permissions
What the tasks involve
- Route a request to the tool that can answer it
- Answer a question about a capability, or decline correctly when the capability does not exist
- Complete a task when a tool call fails or returns an error first
Roles and handoffs
Tool access depends on the acting role. A correct outcome can be a refusal, when the role or the workspace cannot do what was asked.
What a correct outcome means
The task's required effect exists in stored state, or the agent declined correctly with state unchanged. Calls the task forbids never happen.
How it is checked
The checker verifies the effect of the calls in stored state, not the agent's description of them. For refusals and read-only tasks it confirms that nothing was written.
What this category does not test
- Speed or style of tool use beyond the recorded limits
- Tools outside the workspace, such as the open web
- Tool design; each configuration uses the tool surface it is listed with
FAQ
Frequently asked questions
- What does the GeistBench Tool reliability category test?
- Whether an agent uses tools correctly: picks the right tool, calls it validly, recovers from errors, and declines what the tools cannot do.
- How are Tool reliability tasks graded?
- The checker verifies the effect of the calls in stored state, not the agent's description of them. For refusals and read-only tasks it confirms that nothing was written.
- What does the Tool reliability category not test?
- Speed or style of tool use beyond the recorded limits. Tools outside the workspace, such as the open web. Tool design; each configuration uses the tool surface it is listed with.
- Where are the Tool reliability scores?
- Category scores are published per release on the GeistBench leaderboard, with 95% intervals. This page does not repeat them. A release may not include every category; then the category has no score in that release.