Diagnostic release 0.1.0 · 2 models · not ranked

GeistBench

Name: GeistBench release 0.1.0
Creator: Geist
Published: 2026-10-08T04:43:23.919Z

Observed results for configured AI agents in synthetic enterprise workspaces. Each release states which capabilities were evaluated, the limits of the evidence and whether comparisons are supported.

Score: mean pass rate per task family, averaged within each category, then across categories. Bars show the score, whiskers the 95% interval. How the score works

Download JSON Download CSV Request a run

About this release

Initial diagnostic release: 48 coached tool-contract trials across eight finance and logistics tools, with three repetitions per model. DeepSeek V4.1 Flash passed 24/24; Gemini 3.5 Flash Lite passed 21/24. All three Gemini failures were exact-argument mismatches in the projection-rebuild probe; the rebuild itself succeeded each time. Real API-funded calls used the in-app execution surface, launched through the runtime CLI. Frozen product source: de42f6b4c004181007bfd4b97d87867a2e9659cd. These probes measure supplied-argument adherence, not end-to-end workflow ability or general model quality. No provider ranking or latency comparison. Later write-safety changes and workflow trials are excluded. Recorded cost: USD0.208726, a catalog estimate rather than a reconciled invoice.

What this release evaluates

Business categories group tasks. They do not establish which product capabilities were tested. Coverage below refers only to this release's published evidence.

Categories included in this release: Tool reliability.

Capability	Published evidence	What the evidence supports
Tool argument adherence	Published diagnostic only	Coached checks with supplied arguments. These do not establish end-to-end workflow ability.
World Map writes	Not reported	Capability-specific coverage is unavailable. Neither evaluation nor absence of evaluation can be inferred.
End-to-end World Map editing	Not evaluated in this release	This capability is outside the release's evaluated scope.
Video tool workflows	Not evaluated in this release	This capability is outside the release's evaluated scope.
Video understanding quality	Not evaluated in this release	This capability is outside the release's evaluated scope.

Planned or implemented checks describe evaluation readiness. Executed means a run has retained evidence; validated means that evidence passed its review gates; published means approved results appear in a release. None of these states is inferred here from a category name or the presence of code.

Models: 2
Configurations: 2
Categories: 1
Runs per task: 3

This release did not pass the claim gates for a comparison. Scores are shown in a fixed order, without ranks.

deepseek-flash (high)DeepSeek100.0%

gemini-3.5-flash-lite (minimal)Google87.5%

Score; bar colour follows the provider's brand accent, grey for monochrome brands95% interval

Show as table

GeistBench score
Configuration	Rank	Score	95% interval	Status
deepseek-flash (high)DeepSeek · reasoning high · direct · in-app tools · direct writes · 16,384 max output tokens	–	100.0%	100.0% to 100.0%	Diagnostic release, no ranks
gemini-3.5-flash-lite (minimal)Google · reasoning minimal · direct · in-app tools · direct writes · 16,384 max output tokens	–	87.5%	62.5% to 100.0%	Diagnostic release, no ranks

Cost

Cost per task attempt

Recorded cost divided by the trials that have a cost, in USD. The table states the cost basis for each configuration. Lower is cheaper.

deepseek-flash (high)DeepSeek$0.0013

gemini-3.5-flash-lite (minimal)Google$0.0074

Linear scale from zero.Prices as of 14 August 2026.

Show as table

Cost per task attempt
Configuration	Cost per attempt	Cost basis	Median wall time
deepseek-flash (high)DeepSeek · reasoning high · direct · in-app tools · direct writes · 16,384 max output tokens	$0.0013	Catalog price estimate	Unavailable
gemini-3.5-flash-lite (minimal)Google · reasoning minimal · direct · in-app tools · direct writes · 16,384 max output tokens	$0.0074	Catalog price estimate	Unavailable

Value

Score vs cost

Each point is one configuration. On the Pareto frontier, no other plotted configuration costs no more and scores at least as high, with a strict improvement on either measure. Comparative releases plot ranked configurations only.

On the frontierOther configurationsPareto frontierFewer than three plotted configurations, so no median split.Colour follows each provider's brand accent where it has one. Providers with monochrome brands share grey and are told apart by marker shape and label.

Show as table

Score vs cost
Configuration	Score	Cost per attempt	On frontier
deepseek-flash (high)DeepSeek · reasoning high · direct · in-app tools · direct writes · 16,384 max output tokens	100.0%	$0.0013	Yes
gemini-3.5-flash-lite (minimal)Google · reasoning minimal · direct · in-app tools · direct writes · 16,384 max output tokens	87.5%	$0.0074	No

By category

Where each configuration is strong

Category score per configuration. Stronger teal marks a higher score; every cell prints its value. Missing categories stay empty and are never filled in.

Category scores for 2 configurations across 1 categories.
Configuration	Tool reliability	Overall
deepseek-flash (high)DeepSeek	100.0%	100.0%
gemini-3.5-flash-lite (minimal)Google	87.5%	87.5%

Unavailable

Reliability

Peak is not the floor

For each configuration's published k, pass@k estimates the chance that at least one of k attempts passes; pass^k estimates the chance that all k pass. Neither is a guarantee for a production attempt. The table lists k for each configuration.

deepseek-flash (high)DeepSeek100.0% / 100.0%

gemini-3.5-flash-lite (minimal)Google87.5% / 87.5%

pass@k: at least one of k passespass^k: all k pass

Show as table

Peak is not the floor
Configuration	k	pass@k	pass^k
deepseek-flash (high)DeepSeek · reasoning high · direct · in-app tools · direct writes · 16,384 max output tokens	3	100.0%	100.0%
gemini-3.5-flash-lite (minimal)Google · reasoning minimal · direct · in-app tools · direct writes · 16,384 max output tokens	3	87.5%	87.5%

Robustness

German and noisy prompts

Deltas use only task families with both variants: German minus English, and noisy minus clean, in percentage points. Missing paired observations stay unavailable.

German vs English

deepseek-flash (high)DeepSeekUnavailable

gemini-3.5-flash-lite (minimal)GoogleUnavailable

Noisy vs clean

deepseek-flash (high)DeepSeekUnavailable

gemini-3.5-flash-lite (minimal)GoogleUnavailable

Right of zero: the German or noisy variant scored higher.Left of zero: it scored lower.

Show as table

German and noisy prompts
Configuration	German vs English	Noisy vs clean
deepseek-flash (high)DeepSeek · reasoning high · direct · in-app tools · direct writes · 16,384 max output tokens	Unavailable	Unavailable
gemini-3.5-flash-lite (minimal)Google · reasoning minimal · direct · in-app tools · direct writes · 16,384 max output tokens	Unavailable	Unavailable

Methodology

How a number gets on this page

We measure configured agent systems: one model with fixed reasoning, routing, tool surface and output bounds.

Every ranked task is graded by a deterministic check of its end state. No model grades another model.

The score averages task-family pass rates within each category, then across categories, with published 95% intervals.

Read the full methodology

A release is ranked only if

every configuration ran the same tasks from one campaign plan
every task has at least three graded attempts
cost is settled for every attempt
at least 20% of each category is held out

Releases

Release history and citation

Every published release keeps its own permanent page and JSON file. Releases are never pooled.

Release history and citation
Version	Published	Claim	Links
0.1.0LatestShown	8 October 2026	Diagnostic	Open JSON

Cite this release

BibTeX

@misc{geistbench-0-1-0,
  title = {GeistBench release 0.1.0: observed results},
  author = {{Geist}},
  year = {2026},
  url = {https://geist.company/geistbench/releases/geistbench-b12-contract-20261008-diagnostic},
  note = {Summary hash bcd88ab15069cf9adc01846328722f7af64e6bff8741f4b6b39b93e64aad08a9}
}

Plain text

Geist (2026). GeistBench release 0.1.0: observed results. https://geist.company/geistbench/releases/geistbench-b12-contract-20261008-diagnostic. Summary hash bcd88ab15069cf9adc01846328722f7af64e6bff8741f4b6b39b93e64aad08a9.

Summary hash: bcd88ab15069cf9a
Permanent link: https://geist.company/geistbench/releases/geistbench-b12-contract-20261008-diagnostic

For AI labs and vendors

Request a benchmark run

We run every configuration ourselves, under the published methodology and on the same tasks as every other configuration in the campaign.
Results publish only in a release that passes the release gates, whatever they show.
No pay-to-rank: a request cannot buy a rank, a placement or an exception to the gates.

Request a run

GeistBench

What this release evaluates

GeistBench score

Cost per task attempt

Score vs cost

Where each configuration is strong

Peak is not the floor

German and noisy prompts

German vs English

Noisy vs clean

How a number gets on this page

A release is ranked only if

Release history and citation

Cite this release

BibTeX

Plain text

Request a benchmark run