How GeistBench scores are built
The full method behind every GeistBench number: what is measured, how it is graded, how scores and intervals are computed, when a release may rank, and where the method has limits.
From task to score
What is measured
A configured agent system, not a bare model: one model with fixed reasoning, routing, tool surface and output bounds, working through enterprise workflows. The same model with different settings is a separate configuration.
How it is graded
Every ranked task is graded by a deterministic check of the end state. No model grades another model. Work that needs a judge stays out of the ranking until the judge is calibrated.
How the score is built
- Variants of one task (English and German, clean and noisy) form a task family.
- Family pass rate: PASS / (PASS + TASK_FAILURE). Blocked, void, insufficient-evidence and not-exercised outcomes are excluded from this denominator.
- Category score: the unweighted mean of family pass rates in that category.
- GeistBench score: the unweighted mean of the category scores. No category is reweighted.
- A configuration missing any category has no overall score and is not ranked.
Uncertainty and ranks
Published 95% intervals use a percentile cluster bootstrap with 2,000 resamples, seeded from the release. Task families are resampled within each category independently, then category means are averaged for the overall interval. The website displays these published intervals. Ranks use competition ranking (1, 2, 2, 4); overlapping intervals do not establish a statistically reliable difference.
Rank ranges
Each ranked configuration has a descriptive range based on overlap between the published 95% score intervals. The best rank is 1 plus the number of ranked configurations whose interval lies strictly above its own; the worst is the number of ranked configurations minus those whose interval lies strictly below. Touching intervals count as overlapping. This is not a simultaneous 95% confidence interval for rank and does not assign probabilities to the ranks within it.
Repeats and held-out work
The release's minimum repeat parameter is k = 3. At least 0% of the task families in each category are held out: they count in the score and are never listed. Diagnostic releases may have incomplete attempts.
A release is ranked only if
- every configuration in a category ran the same tasks from one campaign plan
- every task has at least three graded attempts
- no planned attempt was skipped, and blocked, void, insufficient-evidence and not-exercised outcomes together are at most 5% of planned attempts
- cost is settled for every attempt
- grading is deterministic
- at least 20% of each category is held out
A configuration that misses a gate is listed as not ranked, with the reason.
What the score does not mean
- It is not a certification, and not a general intelligence score.
- It describes these configurations on this release's tasks. Another harness, prompt or setting can change it.
- Scores from different releases are not comparable unless the release notes say so.
Published and withheld
Published: category and overall scores with intervals, reliability, cost per attempt, robustness deltas.
Withheld: task content, task names and per-task results. This reduces exposure but cannot rule out contamination or tuning to aggregate results.
Where the method has limits
Synthetic environments
Tasks run in constructed enterprise workspaces with synthetic records, reached through tool interfaces. Real systems carry more data, more exceptions and more history. Scores describe behaviour in these environments.
Categories weighted equally
Categories included in a release have equal weight in its overall score, whatever the number of task families in each. Releases may include fewer than five categories. Equal weighting is a design choice, not a claim that the areas matter equally to every organisation.
Cost is per attempt
Cost is the recorded spend divided by the attempts with a recorded cost, in USD. Its basis is stated per configuration: catalog price estimate, subscription without a per-call price, mixed or unknown. Where recorded, catalog estimates state the date of the price list. Incomplete cost is shown as a known subtotal. It is not a price quote; list prices change and contract prices differ.
Small k
Ranking requires at least three graded attempts per task and configuration. Diagnostic results may fall below this threshold and remain unranked. Reliability figures from few attempts can be coarse, and intervals may overlap.
Held-out tasks cannot be audited
Ranking requires at least 20% held-out task families in each included category. Diagnostic results may fall below this threshold and remain unranked. Each release states the share. Task content is withheld, so readers cannot audit individual tasks.
Model versions change
A provider can change the model behind a name. A release records the configuration it ran; a later run under the same name may behave differently. Compare releases only where the release notes say they are comparable.
Observed, not a certification
Results are observations from our own runs. No independent body has audited them, and they are not a certification of any model or product.
Conflict of interest
What we do to limit bias
Disclosed grading
Ranked results require code-based checks of the stored end state. Unranked diagnostics can also include model-judged answers, with the judge and its limits disclosed in the release notes.
Held-out tasks
Ranking requires at least 20% held-out task families per included category. Withholding task content reduces public exposure but does not rule out contamination, tuning or bias in task selection.
One plan for every configuration
Ranked comparisons require a common campaign plan, matching task sets and the planned repeat count within each included category. Compute budgets and actual spend are not necessarily equal. Published configuration settings include output-token limits. Diagnostic, unranked results may fail these comparison requirements.
Published methodology
Scoring, gates and limitations are published here. Every release is immutable and carries a summary hash.