Methodology

Fair fights, shown in full.

Transparency is the point. This page covers how challenges are chosen, how runs are set up, what agents can and can't do, how autonomy is measured, how scoring works, and where the method falls short.

01 Workflow

Eight steps from idea to verdict

  1. 01

    Select a challenge

    An app, game, tool or simulation that is ambitious, visually or technically interesting, and possible to evaluate.

  2. 02

    Write a versioned spec

    Product behavior, architecture constraints, integrations, deliverables, acceptance criteria and test cases, all fixed before any run starts.

  3. 03

    Set up comparable environments

    Starting conditions, tool versions where available, access permissions, credentials policy and deployment targets are recorded.

  4. 04

    Run each agent

    Each agent builds alone for up to the time budget. The session is screen-recorded and meaningful interventions are noted.

  5. 05

    Preserve evidence

    Repositories, spec versions, logs, screenshots, recordings, deployment details and build reports are kept.

  6. 06

    Test after completion

    Automated checks plus realistic, hands-on user journeys: functionality, code quality, deployment, security and failure modes.

  7. 07

    Score and compare

    A transparent rubric, with observed facts kept separate from subjective judgement.

  8. 08

    Publish

    The episode and the experiment page go out together: goal, process, outcomes, evidence and an honest verdict.

02 Time budget

Eight hours per agent, not eight hours total

Each agent gets up to eight hours of wall-clock time. Planning, installs, testing, deployment and writing the report all count. The budget is a ceiling, not a target. In the first experiment, all three agents finished in under two hours.

Eight hours is the current convention, not a permanent rule. Any experiment that changes the budget documents the change on its page.

03 Rules of a run

What agents can and can't do

Allowed

  • Making every routine design and technical decision without asking
  • Choosing frameworks, libraries and architecture within the spec's constraints
  • Using provided credentials and pre-configured infrastructure
  • Requesting approval where an operation genuinely requires it
  • Finishing early. The finish time is recorded.

Not allowed

  • Human-written code, fixes or architecture guidance during the run
  • Agent-specific instruction files or hidden coaching (CLAUDE.md, AGENTS.md, .cursor/rules)
  • Bypassing approval mechanisms or touching production secrets
  • Claiming tests, features or deployments that weren't actually verified
  • Sharing code or feedback between agents before every run is finished

The rules above summarize what the published specs have in common. Each experiment's own spec is the authority and is linked from its page.

04 Measuring autonomy

Interventions are recorded, never hidden

Human involvement is limited to setup, credentials and unavoidable permission prompts. Every intervention is logged with a timestamp. An agent is never described as autonomous if it needed approvals or fixes.

Approval behavior depends on the tool's configuration as much as the model. In the first experiment, one agent asked before each new command, one asked once and offered a "run everything" mode, and one ran without asking for further input. These observations belong to those runs and configurations. They aren't permanent claims about the products.

Autonomy and approval counts are reported separately from quality scores, so a fast but poor build doesn't win by default.

05 Evidence levels

Every fact says where it came from

  • Spec: A requirement from the versioned specification.

    Spec: a requirement from the versioned specification. It describes what was asked, not what was built.

  • Agent claim: Reported by the agent in its own build report. Not independently verified.

    Agent claim: what an agent wrote about its own work in BUILD_REPORT.md or BUILD_RESULTS.json. Useful, but unverified, and agents measure things differently.

  • Git record: Taken from the public repository's commit history.

    Git record: commit times and messages from the public repository. Factual about when, silent about quality.

  • Observed: Observed by The Build Hypothesis during the recorded run.

    Observed: something seen during the recorded session, such as approval prompts or interventions.

  • Verified: Confirmed by independent testing.

    Verified: confirmed by independent, hands-on testing after the run. This is the only level that can support a score or verdict.

A screenshot or a polished UI is never treated as proof that a feature works. "Code exists in the repository" isn't proof either.

06 Scoring

Published rubrics, visible components

Each spec defines how its builds will be judged. Requirements are tiered P0 (core),P1 (required target) and P2 (stretch). Recent specs add a 100-point rubric and hard-failure gates. For example, a video editor with no playable export fails, whatever it looks like.

Winners are never declared on impressions alone. When results are published, each score is broken down by category, and the same core acceptance tests are used across agents where feasible.

07 Choosing challenges

What makes a good experiment

  • A clear, memorable premise that works as a video title
  • Enough difficulty to reveal real differences between agents
  • A finished artifact that can be shown visually or used interactively
  • Objective acceptance criteria and a repeatable evaluation
  • A scope that fits an eight-hour autonomous build
  • Room for surprising successes, failures and real insight

08 Limitations

What this method can't tell you

  • One run per agent. AI agents are not deterministic. A single run is a sample, not a measurement of the tool's ceiling or floor.
  • Configurations matter. Approval modes, sandboxing and environment differences shape results. They are recorded, but they can't be perfectly equalized.
  • Tools change fast. A result describes a specific version on a specific date, not the product forever.
  • Self-reports aren't comparable. Agents count tests, requirements and interventions differently, and until evaluation those numbers are the agents' own.
  • Runs happen in sequence. Agents run at different times, so external services and network conditions may differ.
  • Judgement is involved. Usability and polish involve opinion. Rubrics keep it visible and separate from measured facts.