01 Workflow
Eight steps from idea to verdict
- 01
Select a challenge
An app, game, tool or simulation that is ambitious, visually or technically interesting, and possible to evaluate.
- 02
Write a versioned spec
Product behavior, architecture constraints, integrations, deliverables, acceptance criteria and test cases, all fixed before any run starts.
- 03
Set up comparable environments
Starting conditions, tool versions where available, access permissions, credentials policy and deployment targets are recorded.
- 04
Run each agent
Each agent builds alone for up to the time budget. The session is screen-recorded and meaningful interventions are noted.
- 05
Preserve evidence
Repositories, spec versions, logs, screenshots, recordings, deployment details and build reports are kept.
- 06
Test after completion
Automated checks plus realistic, hands-on user journeys: functionality, code quality, deployment, security and failure modes.
- 07
Score and compare
A transparent rubric, with observed facts kept separate from subjective judgement.
- 08
Publish
The episode and the experiment page go out together: goal, process, outcomes, evidence and an honest verdict.
02 Time budget
Eight hours per agent, not eight hours total
Each agent gets up to eight hours of wall-clock time. Planning, installs, testing, deployment and writing the report all count. The budget is a ceiling, not a target. In the first experiment, all three agents finished in under two hours.
Eight hours is the current convention, not a permanent rule. Any experiment that changes the budget documents the change on its page.
03 Rules of a run
What agents can and can't do
Allowed
- Making every routine design and technical decision without asking
- Choosing frameworks, libraries and architecture within the spec's constraints
- Using provided credentials and pre-configured infrastructure
- Requesting approval where an operation genuinely requires it
- Finishing early. The finish time is recorded.
Not allowed
- Human-written code, fixes or architecture guidance during the run
- Agent-specific instruction files or hidden coaching (CLAUDE.md, AGENTS.md, .cursor/rules)
- Bypassing approval mechanisms or touching production secrets
- Claiming tests, features or deployments that weren't actually verified
- Sharing code or feedback between agents before every run is finished
The rules above summarize what the published specs have in common. Each experiment's own spec is the authority and is linked from its page.
04 Measuring autonomy
Interventions are recorded, never hidden
Human involvement is limited to setup, credentials and unavoidable permission prompts. Every intervention is logged with a timestamp. An agent is never described as autonomous if it needed approvals or fixes.
Approval behavior depends on the tool's configuration as much as the model. In the first experiment, one agent asked before each new command, one asked once and offered a "run everything" mode, and one ran without asking for further input. These observations belong to those runs and configurations. They aren't permanent claims about the products.
Autonomy and approval counts are reported separately from quality scores, so a fast but poor build doesn't win by default.
05 Evidence levels
Every fact says where it came from
- Spec: A requirement from the versioned specification.
Spec: a requirement from the versioned specification. It describes what was asked, not what was built.
- Agent claim: Reported by the agent in its own build report. Not independently verified.
Agent claim: what an agent wrote about its own work in BUILD_REPORT.md or BUILD_RESULTS.json. Useful, but unverified, and agents measure things differently.
- Git record: Taken from the public repository's commit history.
Git record: commit times and messages from the public repository. Factual about when, silent about quality.
- Observed: Observed by The Build Hypothesis during the recorded run.
Observed: something seen during the recorded session, such as approval prompts or interventions.
- Verified: Confirmed by independent testing.
Verified: confirmed by independent, hands-on testing after the run. This is the only level that can support a score or verdict.
A screenshot or a polished UI is never treated as proof that a feature works. "Code exists in the repository" isn't proof either.
06 Scoring
Published rubrics, visible components
Each spec defines how its builds will be judged. Requirements are tiered P0 (core),P1 (required target) and P2 (stretch). Recent specs add a 100-point rubric and hard-failure gates. For example, a video editor with no playable export fails, whatever it looks like.
Winners are never declared on impressions alone. When results are published, each score is broken down by category, and the same core acceptance tests are used across agents where feasible.
07 Choosing challenges
What makes a good experiment
- A clear, memorable premise that works as a video title
- Enough difficulty to reveal real differences between agents
- A finished artifact that can be shown visually or used interactively
- Objective acceptance criteria and a repeatable evaluation
- A scope that fits an eight-hour autonomous build
- Room for surprising successes, failures and real insight
08 Limitations
What this method can't tell you
- One run per agent. AI agents are not deterministic. A single run is a sample, not a measurement of the tool's ceiling or floor.
- Configurations matter. Approval modes, sandboxing and environment differences shape results. They are recorded, but they can't be perfectly equalized.
- Tools change fast. A result describes a specific version on a specific date, not the product forever.
- Self-reports aren't comparable. Agents count tests, requirements and interventions differently, and until evaluation those numbers are the agents' own.
- Runs happen in sequence. Agents run at different times, so external services and network conditions may differ.
- Judgement is involved. Usability and polish involve opinion. Rubrics keep it visible and separate from measured facts.