EXP-001 · comparison

AI Travel Planner

3 agents, one spec (v1.7). Every row below shows where its data comes from. Independent scores have not been published yet, so nothing here declares a winner.

Read before comparing

  • Agent claim: Reported by the agent in its own build report. Not independently verified. The agents split the same spec into very different requirement lists (73, 55 and 24 items), so the counts are not comparable.
  • Agent claim: Reported by the agent in its own build report. Not independently verified. All three ran into Cloudflare's production limit of 100,000 PBKDF2 iterations for password hashing, and each handled it differently.
  • Agent claim: Reported by the agent in its own build report. Not independently verified. The agents counted tests, requirements and interventions differently. Raw totals don't compare like for like.
Side-by-side comparison of agent runs. Every row shows where its data comes from.
MetricClaude CodeAnthropicOpenAI CodexOpenAICursor CLICursor
Run lengthAgent claim: Reported by the agent in its own build report. Not independently verified.Elapsed time the agent recorded in BUILD_RESULTS.json.1h 25m5,125 seconds, measured until the report was generated. The final commit and build check came after.1h 28m5,262 seconds, measured when the report was generated. The end time was left empty on purpose while publishing was still in progress.24.5 min1,470 seconds. The start time is given as approximate ("~19:01 UTC").
Approval behaviorObserved: Observed by The Build Hypothesis during the recorded run.What we saw during the recorded session.Ran through without asking for further input during the run.Asked for approval whenever it needed to run a new command. Commands it had already been allowed could be allowed again.Asked for approval once and offered a "run everything" option, which was selected.
Human interventionsAgent claim: Reported by the agent in its own build report. Not independently verified.Each agent defined "intervention" differently.None reported ("No human interventions occurred.")Counts tool approvals as interventions (dependency installs, Cloudflare provisioning and migrations, test runtime). No manual code changes."None required beyond tool permissions."
Requirements trackedAgent claim: Reported by the agent in its own build report. Not independently verified.Each agent split the spec into its own list, so these totals are not comparable.
73
  • 71 Verified (per agent)
  • 1 Implemented but unverified
  • 1 Partial
The agent split the spec into 73 requirements.
55
  • 54 Verified (per agent)
  • 1 Partial
The agent says "Verified" often means a local test with mocked services, not production verification.
24
  • 19 Verified (per agent)
  • 3 Implemented but unverified
  • 2 Partial
BUILD_RESULTS.json lists 24 requirements. The Markdown report's table has 27 rows (19 Verified, 5 Unverified, 3 Partial).
Automated testsAgent claim: Reported by the agent in its own build report. Not independently verified.Suites differ in scope (unit, browser, production smoke).
  • 236✓Unit and integration (Vitest in workerd, real D1, mocked external APIs)
  • 9✓ 1 skippedProduction browser tests (Playwright, real OpenAI, Places and email)
  • 13✓Production admin bootstrap activation
  • 20✓Production walkthrough (spec §14.5)
  • 39✓Worker API tests (Vitest in workerd, local D1, mocked externals)
  • 8✓Local browser tests (Playwright, desktop and mobile)
  • 8✓Production browser tests
  • 16✓Production admin browser checks
  • 22✓Production smoke checks (real OpenAI, Google and email binding)
  • 24✓Production bootstrap smoke
  • 15✓Unit tests (Vitest, local)
  • 6✓Production checks (health, config, guest itinerary, register and migrate, admin bootstrap, forgot password)
DeploymentAgent claim: Reported by the agent in its own build report. Not independently verified.Deployed through GitHub-connected Workers Builds, with the build verified for the deployed commit.Deployed through the GitHub-connected Workers Builds pipeline. Of 6 push builds during the run, 1 failed and 5 succeeded.Reports GitHub Workers Builds success for the deployed commit. The same executive summary also lists "GitHub Builds status to be verified post-push" as a gap.
Estimated API spend while testing in productionAgent claim: Reported by the agent in its own build report. Not independently verified.This reflects how much each agent tested in production, not the app's efficiency.$4.94Estimated from list prices for the agent's own production testing (67 LLM requests and 146 billable Places requests). About 99% of it is Places. This reflects how much the agent tested, not how efficient the app is.$0.066Estimated for the agent's own production checks (4 LLM calls and 2 Places calls).$0.078Sampled from the admin overview after smoke tests (last 7 days); an estimate.
Commits during the runGit record: Taken from the public repository's commit history.774
StackAgent claim: Reported by the agent in its own build report. Not independently verified.
  • TypeScript
  • Cloudflare Workers
  • Hono
  • Zod
  • React 19
  • React Router
  • Vite
  • Tailwind CSS 4
  • Leaflet + OpenStreetMap
  • Vitest
  • Playwright
  • TypeScript
  • Cloudflare Workers (no framework)
  • Zod
  • React 19
  • Vite 7
  • Leaflet + OpenStreetMap
  • @noble/hashes
  • Vitest
  • Playwright
  • TypeScript
  • Cloudflare Workers
  • Hono
  • Zod
  • React 19
  • React Router
  • Vite 7
  • react-leaflet + OpenStreetMap
  • Vitest
Public demoCurrent availability on this site.Demo not publicDemo not publicDemo not public
Independent scoreVerified: Confirmed by independent testing.Evaluation pending. No scores or winner have been published. This row will fill in once the hands-on testing is complete.

Build timelines

  • Commit
  • Problem
  • Fix
  • Deploy
  • Start / end
View the timeline as a table

Claude Code, started 22:28 UTC

OffsetTimeEventTypeSource
+0 min22:28 UTCRun startedStartAgent claim: Reported by the agent in its own build report. Not independently verified.
+2 min22:30 UTC ≈Found the existing DB binding pointed at a missing database, so created a new D1 databaseProblemAgent claim: Reported by the agent in its own build report. Not independently verified.
+42 min23:10 UTCImplement AI Travel Planner on Cloudflare Workers and D1CommitGit record: Taken from the public repository's commit history.
+50 min23:18 UTCClose guest quota gap, handle over-long trips, add browser journeysCommitGit record: Taken from the public repository's commit history.
+54 min23:22 UTCAllow the zone-injected Cloudflare Web Analytics beacon in the CSPFixGit record: Taken from the public repository's commit history.
+1h 4m23:31 UTCFix source badge on stops without a place; add README and production checksFixGit record: Taken from the public repository's commit history.
+1h 9m23:37 UTCOnly open the map when the traveler asksFixGit record: Taken from the public repository's commit history.
+1h 15m23:43 UTCStop rejecting correct Places matches because of bad model coordinatesFixGit record: Taken from the public repository's commit history.
+1h 25m23:53 UTCBuild report generatedEndAgent claim: Reported by the agent in its own build report. Not independently verified.
+1h 26m23:53 UTCAdd build report artifacts and a test for the daily cleanup jobCommitGit record: Taken from the public repository's commit history.

OpenAI Codex, started 16:36 UTC

OffsetTimeEventTypeSource
+0 min16:36 UTCRun startedStartAgent claim: Reported by the agent in its own build report. Not independently verified.
+33 min17:08 UTCImplement Elsewhere travel planner with secure Cloudflare backend and reportingCommitGit record: Taken from the public repository's commit history.
+33 min17:09 UTC ≈First Workers Build failed at npm ci (lockfile missing a peer entry)ProblemAgent claim: Reported by the agent in its own build report. Not independently verified.
+42 min17:17 UTCFix Cloudflare npm lock compatibility and verify responsive travel workspaceFixGit record: Taken from the public repository's commit history.
+44 min17:20 UTCFirst production smoke test returned HTTP 500 on admin login (PBKDF2 iteration limit)ProblemAgent claim: Reported by the agent in its own build report. Not independently verified.
+47 min17:22 UTCRetain full-strength password hashing on Cloudflare production runtimeFixGit record: Taken from the public repository's commit history.
+1h 4m17:40 UTCComplete production verification, reporting drill-downs and build handoffCommitGit record: Taken from the public repository's commit history.
+1h 10m17:45 UTCApply security headers to Cloudflare-served HTMLFixGit record: Taken from the public repository's commit history.
+1h 17m17:52 UTCAccount for deleted-trip usage in unattributed costsFixGit record: Taken from the public repository's commit history.
+1h 28m18:03 UTCBuild results generatedEndAgent claim: Reported by the agent in its own build report. Not independently verified.
+1h 28m18:04 UTCPublish final build reports and verified deployment handoffCommitGit record: Taken from the public repository's commit history.

Cursor CLI, started 19:01 UTC (approximate)

OffsetTimeEventTypeSource
+0 min19:01 UTC ≈Run started (approximate)StartAgent claim: Reported by the agent in its own build report. Not independently verified.
+19 min19:20 UTCImplement AI Travel Planner v1 on Cloudflare WorkersCommitGit record: Taken from the public repository's commit history.
+21 min19:22 UTCFix Workers Builds npm peer dependency resolutionFixGit record: Taken from the public repository's commit history.
+23 min19:23 UTCTrigger Workers Builds after configuring npm run buildDeployGit record: Taken from the public repository's commit history.
+25 min19:25 UTCRun endedEndAgent claim: Reported by the agent in its own build report. Not independently verified.
+25 min19:25 UTCUpdate build reports after verified GitHub Workers Builds successCommitGit record: Taken from the public repository's commit history.

What the evaluation will weigh

Delivery priorities, in order

Spec: A requirement from the versioned specification.
  1. 01Working planner
  2. 02Configured LLM
  3. 03Guest and registered users
  4. 04Secure auth and persistence
  5. 05Admin auth and account management
  6. 06Usage and cost tracking
  7. 07Admin dashboard
  8. 08Email
  9. 09Maps and research
  10. 10Responsive UI
  11. 11Automated tests
  12. 12Production deployment and verification
  • The spec lists these as delivery priorities, in order. It defines no numeric score and names no winner.
  • Requirement statuses use a fixed vocabulary: Verified, Implemented but Unverified, Partial, Failed, Not Started, Blocked.