EXP-001 · comparison
AI Travel Planner
3 agents, one spec (v1.7). Every row below shows where its data comes from. Independent scores have not been published yet, so nothing here declares a winner.
Read before comparing
- Agent claim: Reported by the agent in its own build report. Not independently verified. The agents split the same spec into very different requirement lists (73, 55 and 24 items), so the counts are not comparable.
- Agent claim: Reported by the agent in its own build report. Not independently verified. All three ran into Cloudflare's production limit of 100,000 PBKDF2 iterations for password hashing, and each handled it differently.
- Agent claim: Reported by the agent in its own build report. Not independently verified. The agents counted tests, requirements and interventions differently. Raw totals don't compare like for like.
| Metric | Claude CodeAnthropic | OpenAI CodexOpenAI | Cursor CLICursor |
|---|---|---|---|
| Run lengthAgent claim: Reported by the agent in its own build report. Not independently verified.Elapsed time the agent recorded in BUILD_RESULTS.json. | 1h 25m5,125 seconds, measured until the report was generated. The final commit and build check came after. | 1h 28m5,262 seconds, measured when the report was generated. The end time was left empty on purpose while publishing was still in progress. | 24.5 min1,470 seconds. The start time is given as approximate ("~19:01 UTC"). |
| Approval behaviorObserved: Observed by The Build Hypothesis during the recorded run.What we saw during the recorded session. | Ran through without asking for further input during the run. | Asked for approval whenever it needed to run a new command. Commands it had already been allowed could be allowed again. | Asked for approval once and offered a "run everything" option, which was selected. |
| Human interventionsAgent claim: Reported by the agent in its own build report. Not independently verified.Each agent defined "intervention" differently. | None reported ("No human interventions occurred.") | Counts tool approvals as interventions (dependency installs, Cloudflare provisioning and migrations, test runtime). No manual code changes. | "None required beyond tool permissions." |
| Requirements trackedAgent claim: Reported by the agent in its own build report. Not independently verified.Each agent split the spec into its own list, so these totals are not comparable. | 73
| 55
| 24
|
| Automated testsAgent claim: Reported by the agent in its own build report. Not independently verified.Suites differ in scope (unit, browser, production smoke). |
|
|
|
| DeploymentAgent claim: Reported by the agent in its own build report. Not independently verified. | Deployed through GitHub-connected Workers Builds, with the build verified for the deployed commit. | Deployed through the GitHub-connected Workers Builds pipeline. Of 6 push builds during the run, 1 failed and 5 succeeded. | Reports GitHub Workers Builds success for the deployed commit. The same executive summary also lists "GitHub Builds status to be verified post-push" as a gap. |
| Estimated API spend while testing in productionAgent claim: Reported by the agent in its own build report. Not independently verified.This reflects how much each agent tested in production, not the app's efficiency. | $4.94Estimated from list prices for the agent's own production testing (67 LLM requests and 146 billable Places requests). About 99% of it is Places. This reflects how much the agent tested, not how efficient the app is. | $0.066Estimated for the agent's own production checks (4 LLM calls and 2 Places calls). | $0.078Sampled from the admin overview after smoke tests (last 7 days); an estimate. |
| Commits during the runGit record: Taken from the public repository's commit history. | 7 | 7 | 4 |
| StackAgent claim: Reported by the agent in its own build report. Not independently verified. |
|
|
|
| Public demoCurrent availability on this site. | Demo not public | Demo not public | Demo not public |
| Independent scoreVerified: Confirmed by independent testing. | Evaluation pending. No scores or winner have been published. This row will fill in once the hands-on testing is complete. | ||
Build timelines
- Commit
- Problem
- Fix
- Deploy
- Start / end
View the timeline as a table
Claude Code, started 22:28 UTC
| Offset | Time | Event | Type | Source |
|---|---|---|---|---|
| +0 min | 22:28 UTC | Run started | Start | Agent claim: Reported by the agent in its own build report. Not independently verified. |
| +2 min | 22:30 UTC ≈ | Found the existing DB binding pointed at a missing database, so created a new D1 database | Problem | Agent claim: Reported by the agent in its own build report. Not independently verified. |
| +42 min | 23:10 UTC | Implement AI Travel Planner on Cloudflare Workers and D1 | Commit | Git record: Taken from the public repository's commit history. |
| +50 min | 23:18 UTC | Close guest quota gap, handle over-long trips, add browser journeys | Commit | Git record: Taken from the public repository's commit history. |
| +54 min | 23:22 UTC | Allow the zone-injected Cloudflare Web Analytics beacon in the CSP | Fix | Git record: Taken from the public repository's commit history. |
| +1h 4m | 23:31 UTC | Fix source badge on stops without a place; add README and production checks | Fix | Git record: Taken from the public repository's commit history. |
| +1h 9m | 23:37 UTC | Only open the map when the traveler asks | Fix | Git record: Taken from the public repository's commit history. |
| +1h 15m | 23:43 UTC | Stop rejecting correct Places matches because of bad model coordinates | Fix | Git record: Taken from the public repository's commit history. |
| +1h 25m | 23:53 UTC | Build report generated | End | Agent claim: Reported by the agent in its own build report. Not independently verified. |
| +1h 26m | 23:53 UTC | Add build report artifacts and a test for the daily cleanup job | Commit | Git record: Taken from the public repository's commit history. |
OpenAI Codex, started 16:36 UTC
| Offset | Time | Event | Type | Source |
|---|---|---|---|---|
| +0 min | 16:36 UTC | Run started | Start | Agent claim: Reported by the agent in its own build report. Not independently verified. |
| +33 min | 17:08 UTC | Implement Elsewhere travel planner with secure Cloudflare backend and reporting | Commit | Git record: Taken from the public repository's commit history. |
| +33 min | 17:09 UTC ≈ | First Workers Build failed at npm ci (lockfile missing a peer entry) | Problem | Agent claim: Reported by the agent in its own build report. Not independently verified. |
| +42 min | 17:17 UTC | Fix Cloudflare npm lock compatibility and verify responsive travel workspace | Fix | Git record: Taken from the public repository's commit history. |
| +44 min | 17:20 UTC | First production smoke test returned HTTP 500 on admin login (PBKDF2 iteration limit) | Problem | Agent claim: Reported by the agent in its own build report. Not independently verified. |
| +47 min | 17:22 UTC | Retain full-strength password hashing on Cloudflare production runtime | Fix | Git record: Taken from the public repository's commit history. |
| +1h 4m | 17:40 UTC | Complete production verification, reporting drill-downs and build handoff | Commit | Git record: Taken from the public repository's commit history. |
| +1h 10m | 17:45 UTC | Apply security headers to Cloudflare-served HTML | Fix | Git record: Taken from the public repository's commit history. |
| +1h 17m | 17:52 UTC | Account for deleted-trip usage in unattributed costs | Fix | Git record: Taken from the public repository's commit history. |
| +1h 28m | 18:03 UTC | Build results generated | End | Agent claim: Reported by the agent in its own build report. Not independently verified. |
| +1h 28m | 18:04 UTC | Publish final build reports and verified deployment handoff | Commit | Git record: Taken from the public repository's commit history. |
Cursor CLI, started 19:01 UTC (approximate)
| Offset | Time | Event | Type | Source |
|---|---|---|---|---|
| +0 min | 19:01 UTC ≈ | Run started (approximate) | Start | Agent claim: Reported by the agent in its own build report. Not independently verified. |
| +19 min | 19:20 UTC | Implement AI Travel Planner v1 on Cloudflare Workers | Commit | Git record: Taken from the public repository's commit history. |
| +21 min | 19:22 UTC | Fix Workers Builds npm peer dependency resolution | Fix | Git record: Taken from the public repository's commit history. |
| +23 min | 19:23 UTC | Trigger Workers Builds after configuring npm run build | Deploy | Git record: Taken from the public repository's commit history. |
| +25 min | 19:25 UTC | Run ended | End | Agent claim: Reported by the agent in its own build report. Not independently verified. |
| +25 min | 19:25 UTC | Update build reports after verified GitHub Workers Builds success | Commit | Git record: Taken from the public repository's commit history. |
What the evaluation will weigh
Delivery priorities, in order
Spec: A requirement from the versioned specification.- 01Working planner
- 02Configured LLM
- 03Guest and registered users
- 04Secure auth and persistence
- 05Admin auth and account management
- 06Usage and cost tracking
- 07Admin dashboard
- 08Email
- 09Maps and research
- 10Responsive UI
- 11Automated tests
- 12Production deployment and verification
- The spec lists these as delivery priorities, in order. It defines no numeric score and names no winner.
- Requirement statuses use a fixed vocabulary: Verified, Implemented but Unverified, Partial, Failed, Not Started, Blocked.