Senior Engineer Benchmark

Starting snapshot

Version 1.0 uses a frozen Proof source snapshot from March 17, 2026. The workspace includes production collaboration code and local incident context from an outage where public routes timed out while collaboration work kept running inside the process. One frozen snapshot keeps later fixes and repository drift from changing the task between runs.

That snapshot was reinitialized as a local Git repository with no remote. Agents could use only files and Git history inside the workspace. GitHub, web search, sibling checkouts, archived sessions, shell history, production logs, deployment systems, CLI fetches, and newly added remotes were prohibited so models could not read later human fixes or information unavailable at the historical starting point.

Prompt and follow-ups

Every model received the same opening prompt and the same frozen codebase. After the opening response, runs diverged: Each model got different continuation, implementation, and review instructions. Those instructions appear below without normalization. They are part of the experimental conditions and a known source of variance.

The benchmark compares work from the same starting state and rubric under non-uniform intervention. GPT-5.5 executed a plan written by Claude Opus 4.7. Several runs were pushed to continue after incomplete claims. The Claude Opus 5, GPT-6 Astra, and Claude Fable 5.1 runs received no follow-up instructions: Each was a single pass from the opening prompt.

The collaboration problem

The benchmark centers on Proof’s live editing subsystem. Users co-edit documents in real time; every change must converge on one authoritative document state. During the motivating outage, several parallel sources of truth—database rows, projection tables, repair and reconcile paths, and the live document—could all mutate what counted as the document. That produced crashes, duplication, and data inconsistency.

The editor surface runs on ProseMirror. Real-time collaboration syncs through Yjs over WebSocket. A senior rewrite picks one load-bearing authority invariant—for example, the live ProseMirror fragment and Yjs document become the runtime content authority, with markdown and projections demoted to derived views—then deletes parallel machinery and routes every write surface through that authority while keeping incident knowledge intact. The rubric scores that authority rewrite. Incremental patches and self-authored test passes alone are insufficient.

What each run includes

Each official result identifies its benchmark version and model configuration. Reviewers read the architecture plan, implementation diff, which authority paths were deleted versus left reachable, how REST routes, agent edits, WebSocket sessions, and mark updates reach the new authority, test output, and any gaps the model documented.

Test counts are supporting evidence. A run can report many passing tests and still lose points if those tests skip the production path, preserve parallel authorities, leave the old architecture reachable, or fail to show the benchmark invariants under concurrency, restart, and failure conditions.

Scoring rubric

All leaderboard runs use the same 100-point rubric: invariant clarity and plan quality (20); simplification outcome (25); end-to-end wiring (10); removal of stubs and parallel authority (17); correctness proof and incident preservation (25); runtime durability and lifecycle behavior (3). Dimensions and point ceilings stay fixed so reviewers cannot redefine success to fit one implementation.

Reviewers reward a clear authority invariant in a reviewable plan; a smaller system with old write paths deleted; every production entry point wired through that authority; proof that incident regressions still pass; and durability across restart and lifecycle transitions. Models often name the right invariant early and still lose points when they drop incident scar tissue during the rewrite.

Official totals come from rubric adjudication. Test pass rates do not set the total. Human calibrations at 96 and 89 anchor the scale: One reference rewrote inside existing production constraints with strong incident replay; another rebuilt a smaller kernel from scratch with a clearer invariant but thinner brownfield coverage. Mid-90s scores mark the senior baseline for this task; high 80s mean credible architecture with gaps in wiring, authority shrinkage, or incident preservation. Where multiple rubric graders are documented, aggregation is preserved—GPT-6 Astra’s official 71 is the mean of three grader scores, Claude Opus 5’s official 54 is the median, and each Claude Fable 5.1 total aggregates three grader scores.

Version controls

Results compare only within the selected benchmark version. One ungraded frame-shift proposal is excluded from this version 1.0 leaderboard. Internal source records keep immutable run provenance so a score cannot silently move to a later implementation by the same model.

Limitations

The leaderboard generally shows selected official results rather than averages across repeated trials. Where repeated runs exist, their separate scores appear as a range. It does not estimate confidence intervals. Model families, reasoning settings, tool surfaces, and follow-up instructions also differ. Those differences are visible on the page but were not experimentally removed.

Official totals are canonical. The 96-point human reference uses its measured category scores. The model dimension table and radar charts are rubric-based reconstructions from the checked-in evidence, constrained to sum to each official total because the original model grader worksheets were not published. The widest gap between model runs and human references tends to sit in incident replay and authority surface shrinkage. Naming the right invariant on paper rarely closes it.