CI coverage and efficiency

CI protects the distributable plugin and converter. Release automation does not provide a second validation gate, so platform-specific regressions still belong before merge.

The test job always runs release metadata validation, strict plugin schema validation, baseline Linux tests, and typecheck. PR title validation and the docs audit retain their existing coverage. Ordinary bun run test, main pushes, and manual dispatches run the full suite.

Linux test selection

PRs pass --changed-since=<base SHA> to the same test runner. scripts/test-scope.ts lists only expensive test groups; every other discovered test is baseline and runs on every PR. New tests therefore run without registration. Discovery honors bunfig.toml fixture exclusions. The runner uses exact file paths and retains parallel execution, lost-child recovery, timeout-only serial reruns, and failure propagation.

Changing a grouped test selects its entire group. Shared helpers and fixtures, runner/CI/dependency/configuration changes, unknown paths, and unavailable or empty comparisons run full coverage. Inputs inside an owning directory include prose and resources: the subprocess tests read these files as well as executing scripts. Outside those owners, converter source, ordinary documentation, ungrouped tests, and skill Markdown need only baseline coverage. Unrecognized skill executables run the full suite.

For local pre-push validation, use the intended PR base SHA on a clean, committed candidate:

bun run test --changed-since=<40-character-base-SHA>

This compares the base to HEAD. If tracked or untracked changes are present, or Git cannot establish a clean checkout, the runner falls back to full coverage so uncommitted inputs cannot escape selection. Selected mode accepts only the optional --timings=<path> and --update-timings Bun flags; other arguments are rejected so a caller cannot accidentally narrow the baseline. An ordinary bun run test always runs full coverage.

During implementation, use focused tests and skill guards for feedback. Before publishing, the shared selection (or a full run), typecheck, release validation, and strict plugin validation must pass. Skill behavior changes also need the fresh-agent evals required by the repository instructions. CI remains the merge gate, including platform checks that were not run locally.

Carry successful verification evidence through review and shipping while its inputs, selection base, and relevant environment remain unchanged. Record the command, result, and tested state; rerun checks whose evidence was invalidated by later changes. A commit-message-only amendment preserves evidence for checks that do not inspect commit metadata. Capture the runner’s actual exit status before formatting its output: a successful tail is not a successful test run. Missing or failing required checks block publication.

The runner prints selection mode, file counts, groups, and fallback reason, and copies them to the Actions summary. Linux starts independently of the specialized-job selector. Actions uploads Bun’s per-file timing data as linux-test-timings, including on test failure when available; a missing timing artifact does not turn a failing test green.

Specialized job selection

scripts/ci-scope.ts selects the Windows job and the acpx/macOS jobs for pull requests. It compares the PR base SHA with the checked-out merge tree, including deleted paths and both sides of renames. It uses local git rather than an API’s potentially truncated changed-file listing.

Markdown inside skills contains executable scratch-root preambles. Those need real Git Bash coverage: Linux previously passed a shell primitive that failed on Windows. The acpx and macOS jobs exercise scripts and their test fixtures, so prose-only changes do not require them. The allowlist covers independent inputs rather than enumerating every dependency of those jobs; new and unclassified paths run everything.

When the acpx job runs on a pull request, it tests only the supported range’s minimum and maximum unless the PR changes a contract input: the runner script, the contract test, the stub agent, the pin helper, the CI workflow, or package.json. Published acpx releases are immutable, so the middle of the range can only change verdict through those files or through acpx’s own dependencies drifting. Main pushes, manual dispatch, and unavailable comparisons test every release in range, which catches dependency drift after merge. The runner tests up to four releases at once (ACPX_CONTRACT_JOBS).

The selector publishes its decision in the workflow summary. If the selector job fails or produces no output, specialized jobs run. The Linux gate starts independently, so classification adds no dependency to its critical path. Specialized jobs retain their names for existing required-check configuration.

New commits cancel older CI runs for the same PR. Main pushes and manual dispatches have unique concurrency groups and are never coalesced. All jobs run on main, providing an ongoing check of the selection boundary, and manual dispatch is the full-coverage escape hatch.

Measuring further changes

Compare both elapsed time to green and aggregate runner time. Skipping a two-minute job saves runner capacity but does not shorten a three-minute independent Linux job. Count selector startup time against the savings. Compare similar changes and the same acpx range: expanding the supported range changes the network contract suite’s cost.

Use Actions job/step timestamps and the per-file timing recipe in CI wall-time notes. Benchmark on hosted runners before claiming speedups; laptop worker counts are not a reliable proxy. Existing history records an 8% hosted-runner gain from splitting a slow file where a laptop proxy predicted 26%.

Before narrowing selection further, check full main runs for failures in groups or jobs that the PR skipped and add a regression case to the relevant selector test. Keep every transitive input in its owner mapping or the full-coverage fallback. Broad guards that scan multiple skills and repository metadata stay in baseline; an import graph alone cannot capture their dependencies.

The initial replay of 50 commits ending at e86a51a744de2fc884752d86c817dc5351884fa2 selected baseline only for 10, some expensive groups for 9, and full coverage for 31. It omitted cross-model tests in 13 changes. Applying one local full run’s summed case times estimates 22.5% less aggregate test work across that sample. This is a workload estimate, not a hosted wall-time speedup. The earlier 45/50 figure described specialized jobs, not Linux selection, and is not the same metric.

Full fallbacks in that sample included eval infrastructure such as tests/skill-eval-cell/catalog.ts (10 commits), package.json (7), and release metadata (6). These counts overlap. Those paths remain shared or unclassified; full selection is a conservative boundary, not evidence that every group depends on every changed file.