AstroAI: Review Evidence for Agent-Built Pull Requests

The problem

AstroAI moves quickly because most changes arrive as pull requests written by coding agents: 171 merged PRs between July 1 and October 31, 2026. With one reviewer, reading every diff doesn't scale. A green CI badge doesn't show what a user will actually see, and screenshots pasted into a PR can easily come from an older commit.

Reviews needed two properties. The evidence had to match the exact commit under review, and reviewing a feature had to take a couple of clicks rather than a local setup session.

Constraints

• Zero cost and fully offline. No real credential is used and nothing leaves localhost. OAuth, the chart API, geocoding, space weather, the LLM, and Stripe are each replaced by a local stand-in.
• Previews stay private: tailnet-only Tailscale Serve (never Funnel), a server bound to 127.0.0.1, isolated synthetic data, and production credentials stripped from the environment.
• A hosted CI runner can check the PR body but can't reach a private tailnet, so reachability has to be verified locally and shape in CI.
• The reviewer starts from a ready state and should never have to create a chart or generate a reading just to review an unrelated change.

What I built

• A headless Playwright suite (26 test files) that drives the real app through real routes, browser JS, and a throwaway SQLite database. A fake chart service still computes real planetary positions offline, and the app subprocess's environment is scrubbed of real keys.
• An evidence.step(...) context manager that records each step's intention, expected result, assertion outcome, and a screenshot at that exact boundary. Failures are recorded and then re-raised, never masked.
• A self-contained HTML evidence report with images embedded as data URIs, an accessible lightbox dialog, and a no-JS fallback. The report embeds the head SHA it was built from, and preview routes refuse to serve a report whose SHA doesn't match the running process.
• A separate feature-testing report, generated by running the PR's own numbered manual plan against the live seeded preview with a PR-specific Playwright driver.
• A review-mode launcher with a seeded synthetic reviewer (sign-in via /dev-login) and JSON seed manifests for richer states: extra users, subscription tiers, friendships, and ready synastry reports.
• A PR review contract with a deterministic validator, run as a GitHub Actions check. It requires the preview URL, links to both reports, the 40-character head SHA, verbatim safety statements, and a manual plan of at least two numbered steps, each with an expected visual result and an expected behavioral result.

How it's verified

• On main at commit 71b8ad7 (2026-10-31), CI ran 2,814 unit and integration tests and 62 browser end-to-end tests, and all of them passed.
• The contract validator has 59 tests. The evidence report has 11 markup tests, and the lightbox's keyboard and focus behavior is tested in a real browser over file://.
• CI uploads the full evidence directory as an artifact on every run, pass or fail.
• Stale-evidence guards: the metadata route reports the process's startup SHA, and both report routes refuse a missing report or one built from a different commit.

Results

• Every open PR links to a preview of its exact head commit, a regression evidence report, and a report from running its own manual plan. CI fails the contract check when any of these is missing, stale, or a placeholder.
• The whole review loop runs at zero external cost, because neither the browser suite nor the previews touch a paid API or a real credential.

How coding agents fit in

• This is the system that makes agent-written code reviewable. Agents open the PR, start the seeded preview, run the manual plan, and publish the reports. A person reviews evidence pinned to the exact commit instead of trusting the agent's summary.
• The contract itself lives in docs/pr-review-contract.md, so every agent session reads the same rules, including how to repair a PR body that a generator overwrote.