Contract-test specs MUST probe live API shape before encoding expectations
When writing any contract-shape assertion in a smoke/regression spec, curl the live endpoint once and encode the actual shape — never assume RFC 7807 envelope, status code, or field names by convention. Example failure: R-Dash nightly-smoke test 4 assumed MFA-required returns 4xx + {title: 'RDASH-AUTH-MFA-REQUIRED'} (RFC 7807 error envelope), but the actual contract is HTTP 200 + {requires_mfa: true, mfa_challenge_token: "<jwt>"} (challenge-continuation pattern, design choice). Two GHA runs wasted before live-probing revealed the design intent.
Behavioral rule: any new expect(body.title).toBe(...) or expect(res.status()).toBe(401) in a contract test must be preceded by curl/probe evidence proving the shape was observed, not guessed. If the endpoint can return multiple valid shapes (200 happy + 4xx error + 200 challenge), the spec accepts ALL valid shapes.
Generalises to: every “I know the API should return X” assumption is a future broken test. Cheap mitigation (one curl call) prevents expensive CI debugging.
Reference: r-dash commit e9fa0bc (main), GHA runs 25962617127 (failed) → 25962685301 (passed after fix).
Related
- researcher-to-nova-20260503-152531-cro-heartbeat-drift
- snowflake-mcp-v22-retro-bible-law-11-r38-r40-upstream-verifi
- r-dash-wave-2-pass-1-m4-semantic-m5-governance-models-rls-en
- r-dash-wave-2-code-complete-closeout
- nightly-post-deploy-smoke-is-live-in-ci-on-main-end-to-end-v
- pre-push-ci-gate-for-python-ruff-format-can-introduce-mypy-e