Skip to content

ponytail-audit: no pass to flag AI-shaped test brittleness (fixture-name branches in prod, snapshot/mock-only assertions, 1:1 method-mirror tests) - #733

Open
Lockyer228 wants to merge 1 commit into
DietrichGebert:mainfrom
Lockyer228:feat/683-audit-test-brittleness
Open

ponytail-audit: no pass to flag AI-shaped test brittleness (fixture-name branches in prod, snapshot/mock-only assertions, 1:1 method-mirror tests)#733
Lockyer228 wants to merge 1 commit into
DietrichGebert:mainfrom
Lockyer228:feat/683-audit-test-brittleness

Conversation

@Lockyer228

Copy link
Copy Markdown

Why

ponytail's test stance is "non-trivial logic leaves one runnable check," but ponytail-audit — scanning existing code — has no pass for AI-shaped test slop. Distinct from #602 (which constrains the agent while writing tests), this is about detecting brittleness in tests that already exist.

What

Add a test: tag and a Hunt entry flagging three failure modes:

  • tests that mirror a method 1:1 (break on any refactor; they assert structure, not behavior)
  • snapshot-only or mock-call-only assertions (they pin the AI's own output, not observable behavior)
  • worst: fixture-name / id / timestamp branches leaking into production code to make a test pass

Plus a Boundaries clarification that test brittleness is dead weight to cut, not a correctness fix — keeping audit's "correctness out of scope" stance intact.

Tests

  • node --test tests/openclaw-skills.test.js → 18/18
  • node scripts/check-rule-copies.js → exit 0
  • npm test → all pass except the pre-existing csv: correct pandas one-liner passes failure (fails identically on base; no pandas installed locally) — unrelated to this change.

Fixes #683

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ponytail-audit: no pass to flag AI-shaped test brittleness (fixture-name branches in prod, snapshot/mock-only assertions, 1:1 method-mirror tests)

1 participant