Short answer
Use layered automatic checks: (1) static analysis/type checks, (2) test discovery/import-only collection, (3) fast CI “run tests” with new/changed tests only, plus lightweight heuristics to flag likely hallucinations (undefined names, TODOs, NotImplemented). Failing any of those blocks the PR.

Recommendation
Add pre-commit hooks + a CI pipeline with three fast gates: lint/typecheck -> pytest/jest --collect-only -> run only generated/changed tests. Enforce a human review for generated tests that pass automated gates but include suspicious patterns. This catches most hallucinations before merging.

Why this works
Hallucinations usually manifest as undefined names, bad imports, or assertions that never exercise real code. Static linters and typecheckers catch undefined references and bad imports early. Collect-only test discovery detects import-time errors and broken test structure. Running targeted tests validates runtime existence and behavior without executing your whole suite.

Decision criteria (pick based on team needs)
- Fast feedback, small team, limited infra: pre-commit + lint/typecheck + --collect-only in CI. Cheap and quick.
- Medium team, frequent PRs: add per-PR job that runs only changed/generated tests in a containerized sandbox.
- High-reliability, regulatory or production-critical: keep the above and add full test run + mutation testing to measure test quality (more compute and time).
- Language tradeoffs: TypeScript/JS gets most benefit from tsc/ESLint no-undef; Python benefits from mypy/flake8/pyflakes and pytest --collect-only.

Practical checklist to add to repo/CI
1) Pre-commit: run linters and typecheck (mypy/flake8 for Python; tsc + ESLint for TS). Reject commits with undefined names, missing imports, or type errors.
2) Pre-merge CI stage A: lint & typecheck (fast).
3) Pre-merge CI stage B: test discovery/collection
- Python: pytest --maxfail=1 --collect-only (fails on import-time errors)
- JS: jest --listTests or a dry-run import check
4) Pre-merge CI stage C: run only generated or changed tests
- Use git diff to find new/modified test files, run pytest or jest
- Fail fast (maxfail=1), capture stack traces to surface undefined names and missing imports.
5) Automated heuristics: scan generated tests for patterns: TODO, FIXME, assert False, raise NotImplementedError, magic strings like "nonexistent"; mark PRs needing human attention.
6) Human review gate: require a reviewer for generated tests even if CI passes. Reviewer checks test intent and that mocks/spies are realistic.
7) Optional for high quality: add mutation testing (mutmut/pytest-mutation or Stryker for JS) and flaky-test detection.

Best-for / Avoid-if
- Best-for: teams that accept quick CI runtime and want to prevent noisy broken tests on main branches. Works for both small and large codebases.
- Avoid-if: you have zero CI time budget or extreme test-suite runtime constraints — then prioritize lint/typecheck + strict PR review instead.

Example commands (quick)
- Python lint: pre-commit -> flake8 && mypy
- Collect-only: pytest --maxfail=1 --collect-only
- Run changed tests: git diff --name-only origin/main...HEAD | grep -E 'test_.*.py' | xargs pytest

Notes on cost and effort
Linters and collect-only are cheap. Running many tests or mutation testing costs CI minutes. Start with linters + collect-only + changed-test runs, then raise the bar if you still see hallucinations.

If you want, I can draft a pre-commit config + minimal GitHub Actions workflow that implements these gates for Python or TypeScript.

Compare ChatGPT and Gemini