Short answer / recommendation
Use Copilot-suggested tests only as a first draft. Add automated checks in CI (linting, type checks, network/secret sandboxing, mutation testing or test-strength metrics) and a lightweight human-review checklist that requires an explicit test intent and a threat/security review before merging.

Why this works (one-line):
Copilot hallucinates APIs and insecure mocks because it predicts likely code, not what your repo actually exposes; the safeguards below catch mismatches or dangerous patterns before they land in main.

Decision criteria (pick what matters for your project)
- Budget / CI time: mutation testing and full sandboxing increase compute cost. If budget is tight, prioritize static checks and network-blocking.
- Skill level: teams comfortable with advanced CI can add mutation testing and contract tests; smaller teams should tighten code review and add simple linters.
- Workflow stage: prototypes can accept looser checks; production/security-critical code needs stricter pipelines.
- Team size: larger teams can rely more on human review; small teams should compensate with stronger automated checks.

Concrete checklist to reduce and catch hallucinated or insecure tests
1) Treat AI tests as “drafts” only
- Require PRs that include a short rationale for each generated test (what behavior is it verifying?). If intent is unclear, request changes.
2) Add fast CI gates (run on PR)
- Type checking (mypy/TypeScript) to spot calls to non-existent members.
- Lint rules that forbid common unsafe mock patterns (e.g., mocking private internals, monkeypatching global state, disabling auth checks).
- Secret scanning to catch accidentally pasted credentials.
3) Sandbox test execution
- Run PR tests in an isolated container with network disabled and read-only mounts to detect external calls or side effects. Fail PRs that reach out to external services.
4) Strengthen test quality signals
- Coverage thresholds (but avoid over-reliance).
- Mutation testing (Stryker/Mutmut) occasionally or on nightly builds to find weak/false-positive tests.
- Use contract or integration tests for critical flows instead of purely mocked unit tests.
5) Static analysis for suspicious patterns
- Add rules that flag e.g., assert true/false, empty assertions, overly broad mocks (Mock() used where a typed fake would be better).
6) Human-review checklist on PRs
- Does the test assert behavior, not implementation? Is the fixture realistic? Does it create/restore global state? Are any auth/network calls artificially bypassed? Reviewer must mark each point.
7) CI policy and automation
- Require the “AI-generated” label and a mandatory reviewer for such PRs.
- Optionally block merges unless mutation score or contract tests pass for critical modules.

Best-for / Avoid-if
- Best-for: teams who want to boost test coverage quickly and have reviewers to vet tests. Copilot is great for scaffolding assertions and fixtures.
- Avoid-if: you have strict security/safety requirements and no capacity for review or sandboxing — don’t merge AI-suggested tests blindly.

Tool note (practical tip)
If you use GitHub Copilot to generate tests, pair it with a short comment prompt (example: “Generate a unit test for X that asserts behavior Y and does not call external services”) and use ChatGPT only to explain the intent of a suggested test before reviewer approval.

Quick checklist you can paste into PR template
- [ ] Intent described for each AI-generated test
- [ ] Type/lint checks pass
- [ ] Tests run in network-disabled sandbox
- [ ] No secrets included
- [ ] Reviewer confirms test asserts behavior, not implementation

If you want, I can suggest concrete linter rules, a pre-commit script, or a minimal GitHub Actions workflow to implement the sandbox+type-check gate.

Compare GitHub Copilot and Cursor