troubleshooting: GitHub Copilot hallucinating during refactors

Recommendation
Use Copilot for local, small-surface refactors (boilerplate, repetitive edits) but gate every AI-suggested change to automated tests + a human reviewer. Combine editor-side controls, repo-level linters/policies, and PR-process signals to make hallucinations and unsafe patterns visible and blockable.

Why this helps (short)
Copilot is context‑driven and will guess missing intent. Grounding the editor (types, docs, stubs), adding automated checks, and forcing manual PR review turns guesses into verifiable outputs rather than accepted code.

Configuration tactics (editor + repo)
- Limit suggestion scope: prefer selection-based completions (select the region you want changed) or accept single-line suggestions rather than whole-file rewrites. This reduces context collapse that causes hallucinations.
- Add strong typing and canonical imports: TypeScript typings or explicit signatures make incorrect API shapes fail at compile time. For dynamic languages, add type hints (pyright, mypy stubs) where possible.
- Make unsafe APIs explicit: add linter rules (ESLint custom rules, flake8 plugins) that flag deprecated/insecure APIs and fail CI. Add those rules to pre-commit and CI so suggested code cannot merge if it uses forbidden functions.
- Repo-level docs header: add a short comment block in sensitive modules with “Allowed APIs / Forbidden APIs / Security notes” so Copilot uses local doc context when generating suggestions.
- Enable server-side scanning: require SAST, dependency-scan, secret-scan jobs in PR pipelines before merge.

Prompt & in-file prompt patterns
- Use short, explicit comments to instruct Copilot, e.g. "// Copilot: use internalHttpClient.request(), do not use global fetch; validate inputs" before the function you want changed.
- Ask for one concern at a time: “Rename variables and keep behavior identical” vs “replace insecure auth flow.” Multi-goal prompts increase hallucination risk.
- For large refactors, write a small unit test first that documents the behavior, then ask Copilot to change code so tests still pass.

Review strategies (process)
- PR metadata: require an "AI-suggested" checkbox and a reviewer with seniority to sign off.
- Automated checks: block merge until linters and unit tests pass. Add an extra security test-run for PRs touching auth/crypto/networking.
- Pair-review: for security-sensitive code, review PRs in person (pair or mob) or route to the relevant domain owner (CODEOWNERS).
- Sample audits: periodically scan merged AI-suggested commits for patterns of insecure code and update linters accordingly.

Decision criteria: when to trust Copilot vs not
- Trust for: boilerplate, repetitive renames, test scaffolding, small API adapters where types exist.
- Don’t trust for: auth, crypto, input validation, permission checks, policy logic, or unfamiliar external APIs.
- Depends on budget/team: paying for organization-level Copilot (or enterprise) gives better admin controls; larger teams should invest in CI guards and a security reviewer role.

Practical checklist (before merging any AI-suggested refactor)
- [ ] Did CI pass (unit, integration, security scans)?
- [ ] Is there a failing linter rule for forbidden APIs? Fixed?
- [ ] Is the change covered by tests or did you add tests?
- [ ] Is the PR labeled "AI-suggested" and approved by an experienced reviewer?
- [ ] Any new dependencies introduced? Are they scanned/approved?
- [ ] Add/update a rule if Copilot keeps suggesting a bad pattern.

Best-for / Avoid-if
- Best for: rapid boilerplate, consistent renames, generating repetitive code when strong typing exists.
- Avoid if: code touches security/performance correctness, or when you need API-accurate calls to unfamiliar services.

If you want, start by adding the linter rule and the CI gate (fast, high ROI). If you have budget and need org-wide control, consider Copilot Business/Enterprise for admin settings and audit logging.

Mentioned tools: GitHub Copilot

Compare GitHub Copilot and Cursor