Troubleshooting: Copilot hallucinating project-specific SDK calls

Short answer / recommendation
Make the correct SDK surface visible to the editor and the model, then catch hallucinations with fast guards in CI. That is: ship accurate type signatures/docs/examples into the workspace (or install the SDK in editable form), add lightweight static checks/tests, and provide snippet templates so Copilot suggests only valid calls. If you have budget, consider a private/tuned model or Copilot for Business policies.

Why this works (two causes of hallucinations)
- Copilot/completion models predict text based on context. If the editor can’t see your SDK’s source/type info or canonical examples, the model invents plausible APIs.
- Automated builds catch these errors late. Fast local/CI checks stop them earlier.

Practical strategies (concrete)
1) Make the SDK visible to the editor and Copilot
- Add the SDK source to the workspace (git submodule or monorepo) or pip install -e ./internal_sdk in your virtualenv so completions use real symbols.
- Ship .pyi/type stubs or full type annotations (PEP 484) for public APIs. Typed signatures and docstrings drastically reduce invented method names.
- Put canonical usage examples in a top-level README or an examples/ directory; small, explicit snippets are prime completion context.

2) Create authoritative surface files
- Add small interface files that re-export only public functions/classes (e.g., internal_sdk/__init__.py with explicit imports). These act as the single visible “catalog” for completions.
- Add docstring examples and short usage templates in the same file — completions often copy examples.

3) Editor-side tooling and templates
- Add editor snippets or code templates for the most common calls so devs accept correct completions instead of free-form ones.
- Encourage developers to open the SDK’s source or docs in the same workspace before writing code.

4) Fast automated guards
- Run type checking (mypy/TypeScript compiler), unit smoke tests, and build/test in a pre-commit/PR check to block hallucinated calls.
- Add a simple static linter or AST check that flags usages of unknown symbols or calls that don’t resolve to your package (a grep+import-check or small Python script works).

5) Longer-term / higher-budget options
- Use Copilot for Business / private model options or train a private code model on your repo so completions reflect your APIs (requires budget and infra).
- Maintain an internal code completions plugin or language-server indexing of internal libraries.

Decision criteria
- Low budget / small team: start by adding editable installs, stubs, README examples, snippets, and a CI type-check. High ROI, low cost.
- Medium budget: add linters/AST checks and more examples; enforce in pre-commit/CI.
- High budget / enterprise: evaluate private model tuning or Copilot for Business restrictions if completions must be tightly controlled.

Best-for / Avoid-if
- Best for: teams that want quick, low-friction improvement with minimal infra changes (use stubs + examples + CI).
- Avoid if: you expect a 100% deterministic generation guarantee — models will still suggest oddities unless the exact API is present in context.

Practical checklist (apply now)
- [ ] Add SDK source or pip install -e into developer envs
- [ ] Add .pyi/type hints and docstrings for public API
- [ ] Add examples/usage snippets and editor snippets
- [ ] Add mypy/typecheck and unit smoke tests in CI
- [ ] Add a pre-commit hook or PR job to block unknown-symbol calls
- [ ] If needed, evaluate Copilot for Business / private model options

Tools that help: GitHub Copilot works best when the SDK is in the workspace; if you want conversational help to design stubs/templates, ChatGPT can draft those artifacts. Quality score: 92

Compare GitHub Copilot and Cursor