How to integrate Copilot with CI for auto-PRs

Short answer / recommendation
Use CI to capture the failing test context, then call an LLM to produce a suggested patch, push that patch to a draft branch and open a draft PR. In practice, GitHub Copilot currently doesn't offer a stable public automation API for CI-triggered generation, so the reliable automation path is to call an LLM (e.g., ChatGPT/OpenAI) from Actions. If you must use Copilot interactively, keep it for developer-driven local fixes; for automated PRs use an API-based LLM.

Decision criteria
- Use Copilot interactively if your team prefers manual vetting and local IDE assistance.
- Use an LLM (ChatGPT/OpenAI) if you want fully automated PR generation in CI. Requires API keys and budget for API usage.
- Consider team size & QA: smaller teams should generate draft PRs only; larger teams can create PRs with CI checks and label them for triage.

What you need (permissions & secrets)
- GitHub Actions permissions: contents: write, pull-requests: write, checks: write (set at workflow or repo level). This allows creating branches, pushing commits, and creating draft PRs.
- Secrets: OPENAI_API_KEY (or other LLM key) and optionally GH_TOKEN (GITHUB_TOKEN usually suffices but give explicit write scopes if needed).
- A workflow user must be allowed to push branches (repo policy) and create PRs.

Example flow (high-level)
1) Run tests in CI. On failure, collect failing test names, stack traces, and relevant file diffs.
2) Call an LLM with a prompt that includes the failure context and asks for a patch + commit message + explanation.
3) Apply the patch to a new branch, commit and push.
4) Create a draft PR with the LLM’s explanation and mark it for review.

GitHub Actions + script examples
.github/workflows/auto-fix.yml (core parts)
- name: Run tests
run: |
set -o pipefail
pytest -q 2>&1 | tee test_output.log || true

- name: On failure: prepare fix
if: failure()
uses: actions/checkout@v4

- name: Generate and push fix
if: failure()
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
python .github/scripts/generate_fix.py --test-log test_output.log --branch auto/fix-${{ github.run_id }}

Example Python script (generate_fix.py) — outline
- collects test_output.log
- extracts failing tests, stack traces, file paths and a small patch context (git diff --name-only, plus head file content)
- composes a prompt (template below)
- calls OpenAI Chat Completions API (or your LLM) to request a patch in a unified-diff format
- applies the patch (git apply), commits, pushes branch (via git/gh), and uses GitHub API to open a draft PR

Prompt template (trim to token budget)
System: You are a code assistant that returns a unified diff and a short commit message for the repository provided.
User: Tests failing:
Stack traces:

Files referenced (provide 3-5 files w/ small context):
:
Task: Produce a minimal fix in unified diff format, a 1-line commit message, and a 2–3 sentence explanation. If uncertain, add TODO comments and do not change unrelated files. Keep changes minimal and runnable.

Best-for / Avoid-if
- Best for: teams that want fast triage, draft PRs, and human review. Works well when failing tests have clear stack traces and small fixes.
- Avoid if: you require 100% deterministic fixes or if tests are flaky; avoid auto-merging without human review.

Practical checklist before enabling
- [ ] Add OPENAI_API_KEY (or chosen LLM key) to repo secrets.
- [ ] Ensure Actions permissions: contents/pull-requests write.
- [ ] Limit generation token budget and set rate limits for CI runs.
- [ ] Add labels and reviewers in PR creation step.
- [ ] Add tests to validate generated patches in sandbox runs before enabling on main branches.

If you want, I can paste a ready-to-run generate_fix.py that calls OpenAI and the exact GitHub Action YAML. Note: I used ChatGPT/OpenAI for automation examples because Copilot does not provide a supported programmatic endpoint for unattended CI automation as of now.

Compare GitHub Copilot and Cursor