GitHub Copilot workflow: generate PR summaries from code

Recommendation
Use GitHub Copilot for tight, per-file inline summaries and quick commit-message suggestions in the editor, and run a GitHub Action that calls a chat model (ChatGPT / OpenAI) at PR creation to produce the PR body, changelog, and reviewer suggestions. Keep the Action deterministic (temperature=0), feed only curated context (commits, diff hunks, CODEOWNERS, recent reviewers), and always post an editable draft comment rather than auto-merge based on the AI output.

Why this works
- Copilot = fast developer-facing hints during authoring. ChatGPT (via Actions) = reliable multi-file compilation and natural-language synthesis. Use both to reduce friction and keep a human in the loop.

Decision criteria
- Choose Copilot-only when you want inline help and your team prefers editor-integrated suggestions.
- Add an Action + ChatGPT when you need a single authoritative PR summary, changelog, and reviewer list across many files/commits.
- Consider budget: ChatGPT API calls cost per usage; larger repos/diffs raise cost. Team size matters: larger teams benefit more from automated reviewer suggestions.

Concrete prompt (system + user) to reduce hallucinations
System: You are an assistant that summarizes code changes. ONLY use the following explicit context sections. If asked for anything not present, respond with INS UFFICIENT_CONTEXT and list what’s missing. Cite file paths and specific commit SHAs if referenced.
User: Context sections: COMMITS, DIFFS, FILE_LIST, CODEOWNERS, RECENT_REVIEWERS. Produce: 1) one-paragraph PR summary (why/what changed), 2) bullet changelog grouped by file/type (with file paths, function/class names if present), 3) reviewers (explain reason: code ownership or recent reviewer), 4) short checklist of risk/impact and testing notes. Keep temperature 0. Limit output to 700 words.

GitHub Action (outline YAML)
- name: PR-summary
on: pull_request_target: types: [opened, synchronize, reopened]
jobs:
summarize:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Gather context
run: |
echo "COMMITS< context.txt
git --no-pager log --format="%H|%an|%ae|%s" ${{ github.event.pull_request.base.sha }}..${{ github.event.pull_request.head.sha }} >> context.txt
echo "EOF" >> context.txt
git --no-pager diff --unified=0 ${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }} > diff.patch
echo "DIFFS<> context.txt
head -c 200000 diff.patch >> context.txt
echo "EOF" >> context.txt
cat .github/CODEOWNERS || true >> context.txt
- name: Call ChatGPT
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: |
payload=$(jq -n --arg ctx "$(sed -n '1,200p' context.txt)" '{model: "gpt-4o-mini", messages:[{role:"system",content: "[system text here]"},{role:"user", content: $ctx}], temperature:0, max_tokens:1200}')
curl -s https://api.openai.com/v1/chat/completions -H "Authorization: Bearer $OPENAI_API_KEY" -H "Content-Type: application/json" -d "$payload" > ai.json
cat ai.json | jq -r '.choices[0].message.content' > pr_summary.md
- name: Post summary as comment
uses: peter-evans/create-or-update-comment@v3
with:
token: ${{ secrets.GITHUB_TOKEN }}
issue-number: ${{ github.event.pull_request.number }}
body-file: pr_summary.md

Ways to reduce hallucination and missing context
- Always feed exact artifacts: commit list, unified diff (trim to changed hunks), filenames, CODEOWNERS, recent reviewers, last 3 committers. Do not let the model browse the repo beyond the provided snippets.
- Use a clear system prompt: "Only use the provided CONTEXT sections." Return INS UFFICIENT_CONTEXT if missing items.
- Temperature 0, max_tokens limited. Ask model to cite file paths and SHAs for claims.
- Chunk large diffs: summarize hunks server-side (simple regex) and only pass changed functions or top-level context lines.
- Human-in-loop: post draft comment and label PR as auto-summary:pending-review. Require a maintainer to accept before automations (merge, release notes) consume it.

Checklist to implement
- [ ] Add OpenAI API key to repo secrets
- [ ] Add Action file (above) and test on a small PR
- [ ] Tune the system prompt and chunking thresholds
- [ ] Add CODEOWNERS and map reviewer logic (fallback: last committers or frequent reviewers)
- [ ] Keep an explicit INSUFFICIENT_CONTEXT path and surface missing items as comment

Best-for / Avoid-if
- Best for: teams that need consistent PR blurbs, multi-file changes, and reviewer recommendations. Scales for medium+ teams.
- Avoid if: extremely sensitive private code where sending diffs to an external API is disallowed, or if you need zero-cost automation and only want in-editor hints.

If you want, I can produce a tested, copy-pasteable Action with a stronger reviewer-selection algorithm (blame+CODEOWNERS parsing).

Compare GitHub Copilot and Cursor