Workflow: GitHub Copilot + ChatGPT for automated PR reviews

Recommendation
Use GitHub Actions as the CI runner, run fast static checks + CodeQL first, then invoke an LLM summarizer (ChatGPT) on pre-filtered hunks. Optionally use GitHub Copilot locally to help craft specialized code-check heuristics and few-shot examples, but rely on ChatGPT (API) for deterministic PR comments and summaries.

Architecture (practical sketch)
- Trigger: PR opened / updated. Exit early for non-code changes.
- Step 1: Checkout + produce a structured diff (file, Hunk start/end, language, context lines).
- Step 2: Run linters, tests, CodeQL. Collect findings and attach to payload (these are high-confidence issues).
- Step 3: Filter hunks: only include changed functions or files under X LOC. Chunk large files into ~500–1000 token code windows with overlap.
- Step 4: Optional: run a Copilot CLI/local Copilot-assisted pass to auto-generate potential issues or quick fix snippets (developer-run step to improve rules; not required at runtime).
- Step 5: Send each chunk + relevant static findings to ChatGPT API with a strict JSON schema prompt (see patterns below). Keep temperature = 0 and max tokens tuned.
- Step 6: Parse ChatGPT JSON output (summary, severity, suggested fix, single-line comment text). Post comments on PR via GitHub REST/GraphQL. Mark results as “automated” and include a link to the payload stored in artifact for audit.

Prompt patterns (concrete, short)
- System: “You are a concise PR reviewer. Respond in EXACT JSON matching this schema: {issues: [{file,hunk_start,hunk_end,severity,label,explanation,one_line_comment,patch_suggestion}], summary: string}. Do not invent tests, do not assume runtime. Be explicit about uncertainty.”
- User (per chunk): include: repo, file path, language, diff hunk, relevant test failures, static findings. Then a one-line instruction: “Produce up to 3 actionable issues for this hunk; each must include a suggested code patch (unified diff) where possible.”
- Few-shot: include 2 short examples: one true-positive bug + fix, one style suggestion; show exact JSON outputs.
- Safety: force “I’m uncertain” when confidence < 70% and do not auto-apply non-trivial patches.

Decision criteria (when to run what)
- Run full LLM on PRs 500 LOC use static checks + defer LLM to a human reviewer step (cost/latency). Balanced by budget, team size and tolerated latency.
- If code is sensitive or policy forbids sending code externally, disable LLM and stick to in-house analyzers.

Practical checklist (implementation)
- [ ] Create GitHub Action: checkout, diff generator, linters, CodeQL.
- [ ] Implement chunking + metadata builder (language, function name, tests referencing file).
- [ ] Create system prompt and two few-shot examples; lock schema.
- [ ] Integrate ChatGPT API: temperature=0, max_tokens per chunk, rate-limit backoff.
- [ ] Idempotent comment posting: tag comments with run-id; update instead of duplicate.
- [ ] Audit trail: save payloads/artifacts for compliance.
- [ ] Human-in-the-loop: add label “autofix-suggested” and require maintainer approval for applying patches.
- [ ] Security: strip secrets; do not send full repo credentials.

Best-for / Avoid-if
- Best for: teams that want fast, actionable commentary on small-to-medium PRs and accept sending code to an external LLM.
- Avoid if: strict IP/compliance requirements, very large diffs, or budget-conscious projects that can’t absorb frequent LLM calls.

Notes on cost/skill
Expect moderate engineering to build chunking, idempotent comment logic, and robust prompts. Choose thresholds based on budget, latency tolerance, and desired output quality.

If you want, I can provide a drop-in GitHub Action YAML + 2 example prompts and JSON schema to start—would you prefer that now?

Compare ChatGPT and Gemini