Short recommendation
Use a two-stage pipeline: (A) deterministic clause extraction per contract that preserves full provenance; (B) embedding-based clustering + Claude-driven canonicalization and summarization. Keep every extracted clause tagged with a unique ID and minimal positional metadata (contract_id, page, byte/char offsets or PDF coordinates). This gives auditors a direct link back to source text while enabling scalable aggregation.

High-level workflow
1) Ingest & OCR: normalize PDFs → text + page/coordinate mapping. Store originals. 2) Chunk & extract clauses: split by natural markers (headings, numbering) or sliding windows (1,500–3,000 tokens, 200–400 token overlap) and run Claude to extract clause text and metadata into strict JSON. 3) Embed & index: create embeddings for each clause and store in a vector DB (or FAISS) with the JSON metadata. 4) Cluster & dedupe: cluster by cosine similarity (experiment with 0.78–0.88 thresholds) to group functionally equivalent clauses. 5) Canonicalize with Claude: for each cluster, call Claude (temperature 0) to generate a canonical summary, standardized clause label, and a provenance list (clause IDs). 6) Audit export: export CSV/Parquet with cluster_id, canonical_text, member_clause_ids, contract_links, and confidence scores. Keep originals for manual review.

Prompting & traceability (practical template)
- Use a system prompt: “You are an extraction engine. Output only valid JSON arrays. Do not hallucinate. Use provided metadata.”
- Extraction output fields: {clause_id, contract_id, clause_text, clause_type (if detected), start_page, start_offset, end_page, end_offset, raw_text_context}.
- Canonicalization output fields: {cluster_id, canonical_summary, canonical_label, supporting_clause_ids, representative_snippet}.
Set Claude temperature to 0 for deterministic outputs and require strict JSON parsing to avoid manual cleanup.

Decision criteria (when to change approach)
- Budget: embedding & vector DB costs scale with clause count; if low budget, cluster less aggressively and sample for manual review. - Skill/engineering: if you lack dev resources, use lightweight tooling (CSV + manual grouping) for small-scale projects; invest in a vector DB and automation for hundreds of contracts. - Audit strictness: if auditors require exact byte/proof-of-origin, include PDF coordinates and checksums. - Team size & throughput: larger legal teams need richer metadata and UI; small teams can work from exported spreadsheets.

Best-for / Avoid-if
- Best for: hundreds+ contracts, running repeatable reviews, requiring audit trails and aggregated clause analytics. - Avoid if: a handful of contracts where manual review is cheaper, or if you need legally binding automated changes without human sign-off.

Practical checklist
- [ ] Store original PDFs with checksums. - [ ] OCR and map page/coordinate → text. - [ ] Decide chunk size (start 2k tokens, 200 token overlap). - [ ] Create deterministic Claude extraction prompt that outputs JSON. - [ ] Save clause-level metadata and unique IDs. - [ ] Build embeddings and index in vector DB. - [ ] Cluster, review thresholds, and dedupe. - [ ] Use Claude to canonicalize clusters (temp=0). - [ ] Export audit CSV/Parquet with full provenance and representative snippets. - [ ] Implement spot-checks: manual review of random 1–5% of clusters.

If you want, I can draft the exact Claude extraction and canonicalization prompts and a JSON schema you can drop into your pipeline.

Compare Claude and ChatGPT