Troubleshooting ElevenLabs API rate limits for batch jobs

Short answer
Limit concurrency, add exponential backoff with jitter, honor Retry-After, and batch or deduplicate inputs before calling ElevenLabs. For 100 simultaneous synth requests you should be throttling at the client/queue level — not firing 100 parallel HTTP calls at once.

Recommendation (one-liner)
Run a worker pool of a controlled size (start 6–16 workers), use an exponential-with-jitter retry for 429/5xx/timeouts, respect Retry-After, and implement dedupe + local cache for repeated text.

Why this works
ElevenLabs (and most TTS APIs) enforce per-account rate limits and shorter timeouts under load. Sending many parallel requests causes head-of-line queuing, more 429/503 responses, and cascading retries that make things worse. Controlled concurrency and intelligent backoff let you absorb spikes without overwhelming the remote service.

Decision criteria (choose based on needs)
- Throughput vs latency: if low-latency per-item matters, increase worker count and budget for higher SLA; if throughput matters, keep larger queue but lower concurrency.
- Budget: more parallelism may increase cost if you need premium tier or faster rate limits. Start small if budget-constrained.
- Team / complexity: use simple semaphore + retry pattern for small teams; adopt circuit-breaker and dynamic throttling if you need production-grade reliability.

Practical batching & heuristics
- Deduplicate: cache TTS outputs keyed by voice+text+settings. Avoid re-synthesizing identical inputs.
- Batch short items: if your UI allows, merge short texts into one call (e.g., combine multiple lines into 1 request with pauses) to reduce HTTP calls. (Only if audio concatenation quality is acceptable.)
- Chunk long items and stream: synthesize long files in parts if the API supports streaming or partial results.
- Queue priority: separate urgent vs background jobs; let background jobs run at lower concurrency.

Retry & backoff example (Node.js/psuedocode)
- Retry on 429, 5xx, network timeouts. Prefer Retry-After header when present.
- Use exponential backoff with full jitter: wait = random_between(0, base * 2^attempt).

Pseudocode:
const MAX_ATTEMPTS = 6;
async function callWithRetry(req){
for(let attempt=0; attempt ({error:e}));
if(res && res.status === 200) return res.data;
const status = res.status || res.error?.code;
if(res.headers && res.headers['retry-after']){
await sleep(parseInt(res.headers['retry-after'],10)*1000);
} else if(status === 429 || status >= 500 || isTimeout(res.error)){
const backoff = Math.random() * (100 * Math.pow(2, attempt));
await sleep(backoff);
continue;
} else throw res.error || new Error('non-retriable');
}
throw new Error('max attempts reached');
}

Worker pool + semaphore
- Use a semaphore to cap concurrency (e.g., 8 workers). When 429 rate increases, drop semaphore to a lower count or open a circuit for X seconds.

Circuit breaker heuristic
- If >N 429s within M seconds, reduce concurrency by half and pause new jobs for T seconds. Gradually restore if errors drop.

Checklist to implement now
- [ ] Add local cache for generated audio (hash text+voice+settings).
- [ ] Limit concurrency with semaphore/worker pool (start 6–16).
- [ ] Implement exponential backoff with full jitter and use Retry-After when provided.
- [ ] Deduplicate incoming jobs and coalesce short texts where possible.
- [ ] Add circuit-breaker that reduces concurrency on burst 429s.
- [ ] Monitor metrics: 429 rate, latency, success %, queue depth.

Best-for / Avoid-if
- Best for: production services generating many TTS files where reliability matters. Use caching + queueing.
- Avoid if: you need instant single-call low-latency voice for every user and can't tolerate queueing — then you’ll need a higher-rate plan or edge TTS solution.

If you want, I can provide a full Node.js worker implementation or an asyncio Python example wired to ElevenLabs' endpoints.