Moving an AI pipeline from GPUs to managed APIs
Why I moved a call-analysis pipeline off a rented GPU and onto Gemini and Groq, and the queue work that made a network-bound pipeline reliable.
2 min read
On this page
The call-analysis platform I lead listens to every recorded sales and support call, transcribes it, works out who is speaking, and scores the conversation against the company's own procedures. The first version ran all of that on a rented GPU. Today it runs on a CPU-only server, and the models are someone else's problem.
This is how that move went, and why the hard part wasn't the models at all.
The first version
The original pipeline was entirely self-hosted:
- Faster-Whisper for transcription,
- pyannote for speaker diarization (agent versus customer),
- Llama 3 through Ollama for the quality evaluation,
all on one GPU machine, with watchdog services to restart anything that fell over.
It worked, and it was a good way to learn how each stage behaves. But a GPU host is a pet. It needs drivers, model downloads, memory tuning and someone watching it, and the queue was only ever as healthy as that one machine.
What changed
Transcription, speaker diarization and emotion analysis moved to Gemini 2.5 Flash. The evaluation moved to an LLM hosted on Groq. Policy context still comes from our own RAG index in Pinecone, so the scores follow the company's real procedures rather than the model's idea of good service.
Before
After
The new failure mode: the network
Self-hosted models fail loudly: the process dies, the watchdog restarts it. Managed APIs fail quietly. A request hangs, a connection drops halfway, a rate limit kicks in for a few seconds. Moving to APIs swapped hardware problems for network problems, so the queue had to get much more defensive.
Every stage gets a time limit
A stage that never returns is worse than one that fails, because it holds a queue slot forever. Every stage now runs under its own timeout:
async def run_stage(name: str, fn, *args, timeout: float):
try:
return await asyncio.wait_for(fn(*args), timeout=timeout)
except asyncio.TimeoutError:
raise StageTimeout(name)Slots are always released
The queue runs four automatic jobs in parallel. The bug that taught me the most: a dropped connection to the recording storage left a job hanging, its slot was never released, and after a few of those the queue quietly stopped. The fix is boring and essential:
async def run_job(job):
slot = await slots.acquire()
try:
await process(job)
except Exception as error:
await record_failure(job, error)
await schedule_retry(job)
raise
finally:
slots.release(slot)Retries, with limits
Failed jobs are retried, up to ten times, and a guard stops the same recording from being queued twice. Anything that still fails lands in a failure log that an admin can review, and there is an emergency stop for the whole queue.
Recovery after a restart
When the service starts, it looks for jobs that were in progress when it stopped and puts them back in the queue. Deploys no longer lose work.
What I'd tell someone making the same move
- Keep your own context. The RAG index and the scoring criteria stayed ours, stored in the database and editable from the dashboard, so changing a model provider didn't change what a good call means.
- Run the old and new side by side on the same recordings before switching, and compare the scores, not just the transcripts.
- Make prompts data, not code. Category-specific prompts and leniency settings live in the database, so tuning them doesn't need a deploy.
The result is a smaller, calmer system: fewer moving parts to babysit, and a queue that assumes the network will fail and carries on when it does.