Real-time agent assist
Four AI jobs
per spoken sentence.
Speech in, four things out — a streaming transcript, self-drafting call notes, procedure docs pulled by vector search, and a sentiment score that pages a supervisor in under a second. One WebSocket carries all of it; nothing polls.
Pipeline telemetry
Transcript
Speak — or play the sample call — and your words stream in here live.
Call notes
Notes draft themselves — reason, key details, actions, follow-ups — as the call unfolds.
Procedure docs · RAG
Relevant support procedures are retrieved and surface here as topics come up.
How this works
Your browser streams mic audio in 250 ms chunks over a single WebSocket to a Node worker, which relays it to Deepgram for streaming speech-to-text. Each finalized line fans out to three jobs in parallel: an LLM drafts running notes, a pgvector search retrieves the most relevant procedures, and a second LLM scores sentiment — the frustration alert time is measured from transcript to score and shown on screen.
mic ──ws──▶ worker ──▶ Deepgram (ASR)
├──▶ LLM ▶ notes
├──▶ pgvector ▶ docs
└──▶ LLM ▶ sentiment ▶ alert
all results ──ws──▶ browserBuilt with
- Next.js· web UI
- Node + ws· WebSocket worker
- Deepgram nova-2· streaming speech-to-text
- OpenAI gpt-4o-mini· notes + sentiment
- text-embedding-3-small· RAG embeddings
- Supabase + pgvector· docs store + vector search
- Railway· hosting
- Cloudflare· DNS + TLS
Built with Claude Code, Anthropic’s agentic coding CLI — pair-programmed end to end (scaffold, pipeline, deploy). A public re-build of a real-time agent-assist system I built at Capital One; code on GitHub, full write-up on my site.
What this shows
This isn’t a model I trained. Speech-to-text is Deepgram; notes, retrieval, and sentiment are OpenAI API calls — off-the-shelf services anyone can call.
What it demonstrates is composing those commodity models into a real-time product that fills an actual need: helping a support agent, live and mid-conversation, before a call goes sideways.
The engineering is in the orchestration: a single-WebSocket streaming design; cadence and debouncing so LLM calls stay cheap; retrieval thresholds so docs stay quiet until relevant; speaker diarization; sub-second transcript-to-alert latency; and a cached replay so the demo is deterministic and free to run.
Demo only — not a Capital One system
This is an independent portfolio project built by Harrison Jansma. It is not connected to Capital One and uses none of its internal systems, data, models, or code — it’s a public re-build of the shape of that work. The sample call simulates a support session for Shopfolio — a portfolio project I built for my wife to help manage her Pinterest content ↗. The caller and conversation are fictional.