Speaker-Labelled Transcripts From Client Meetings

The challenge

A transcript of an advice meeting is only useful if you can tell who said what. "I'm comfortable with more risk" means one thing coming from the client and something else from an adviser explaining an option. Standard transcription produced one long block of text, or "Speaker 1" and "Speaker 2" at best, which left the adviser working out the roles from memory. The audio needed careful handling too. Client meetings are full of personal and financial detail, and every extra copy of a recording is extra exposure.

What we built

A batch transcription pipeline for FinPlan with speaker separation and calendar-aware naming:

• Speaker separation: recordings go to Azure AI Speech batch transcription in the Australia East region with diarisation switched on, so every phrase comes back tagged to a speaker.
• Straight from storage: in production the recording is passed to the speech service as a time-limited link. Our servers never download the audio.
• Names from the invite: speakers are ranked by how long they talked and matched against the attendees on the calendar event. The meeting organiser, usually the adviser, is matched to the longest speaker, and invitees who accepted are ranked ahead of those who did not. Five matching strategies cover the different cases, from an exact head count to more voices than invitees, and each one records a confidence score.
• Names applied before clean-up: speaker labels are swapped for attendee names before the transcript reaches the clean-up model. The finished transcript reads as a conversation between the people on the invite, with any extra voices marked as unknown speakers.
• Ready for long meetings: the pipeline waits up to an hour for a transcription job to finish, and retries job creation with backoff when the speech service has a temporary fault.
• Timed stages: every step records when it started and how long it took, so a slow stage shows up in the data.

Technology

Azure AI Speech batch transcription with diarisation, called from a Kotlin and Spring Boot service using a reactive web client. Speaker matching uses attendee data from the Recall.ai calendar integration. A provider interface lets transcription run on OpenAI Whisper instead, without speaker separation, chosen by configuration as covered in Switching AI Providers Without Changing the Product. We operate the pipeline in production as part of An AI Platform That Turns Client Meetings Into Finished Reports.