Testing AI Report Quality Before Advisers See It

The challenge

Changing a prompt or a template changes every report that follows it. Trying those changes on live client records would mix experiments with real work, and waiting for an adviser to report a problem is the wrong way to find one. We needed somewhere to run a template against real material and read the output before anyone relied on it.

What we built

An admin-only test workspace inside FinPlan:

• Paste a transcript: run any template against a pasted transcript and read the draft straight away.
• Upload a recording: MP3, WAV or M4A files up to 100 MB go through transcription, clean-up and drafting. Long recordings run as background jobs, and the page checks on the job every three seconds.
• Test work kept apart: test records and reports are flagged as tests and listed under their own tab, away from real client records.
• Failures you can see: if transcript clean-up fails during a test, the report is drafted from the raw transcript and labelled that way, so the failure shows up in the output.
• Text-loss diagnostics: a diagnostic endpoint compares the raw and cleaned transcripts for any record and reports the percentage of text that went missing, which catches a clean-up model that has started to summarise.

Technology

Admin role checks on the GraphQL resolvers, REST controllers and front end. Uploaded test audio goes to Azure Blob Storage behind a 24-hour read-only link for Azure AI Speech. Long jobs run in the background and are tracked in MongoDB. This workspace is how we check changes while running An AI Platform That Turns Client Meetings Into Finished Reports in production.