Introducing Hume's Audio Decision API: A decision layer for speech and sound

Contact centers score every call against a rubric. Changing what gets scored in the call is a project: new rubric, new pipeline, new evaluations, days if not weeks before the first answer.

And the transcript this all runs on is only the words. Not the pause, not the sigh, not the two people talking over each other. It misses the reasons a call went wrong. That information is in the audio, sitting in call recordings or actively getting ignored on real time calls. Until now, there has been no way to extract this information in a form you can act on.

Today we're introducing Hume's Audio Decision API, the first audio-native model that answers questions about speech and sound. You choose the questions and the possible answers, anything from whether a voice is AI generated to identifying if the caller is frustrated or if there is a siren in the background noise.

This is a different architecture. Raw audio goes in, an answer with a confidence score comes out, structured and ready for your code or your agent to act on. Fast enough to ask dozens of questions every few seconds on a live call and fast enough to make an archive with 40,000 audio recordings searchable in minutes. There is no system today where you can make that query.

It is built on years of Hume's emotion science research and millions of human judgments about how people perceive speech and sound. What used to take a composition of human listeners and distinct model judgements, now takes a single API call.

Ask your audio a question

Send raw audio and ask a question. The API returns probabilities over the answers you defined.

You write the question as a sentence along with the possible answers. The Audio Decision API returns probabilities for the answers you provide, whether they're yes/no, multiple choice, rating scales or spans across a clip:

• Yes/No: "Does the customer sound frustrated?" returns yes or no
• Choice: "Which best describes the agent's tone?" returns the option from your list
• Scale: "How patient does the agent sound?" returns a score on your scale
• Span: "Where does someone laugh, sigh or gasp?" returns start and end times

You can ask multiple questions and update them in real time. Adding a question is a line of JSON. No labels, no retraining, no added complexity. The same recording can be asked about background sound, language, accent and emotional expression at once, and you can set confidence thresholds that link responses to business rules.

You can also add additional context to each request, which can further optimize performance.

Questions can cover:

• Speech & languages
• Speakers & voices
• Emotion & expression
• Acoustic environment
• Non-speech events
• Conversation dynamics
• Communication quality
• Intent & outcomes
• Social & behavioural signals
• Temporal & contextual signals

We’ve included a guide on how to ask questions along with example questions across dimensions and industries in our Docs.

To walk through an example: For a retail customer-service voice agent, you might check whether the speaker sounds uncertain or frustrated, identify how many speakers are on the call, whether there was excessive background noise and ask what user intent was: return an item, cancel a subscription, etc. In a business workflow you could pair those judgments with transcription and give both as inputs to the LLM so that it can take action on business rules such as whether to wait, clarify or escalate.

Teams can also run questions in bulk across audio libraries, track expression over successive segments and pinpoint the moments when particular sounds or vocal events occur. The speed of processing also unlocks an audio search engine capability: find every call where the customer sounded frustrated. There is no system today where you can query audio so quickly without added pre-processing, labeling, indexing or transcription.

Grounded in human judgment

Counting speakers or detecting a siren involves relatively objective facts. Whether a voice sounds frustrated, empathetic or natural is a question of perception, and listeners don't always agree.

For years, Hume has asked panels of listeners to rate emotional expression, naturalness, vocal delivery and conversational fit. Tens of millions of these judgments have informed Real-World VoiceEQ and helped train our Audio Decision Model to reflect how people perceive speech and sound.

In an initial evaluation, Hume's Audio Decision model achieved 70.7% agreement with a human panel's average judgment, compared with 56.5% for the individual-rater baseline. The result shows how human-level perceptual judgment can be brought to speech and sound at scale, without asking a panel to review every recording. And more importantly, it unlocks human-level judgment for live voice AI interactions.

When benchmarked against existing methods for audio processing and understanding, the Audio Decision Model outperforms on three key vectors: accuracy, speed and cost.

Get started

Explore the Audio Decision API and request API access by joining the waitlist. Try out the realtime and upload file experience in the playground, where you can add your own questions and get immediate responses to them, with no added configuration.

Check out our Documentation to learn more about supported questions, experimental features and more.

What you can build

Evaluate and improve voice models

A voice model can be intelligible and still sound awkward. Teams can ask whether generated speech sounds natural, conveys the intended emotion or fits the situation, then compare the answers across models and checkpoints.

These judgments can guide data selection and training feedback, including reinforcement learning. Teams should validate their evaluations against representative human ratings.

Build more perceptive voice agents

A caller may say they're fine while sounding frustrated, or pause because they're thinking rather than because they've finished speaking. Those cues rarely appear in a transcript.

Audio Decision API gives a cascaded voice agent access to those cues. It can weigh an audio judgment alongside the transcript and its business rules before responding. Live and ambient applications remain a longer-term opportunity as real-time capabilities develop.

Analyze and search audio at scale

A contact center could ask which calls include interruptions, whether agents sound reassuring and which conversations deserve review. Teams can apply those questions across an entire library, then change the criteria as new patterns emerge.

They can also track how expression changes during a call and jump to the moments when specific sounds or vocal events occur.

Built for quick decisions

Many audio tasks need a judgment, not a description. Generative models are useful for open-ended explanations, but generating text adds unnecessary work when all an application needs is a structured answer. Fixed classifiers tend to be limited to predefined tasks.

Audio Decision API returns probabilities for the choices developers define. It's designed for fast, cost-efficient judgments that can be repeated across many recordings.

Try out the Audio Decision API today and unlock every call.