Audio | Hume API

Overview

This detailed guide explains how to be successful recording and playing back audio with EVI.

The best way to handle audio is to use the Hume AI React SDK @humeai/voice-react, which takes care of everything in this guide out of the box. If you are using the Hume AI TypeScript SDK directly, or connecting to EVI from a different programming language, follow the instructions in this guide to handle audio recording and playback correctly.

Things to keep in mind when working with audio in EVI:

Recording

Connect to EVI

Before recording, open a WebSocket connection to the /v0/evi/chat endpoint. For more context, see the TypeScript quickstart guide.

import { Hume, HumeClient } from 'hume';
const client = new HumeClient(...);
const socket = await client.empathicVoice.chat.connect(...);

Client-side, not server-side - Typically, you should open the WebSocket connection to EVI on the client-side: either from your web frontend in JavaScript that runs in the user’s browser, or from inside your mobile app, because the server-side does not have direct access to the user’s microphone. Connecting to EVI from your backend is possible, but in this case you will have to transmit audio from the user’s device to your backend, and then from your backend to EVI, which will add latency.

WebSocket before Microphone - Connect to the EVI WebSocket before you start recording from the microphone. Audio formats like wav and webm begin with a header that you must transmit in order for EVI to be able to interpret the audio correctly. If the WebSocket connection is not ready when you begin recording and attempt to send the first bytes, you may inadvertently cut off the header.

Determine the audio format

Different browsers support sending different audio formats, described by MIME types. Use the getBrowserSupportedMimeType function from the Hume TypeScript SDK to determine an appropriate MIME type.

import {
  getBrowserSupportedMimeType,
} from 'hume';
const mimeType: MimeType = (() => {
  const result = getBrowserSupportedMimeType();
  return result.success ? result.mimeType : MimeType.WEBM;
})();

Start the audio stream

Use the getAudioStream helper from the Hume TypeScript SDK. This enables echo cancellation, noise suppression, and auto gain and wraps the standard MediaDevices.getUserMedia web interface.

Use the ensureSingleValidAudioTrack helper to make sure that there is a usable audio track. This will throw an error if there isn’t a single audio track (for example, if the user doesn’t have a microphone).

import {
  getAudioStream,
  ensureSingleValidAudioTrack,
} from 'hume';
let audioStream = await getAudioStream();
ensureSingleValidAudioTrack(audioStream);

Record, base64 encode, and transmit

Use the MediaRecorder API to record audio from the microphone. Inside the .ondataavailable handler, encode the bytes of the audio into a base64 string and send it in an audio_input message to EVI.

import { convertBlobToBase64 } from 'hume';
let recorder = new MediaRecorder(audioStream, { mimeType });
recorder.ondataavailable = async ({ data }) => {
  if (data.size < 1) return;
  const encodedAudioData = await convertBlobToBase64(data);
  socket.sendAudioInput({ data: encodedAudioData });
};
// capture audio input at a rate of 100ms (recommended for web)
recorder.start(100);

Add support for muting

Most EVI integrations should allow the user to temporarily mute their microphone. The standard way to mute an audio stream is to send audio frames filled with empty data (versus not sending anything during mute). This helps distinguish between a muted-but-still-active audio stream and a stream that has become disconnected.

recorder.ondataavailable = async ({ data }) => {
  if (data.size < 1) return;
  if (isMuted) {
    const silence = new Blob([new Uint8Array(data.size)], { type: mimeType });
    const encodedAudioData = await convertBlobToBase64(silence);
    socket.sendAudioInput({ data: encodedAudioData });
    return;
  }
  const encodedAudioData = await convertBlobToBase64(data);
  socket.sendAudioInput({ data: encodedAudioData });
};

Playback

At a high level, to play audio from EVI:

Receive audio

After connecting to EVI, listen for audio_output messages. Audio output messages have a data field that contains a base64-encoded WAV file. In a browser environment, you can use the convertBase64ToBlob function from the Hume TypeScript SDK to convert the base64 string to a Blob object.

socket.on('message', (message) => {
  switch (message.type) {
    case 'audio_output':
      const blob = convertBase64ToBlob(message.data);
      ...
  }
})

Play the audio from a queue

EVI can generate audio segments faster than they are spoken. Instead of playing the audio segments directly, you should place them into a queue on receipt.

To play an audio segment, convert the Blob to an Object URL use the Audio constructor from object from the browser’s HTMLAudioElement API, and call .play. Use the .onended listener to know when the segment has completed and play the next segment in the queue.

const audioQueue: Blob[] = [];
let currentAudio: HTMLAudioElement | null = null;

function playAudio() {
  if (!audioQueue.length || isPlaying) return;
  isPlaying = true;
  const audioBlob = audioQueue.shift();
  if (!audioBlob) return;
  const audioUrl = URL.createObjectURL(audioBlob);
  currentAudio = new Audio(audioUrl);
  currentAudio.play();
  currentAudio.onended = () => {
    isPlaying = false;
    if (audioQueue.length) playAudio();
  };
}

Handle interruption

EVI produces a user_interruption event if it detects that the user intends to speak while it is generating audio. However, it is also possible that a user will speak after EVI has finished generating audio for its turn, but before the audio has finished playing inside the browser. In this case, EVI will not produce a user_interruption event but will produce a user_message event. In both cases, you should stop the currently playing audio and empty the queue.

switch (message.type) {
  case 'user_interruption':
    stopAudio();
    break;
  case 'user_message':
    stopAudio();
    // Any additional handling for user messages
    break;
  ...
}

function stopAudio(): void {
  currentAudio?.pause();
  currentAudio = null;
  isPlaying = false;
  audioQueue.length = 0;
}

Enable verbose transcriptions to prevent interruption delays

By default, user_message events are only sent after the user has spoken for enough time to generate an accurate transcript. This can result in a perceptible delay in the user’s ability to interrupt EVI during the period when EVI is done generating its audio for the turn but before the browser has finished playing it.

To address this, you should set the query parameter verbose_transcription=true when opening the WebSocket connection to the /v0/evi/chat endpoint. This will cause EVI to send “interim” user messages, with an incomplete transcript, as soon as it detects that the user is speaking.

You should use these interim messages to stop the currently playing audio and clear the queue. Modify any other logic that uses the transcript from user_message events to either ignore messages with interim: true, or take into account how several interim messages transcribing the same segment of speech may be sent before the final message.

socket = client.empathicVoice.chat.connect({
  ..., 
  verbose_transcription: true,
})
switch (message.type) {
  ...
  case 'user_message':
    stopAudio();
    if (message.interim) {
      // ignore interim messages for any handling
      // of transcriptions
      break;
    }
    // Any additional handling for user messages
    break;
  ...
}

Using the EVIWebAudioPlayer for advanced playback

For a more advanced audio playback experience, you can use the EVIWebAudioPlayer, a cross-browser audio player. This player is designed for sequential, glitch-free audio playback, offering efficient handling of audio queues, volume control, and real-time Fast Fourier Transform (FFT) for visualization. The player has a dedicated AudioWorklet Mode (default) and Regular Buffer Mode that uses AudioBufferSourceNode on the main thread.

Initialization

First, create an instance of EVIWebAudioPlayer and initialize it. The init() method must be called within a user gesture (e.g., a button click) to comply with browser autoplay policies. The disableAudioWorklet option can be used to explicitly disable AudioWorklet Mode and use Regular Buffer Mode (AudioBufferSourceNode). Regular Buffer Mode is recommended if you’re hearing distortion in the default mode.

import { EVIWebAudioPlayer } from 'hume';

const player = new EVIWebAudioPlayer({
  volume: 0.8, // Set initial volume
  // disableAudioWorklet: true // Optional: defaults to false
});

// Call init() within a user gesture
document.getElementById('startButton')?.addEventListener('click', async () => {
  try {
    await player.init();
    console.log('EVIWebAudioPlayer initialized.');
  } catch (error) {
    console.error('Failed to initialize EVIWebAudioPlayer:', error);
  }
});

Enqueuing Audio

Once initialized, you can enqueue audio_output messages received from EVI. The player will automatically manage the queue and ensure playback.

socket.on('message', async (message) => {
  switch (message.type) {
    case 'audio_output':
      await player.enqueue(message);
      break;
    // ... other message types
  }
});

Handle interruption

EVI produces a user_interruption event if it detects that the user intends to speak while it is generating audio. However, it is also possible that a user will speak after EVI has finished generating audio for its turn, but before the audio has finished playing. In this case, EVI will not produce a user_interruption event but will produce a user_message event.

Although EVI halts response generation when interrupted, the user won’t experience the interruption unless the assistant’s voice also stops. To make the interruption perceptible, the client must stop audio playback by calling player.stop(), which stops the currently playing audio and clears the queue.

import type { SubscribeEvent } from 'hume/api/resources/empathicVoice/resources/chat';

socket.on('message', async (message: SubscribeEvent) => {
  switch (message.type) {
    case 'audio_output':
      await player.enqueue(message);
      break;
    case 'user_interruption':
      player.stop();
      break;
    case 'user_message':
      player.stop();
      // Any additional handling for user messages
      break;
    // ... other message types
  }
});

Handling Events

The EVIWebAudioPlayer emits various events that you can subscribe to for real-time feedback and control.

player
  .on('play', (e) => console.log('Playback started for:', e.detail.id))
  .on('stop', (e) => console.log('Playback stopped for:', e.detail.id))
  .on('fft', (e) => {
    // Use e.detail.fft for real-time audio visualization
  })
  .on('error', (e) => console.error('Player error:', e.detail.message));

Controlling Playback

You can control the player’s volume, mute/unmute, and stop playback.

// Set volume
player.setVolume(0.5);

// Mute/unmute
player.mute();
player.unmute();

// Stop playback and clear the queue
player.stop();

Handling complex audio scenarios

Understanding digital audio

A common source of issues when building with EVI is malformed or unsupported audio. This section explains what audio formats EVI supports, gives some conceptual background on how to understand digital audio more generally, and gives some advice for how to troubleshoot audio-related issues.

Audio formats

Hume attempts to accommodate the widest range of audio formats supported by our tools and partners. However, we recommend converting to one of the industry’s most commonly supported audio formats for troubleshooting. Two excellent choices are:

Audio/WebM

WebM is a container format that contains compressed audio, supported by all modern browsers. A WebM audio stream begins with a header that identifies the stream as being WebM. If you are sending audio in the WebM format, take care not to cut off the header.

Linear PCM

PCM (pulse-code modulation) is a method of representing audio as a sequence of samples that capture the amplitude of the audio signal at regular intervals. You must communicate these details to EVI in order for EVI to interpret the audio correctly. PCM is a headerless format, so you must send a session_settings message specifying the sample rate, the number of channels, and the encoding.

{
  "type": "session_settings",
  "audio": {
    "format": "linear16",
    "sample_rate": 44100,
    "channels": 1
  }
}

Non-supported formats

Mulaw (or μ-law) is a non-linear 8-bit PCM encoding that is commonly used in telephony. This encoding is not presently supported by EVI. You will need to convert it to a supported format.

Clicking

“Clicking” occurs when there is a discontinuity in the audio waveform. This can happen when you stop audio playback abruptly. You can eliminate the clicks by implementing a brief fadeout effect when stopping playback.

Troubleshooting audio

Audio issues can surface in several different ways:

Observe outgoing Session Settings and Audio Input messages

If your application runs in a web browser, you can view all the transmitted WebSocket messages. Open the developer tools, navigate to the Network tab, and filter for WS (WebSockets).

Extract the audio data

Copy the value of the .data property in the first audio_input message your application sends and decode it to analyze it with tools like ffmpeg.