Human Feedback API

One API call. Real human ratings.

Plug human evaluation into your pipeline. Audio in, structured results out.

API-first · Built for ML pipelines

Input

• Audio files
• Questions
• Instructions
• Metadata
• Rater criteria
• Study configuration

Output

{
  "study_id": "std_8f3a...",
  "status": "complete",
  "responses": [{
    "sample_id1": {
      "naturalness": {
        "values": [2, 4, 4]
      },
      "voice_match": {
        "values": [4, 3, 4]
      }
    }
  }]
}

Study types

Three core study types

Designed around the questions voice and conversational AI teams actually ask.

Live conversations

Single-sample rating

Multi-sample comparison

Real human conversations with AI models. Participants hold live conversations with speech-to-speech systems; structured post-call feedback is collected automatically.

Example evaluation areas

• Conversational naturalness
• Responsiveness
• Emotional intelligence
• Voice quality
• Helpfulness
• Engagement

Model development workflow

• Test experimental checkpoints with real users
• Gather rapid human feedback during development
• Iterate faster on conversational speech systems
• Benchmark and compare to other voice AI models

Example use cases

• Transcript accuracy
• Voice likability
• Emotional expression
• Audio quality
• Naturalness
• Pronunciation
• Speaker trustworthiness
• Role fit

Also supports

• Display transcripts, tags, prompts, and speaker info alongside audio
• Free-response questions for human transcription, qualitative feedback, and error analysis

Evaluation formats

• Preference selection
• Likert ratings
• Ranking tasks
• Similarity judgments
• Reference sample comparison

Why our ratings are higher quality

High-quality participants

Pre-screened raters from multiple trusted pools. Active fraud and AI-response detection.

Speed

Data delivered in hours, not days, closing the human eval loop at the pace your models move.

Unique incentive framework

Performance-based bonuses reward thoughtful, conscientious work, tuned per study and task type.

Fully managed ops

We handle payments, comms, support, and logistics. You never touch a crowd platform.

Global participant network

Many countries, languages, and demographics. Coverage expanding daily.

Subjective metrics

A few good places to start

These are subjective qualities that automated metrics can't capture. They need real human ears and real human judgment.

1. Role fit
   Does this voice sound like your ideal commentator? lawyer?
   Get feedback on how well a voice fits its intended use case.

2. Listenability
   Could you listen to this voice for hours and not get tired of it?
   Ask participants how much they would enjoy listening to the voice at length.

Built by experts in emotion science & voice AI

The same stack behind the world's most human-aligned TTS and speech-to-speech models.

Proven in production

Powering evaluation for Hume's own frontier voice models, used daily across training, alignment, and release.

Frontier voice expertise

Built by the team behind Hume's emotionally intelligent voice models, deep expertise in speech quality, prosody, and emotional perception.

Customize or templates

Bring your own questions, or start from the expert-designed templates we use to evaluate our own state-of-the-art models.

Custom integrations, evaluations, and question types

Collaborate with us to start running human evaluations on your next models. Our research engineering team is ready to help.