PARSA-Bench پارسا

Persian Audio Reasoning and Speech Assessment: the first benchmark for large audio-language models on Persian language and culture

16 tasks · 10 new 7,645 samples · 20.6 h Findings of EACL 2027
Paper
PARSA-Bench: A Comprehensive Persian Audio-Language Model Benchmark
Mohammad Javad Ranjbar Kalahroodi*, Mohammad Amini*, Parmis Bathayan, Heshaam Faili, Azadeh Shakery · *equal contribution
University of Tehran & Institute for Research in Fundamental Sciences (IPM), Tehran

Persian is hard to hear

Persian brings audio challenges that no existing benchmark captures: a classical poetry tradition built on metre, a traditional music system of its own, and speakers who switch between Persian and English mid-sentence. PARSA-Bench asks whether large audio-language models (LALMs) actually understand Persian when they listen, across speech understanding, paralinguistic analysis, and culturally grounded audio reasoning.

16
Tasks, 10 of them new
7,645
Evaluated samples
20.6 h
Of audio
8
Models evaluated

Sixteen tasks

Speech understanding 10 tasks

  • Speech recognition
    500 clips · WER
  • Speech translation Fa → En
    500 clips · COMET
  • Speech translation En → Fa
    499 clips · COMET
  • Intent detection
    500 clips · 56 intents
    NEW
  • Named-entity recognition
    497 clips · 42 entity types
    NEW
  • Formal vs. informal register
    496 clips · 2 classes
    NEW
  • Code-switching detection
    500 clips · 2 classes
    NEW
  • Multiple-choice QA
    500 clips · 4 options
    NEW
  • Reading comprehension Wikipedia
    500 clips · 4 options
    NEW
  • Reading comprehension stories
    500 clips · 4 options
    NEW

Paralinguistic analysis 3 tasks

  • Age recognition
    480 clips · 6 age groups
  • Gender recognition
    500 clips · 2 classes
  • Emotion recognition
    388 clips · 6 emotions

Persian poetry and music 3 tasks

  • Poetry metre vazn
    347 recitations · 10 metres
    NEW
  • Poetry style sabk
    442 recitations · 4 styles
    NEW
  • Iranian classical music
    496 clips · 4 options
    NEW

Models know Persian, but struggle to hear it

On most tasks, a model reading the text does better than the same model listening to the audio. The language knowledge is there; the listening is the bottleneck. The strongest open-weight model, Qwen3-Omni-30B, shows the pattern clearly:

Audio minus text, Qwen3-Omni-30B

Change in each task's main metric (F1, COMET or accuracy) when the model hears the audio instead of reading the text

Show as a table
TaskAudioTextAudio − text

Give the model the transcript alongside the audio and weak models climb to near their text-only level. Qwen3-Omni-30B's intent-detection F1 goes from 0.46 with audio alone to 0.72 with the transcript added.

The exception: poetry

On both poetry tasks, audio beats text. Persian script leaves short vowels unwritten, so a line on the page doesn't say which syllables are long and which are short. A recitation does, and that rhythm is exactly what the metre is made of.

An annotated example from PARSA-Bench

Poet صائب تبریزی (Saeb Tabrizi) · Form غزل (ghazal)

همت بلند دار که با همت بلند

Hemmat boland dār ke bā hemmat boland

هر جا روی به توسن گردون سواره‌ای

Har jā ravi be towsan-e gardun savāre-i

Vazn (metre) مضارع مثمن اخرب مکفوف محذوف

مفعول
فاعلات
مفاعیل
فاعلن

"Keep your aspirations high—for with high aspiration, wherever you go, you ride the steed of fortune."

short long

Metre detection is still hard. Chance is 0.12, and only the largest model, Qwen3-Omni-30B, gets above it (0.20), the first sign that the task becomes learnable at scale.

Models evaluated

Qwen3-Omni-30B is the strongest open-weight model (median ASR word error rate 0.31). Gemini-2.5-Flash leads on translation (En → Fa COMET 0.84) and intent detection (F1 0.79).

Where the audio comes from