PARSA-Bench پارسا
Persian Audio Reasoning and Speech Assessment: the first benchmark for large audio-language models on Persian language and culture
@inproceedings{ranjbar2027parsabench,
title={PARSA-Bench: A Comprehensive Persian Audio-Language Model Benchmark},
author={Ranjbar Kalahroodi, Mohammad Javad and Amini, Mohammad and Bathayan, Parmis and Faili, Heshaam and Shakery, Azadeh},
booktitle={Findings of the Association for Computational Linguistics: EACL 2027},
address={Athens, Greece},
publisher={Association for Computational Linguistics},
year={2027}
}
@article{ranjbar2026parsabench,
title={PARSA-Bench: A Comprehensive Persian Audio-Language Model Benchmark},
author={Ranjbar Kalahroodi, Mohammad Javad and Amini, Mohammad and Bathayan, Parmis and Faili, Heshaam and Shakery, Azadeh},
journal={arXiv preprint arXiv:2603.14456},
year={2026}
}
Persian is hard to hear
Persian brings audio challenges that no existing benchmark captures: a classical poetry tradition built on metre, a traditional music system of its own, and speakers who switch between Persian and English mid-sentence. PARSA-Bench asks whether large audio-language models (LALMs) actually understand Persian when they listen, across speech understanding, paralinguistic analysis, and culturally grounded audio reasoning.
Sixteen tasks
Speech understanding 10 tasks
- Speech recognition
- Speech translation Fa → En
- Speech translation En → Fa
- Intent detectionNEW
- Named-entity recognitionNEW
- Formal vs. informal registerNEW
- Code-switching detectionNEW
- Multiple-choice QANEW
- Reading comprehension WikipediaNEW
- Reading comprehension storiesNEW
Paralinguistic analysis 3 tasks
- Age recognition
- Gender recognition
- Emotion recognition
Persian poetry and music 3 tasks
- Poetry metre vaznNEW
- Poetry style sabkNEW
- Iranian classical musicNEW
Models know Persian, but struggle to hear it
On most tasks, a model reading the text does better than the same model listening to the audio. The language knowledge is there; the listening is the bottleneck. The strongest open-weight model, Qwen3-Omni-30B, shows the pattern clearly:
Audio minus text, Qwen3-Omni-30B
Change in each task's main metric (F1, COMET or accuracy) when the model hears the audio instead of reading the text
Show as a table
| Task | Audio | Text | Audio − text |
|---|
Give the model the transcript alongside the audio and weak models climb to near their text-only level. Qwen3-Omni-30B's intent-detection F1 goes from 0.46 with audio alone to 0.72 with the transcript added.
The exception: poetry
On both poetry tasks, audio beats text. Persian script leaves short vowels unwritten, so a line on the page doesn't say which syllables are long and which are short. A recitation does, and that rhythm is exactly what the metre is made of.
همت بلند دار که با همت بلند
Hemmat boland dār ke bā hemmat boland
هر جا روی به توسن گردون سوارهای
Har jā ravi be towsan-e gardun savāre-i
Vazn (metre) مضارع مثمن اخرب مکفوف محذوف
"Keep your aspirations high—for with high aspiration, wherever you go, you ride the steed of fortune."
short long
Metre detection is still hard. Chance is 0.12, and only the largest model, Qwen3-Omni-30B, gets above it (0.20), the first sign that the task becomes learnable at scale.
Models evaluated
- Qwen2.5-Omni-3B
- Qwen2.5-Omni-7B
- Qwen3-Omni-30B-A3B
- Gemma-3n-E2B
- Gemma-3n-E4B
- GPT-4o-mini-audio
- GPT-4o-audio
- Gemini-2.5-Flash
Qwen3-Omni-30B is the strongest open-weight model (median ASR word error rate 0.31). Gemini-2.5-Flash leads on translation (En → Fa COMET 0.84) and intent detection (F1 0.79).
Where the audio comes from
- Real speech from Mozilla Common Voice, for recognition, age, gender and code-switching.
- Emotional speech from the ShEMO corpus.
- Poetry recitations collected through Ganjoor, the online library of Persian poetry.
- Synthesized speech for text-based tasks such as intent, named entities, QA and reading comprehension, voiced with the ParsVoice zero-shot TTS using real speakers as reference voices.