VākQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering

Papers

arxiv:2609.19879

Published on Sep 17

· Submitted by

Akkiraju on Sep 18

· Speech Processing Lab, LTRC, IIITH

Upvote

29

Authors:

Bhavana Akkiraju ,

,

,

,

,

Abstract

Question answering has advanced rapidly with large language models, but predominantly for high-resource languages, in both text and spoken settings. Spoken question answering (SQA) benchmark for Telugu remains unexplored, and the reliability of automatic evaluation in this setting remains unquantified. We introduce VākQA, a Telugu SQA benchmark of 2,001 factoid question-answer pairs across six domains, with 2.53 hours of speech audio, bilingual transcriptions, and human-verified reference answers. We first validate evaluation methods against human judgements: Gemini-as-a-judge best approximates human ratings but is non-uniformly strict, while open-weight judges systematically penalize correct Telugu answers that differ in surface form from the reference. Using this validated setup, we benchmark proprietary and open-weight models across input modality, language, and domain. We observe that Telugu phrasing retains cultural specificity that is lost in translation, speech input introduces phonetic confusions that alter question meaning, and cascaded ASR-MT errors compound progressively. VākQA is publicly released.

View arXiv page View PDF Add to collection

Community

Bhavanaakkiraju

Paper author Paper submitter 5 days ago

We introduce VākQA, a Telugu spoken QA benchmark, baseline proprietary and open-weight models under varying input conditions, and show that among tested metrics, Gemini-as-a-judge correlates best with human ratings, though not perfectly

librarian-bot

4 days ago

This comment has been hidden

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

Upvote

29

Get this paper in your agent:

hf papers read 2609.19879

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2609.19879 in a model README.md to link it from this page.

Datasets citing this paper 1

Spaces citing this paper 1

Collections including this paper 1