Home/task

The Task

Given a question written in Sinhala and its answer options, predict the correct option.

Overview

The Sinhala MMLU benchmark contains multiple-choice questions written natively in Sinhala across a range of subjects. Each question carries a difficulty label - Easy, Medium or Hard and systems must choose the single correct option.

Use any approach that respects the rules: prompting, fine-tuning, ensembles - as long as the entire inference pipeline totals 8B parameters or less and runs from submitted code with no human in the loop.

Data

  • Development set - released with gold answers, for building and validating systems.
  • Hidden test set - released without gold answers to registered teams during the test phase.

The test set is licensed for use in this shared task only. Redistributing test questions, or publishing them online, is strictly prohibited.

Dataset paper

The benchmark is described in:

Ashmari Pramodya, Nirasha Nelki, Heshan Shalinda, Chamila Liyanage, Yusuke Sakai, Randil Pushpananda, Ruvan Weerasinghe, Hidetaka Kamigaito and Taro Watanabe. SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 32943–32961, 2025.

ACL Anthology

Teams must cite this paper in their system description report and presentation slides.

@inproceedings{pramodya-etal-2025-sinhalammlu,
  title     = {SinhalaMMLU: A Comprehensive Benchmark
               for Evaluating Multitask Language
               Understanding in Sinhala},
  author    = {Pramodya, Ashmari and Nelki, Nirasha and
               Shalinda, Heshan and Liyanage, Chamila and
               Sakai, Yusuke and Pushpananda, Randil and
               Weerasinghe, Ruvan and Kamigaito, Hidetaka
               and Watanabe, Taro},
  booktitle = {Proceedings of the 2025 Conference on
               Empirical Methods in Natural Language
               Processing},
  year      = {2025},
  pages     = {32943--32961},
  url       = {https://aclanthology.org/2025.emnlp-main.1673/}
}
phases
i

Development Phase

from 05 Oct

Build and validate systems on the dev set, and the given baseline.

ii

Test Phase

12 – 27 Oct

Submit predictions for the hidden test set, and participate in the live Codabench leaderboard.

iii

Verification Phase

28 Oct – 31 Nov

Organisers reproduce final runs from submitted code. Final rankings are subject to change based on the reproducibility.