Development Phase
from 01 Oct
Build and validate systems on the dev set, and the given baseline.
The Task
Given a question written in Sinhala and its answer options, predict the correct option.
Overview
The Sinhala MMLU benchmark contains multiple-choice questions written natively in Sinhala across a range of subjects. Each question carries a difficulty label - Easy, Medium or Hard and systems must choose the single correct option.
Use any approach that respects the rules: prompting, fine-tuning, ensembles - as long as the entire inference pipeline totals 8B parameters or less and runs from submitted code with no human in the loop.
Data
- Development set - released with gold answers, for building and validating systems.
- Hidden test set - released without gold answers to registered teams during the test phase.
The test set is licensed for use in this shared task only. Redistributing test questions, or publishing them online, is strictly prohibited.
phases
Test Phase
12 – 22 Oct
Submit predictions for the hidden test set, and participate in the live Codabench leaderboard.
Verification Phase
23 Oct – 02 Nov
Organisers reproduce final runs from submitted code. Final rankings are subject to change based on the reproducibility.