Home/submission

How to Submit

Predictions go to Codabench. Code of your final submission goes to the organisers for reproduction.

Open Codabench
  1. Step 1

    Register

    Fill in the form with all members and the corresponding member's Codabench username.

  2. Step 2

    Predict

    Upload a zipped predictions file on Codabench to participate in the leaderboard. Up to 3 submissions per day in the test phase.

  3. Step 3

    Leaderboard

    Make sure your final system is on the leaderboard by the deadline. Otherwise your last successful submission counts.

  4. Step 4

    Reproduce

    Submit a code package that reproduces each selected run end-to-end, plus your system description report. Reports must cite the SinhalaMMLU dataset paper.

Predictions file

One JSON object per line, zipped as predictions.zip containing predictions.jsonl.

Each prediction should match the following format:

{"q_no":1,"subject":"Arts","category":"humanities","level":"easy","answer":"2"}
{"q_no":2,"subject":"Arts","category":"humanities","level":"easy","answer":"4"}