Home/rules

The Rules

By registering, every team member agrees to these rules.

01Teams

  • Teams of 1 to 4 members. Individual participation is welcome.
  • A person may belong to only one team. Duplicate registrations lead to disqualification.
  • Each team names one corresponding member who handles all communication and holds the team's single Codabench account.
  • People who have been given access to the full dataset, are not allowed to participate or advise teams.

02Model size: the 8B limit

  • The total parameters of every component used at inference must not exceed 8 billion.
  • This includes all models, ensemble members, retrievers, rerankers and embedding models.
  • Mixture-of-Experts models count total parameters, not active parameters.
  • Quantisation does not reduce the parameter count.

03Models

  • Only open-weight models released before 1 October 2026.
  • No closed APIs (e.g. GPT, Claude, Gemini) at inference time.

04Training data

  • Any publicly available data may be used for training.
  • Synthetic data generated with open models is permitted.
  • Closed APIs may be used only to generate training data, and this must be disclosed.
  • Training, tuning or prompt design on test questions in any form is strictly prohibited.

05Inference

  • No internet access or live retrieval at inference.
  • The full pipeline must run from submitted code with no human in the loop.

06Reproducibility

  • All teams in the final leaderboard share their code, scripts and a model manifest after the final leaderboard submission deadline.
  • Organisers re-run every final system. Only reproduced runs are added to the final ranking and eligible for prizes.
  • All teams must submit a system description report for final ranking.
!

Integrity. The organisers may disqualify any team, or revoke a prize after the event, for rule violations, including manual predictions, use of test answers from any source, multiple Codabench accounts, or undisclosed closed-API use. Committee decisions are final.