Multilingual & Indian languages

MultiLoKo

by Dieuwke Hupkes and Nikolay Bogoychev (Meta, facebookresearch)

What is MultiLoKo? MultiLoKo is a multilingual & indian languages eval made by Dieuwke Hupkes and Nikolay Bogoychev (Meta, facebookresearch). A knowledge benchmark for 31 languages (30 plus English) in which the questions for each language were written separately by annotators asked to target locally relevant topics, instead of translating one English set. Each non-English subset also comes with human and machine translations into English, and the English set with translations the other way, so you can separate what a model knows from how well it handles the language. Scoring uses exact match, and the leaderboard also reports the gap between the best and worst language and a mother-tongue effect (local-language score minus the score on the same questions in English). The public development set is in the repository; the repository says a private test set behind its leaderboard will be released later.

Page last updated 2026-10-18 · Evals Review

Editor note

A knowledge benchmark for 31 languages (30 plus English) in which the questions for each language were written separately by annotators asked to target locally relevant topics, instead of translating one English set. Each non-English subset also comes with human and machine translations into English, and the English set with translations the other way, so you can separate what a model knows from how well it handles the language. Scoring uses exact match, and the leaderboard also reports the gap between the best and worst language and a mother-tongue effect (local-language score minus the score on the same questions in English). The public development set is in the repository; the repository says a private test set behind its leaderboard will be released later.

What it tests
Locally sourced factual knowledge across 31 languages, cross-language parity and the effect of asking in the local language versus English.
Original source
github.com/facebookresearch/multiloko
Last updated
arXiv:2504.10356; repo last push 2026-11-16 (GitHub API, read 20 Nov 2026); HF dataset facebook/multiloko last modified 2026-10-27
License
MIT (GitHub API and HF dataset tag, read 20 Nov 2026)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related multilingual & indian languages evals

  • IndicGenBench — Text generation quality in Indic languages.
  • MILU — Multi-subject understanding in Indian languages and culture.
  • Global-MMLU — Knowledge across languages and cultural sensitivity.
  • Samiksha — Culturally grounded health chatbot answers in Indian languages.

All Multilingual & Indian languages evals

Discussion

No one has discussed MultiLoKo yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions