MultiLoKo
by Dieuwke Hupkes and Nikolay Bogoychev (Meta, facebookresearch)
What is MultiLoKo? MultiLoKo is a multilingual & indian languages eval made by Dieuwke Hupkes and Nikolay Bogoychev (Meta, facebookresearch). A knowledge benchmark for 31 languages (30 plus English) in which the questions for each language were written separately by annotators asked to target locally relevant topics, instead of translating one English set. Each non-English subset also comes with human and machine translations into English, and the English set with translations the other way, so you can separate what a model knows from how well it handles the language. Scoring uses exact match, and the leaderboard also reports the gap between the best and worst language and a mother-tongue effect (local-language score minus the score on the same questions in English). The public development set is in the repository; the repository says a private test set behind its leaderboard will be released later.
Editor note
A knowledge benchmark for 31 languages (30 plus English) in which the questions for each language were written separately by annotators asked to target locally relevant topics, instead of translating one English set. Each non-English subset also comes with human and machine translations into English, and the English set with translations the other way, so you can separate what a model knows from how well it handles the language. Scoring uses exact match, and the leaderboard also reports the gap between the best and worst language and a mother-tongue effect (local-language score minus the score on the same questions in English). The public development set is in the repository; the repository says a private test set behind its leaderboard will be released later.
- What it tests
- Locally sourced factual knowledge across 31 languages, cross-language parity and the effect of asking in the local language versus English.
- Original source
- github.com/facebookresearch/multiloko
- Last updated
- arXiv:2504.10356; repo last push 2026-11-16 (GitHub API, read 20 Nov 2026); HF dataset facebook/multiloko last modified 2026-10-27
- License
- MIT (GitHub API and HF dataset tag, read 20 Nov 2026)
Related multilingual & indian languages evals
- IndicGenBench — Text generation quality in Indic languages.
- MILU — Multi-subject understanding in Indian languages and culture.
- Global-MMLU — Knowledge across languages and cultural sensitivity.
- Samiksha — Culturally grounded health chatbot answers in Indian languages.
Discussion
No one has discussed MultiLoKo yet. Have you run it? Share what it caught, what it missed or how you set it up.