τ-bench-LILT

Name: τ-bench-LILT
Creator: LILT (Kaden Uhlig, Kiarash Mokhtari, Anıl Arslan, Yunsu Kim, Joern Wuebker)
License: https://spdx.org/licenses/MIT.html

by LILT (Kaden Uhlig, Kiarash Mokhtari, Anıl Arslan, Yunsu Kim, Joern Wuebker)

What is τ-bench-LILT? τ-bench-LILT is a multilingual & indian languages eval made by LILT (Kaden Uhlig, Kiarash Mokhtari, Anıl Arslan, Yunsu Kim, Joern Wuebker). A multilingual version of τ³-bench for tool-using customer service agents, covering German, Korean, Turkish and Farsi in the airline, retail, telecom and banking domains. Native speakers localized the customer databases, user-simulator prompts and tasks rather than machine-translating them. Across eight agent models the paper reports the biggest drops against English in Korean and Farsi retail, up to 61 points in pass^4. Code is on GitHub, data on Hugging Face (lilt-ai/tau-bench-lilt), with a leaderboard on LILT's Aurora site.

Page last updated 2026-10-18 · Evals Review

Editor note

A multilingual version of τ³-bench for tool-using customer service agents, covering German, Korean, Turkish and Farsi in the airline, retail, telecom and banking domains. Native speakers localized the customer databases, user-simulator prompts and tasks rather than machine-translating them. Across eight agent models the paper reports the biggest drops against English in Korean and Farsi retail, up to 61 points in pass^4. Code is on GitHub, data on Hugging Face (lilt-ai/tau-bench-lilt), with a leaderboard on LILT's Aurora site.

What it tests: Whether a support agent built and tested in English still follows policy and uses tools correctly with customers in other languages and locales.
Original source: github.com/lilt/tau-bench-lilt
Last updated: Repo last push 2026-11-06; arXiv v1 2026-11-04
License: MIT

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Related multilingual & indian languages evals

IndicGenBench — Text generation quality in Indic languages.
MILU — Multi-subject understanding in Indian languages and culture.
Global-MMLU — Knowledge across languages and cultural sensitivity.
Samiksha — Culturally grounded health chatbot answers in Indian languages.

All Multilingual & Indian languages evals

Discussion

No one has discussed τ-bench-LILT yet. Have you run it? Share what it caught, what it missed or how you set it up.

Be specific and kind. No promotion or affiliate links. Community rules · All discussions