Multilingual & Indian languages

JFinBench

by Sakana AI (Issa Sugiura, Taro Makino, Takashi Ishida)

What is JFinBench? JFinBench is a multilingual & indian languages eval made by Sakana AI (Issa Sugiura, Taro Makino, Takashi Ishida). A Japanese finance benchmark of 321 examples taken from the 2026 editions of six Japanese certification exams in accounting, tax, financial planning and banking, with multiple-choice, computed answers, answer forms, essays and a few image questions. A human-in-the-loop agentic pipeline turns the exam PDFs into the dataset, so it can be refreshed with each new exam edition to limit contamination. It can be run with no tools or with a shell and web access through a Harbor adapter. The paper (arXiv:2611.18130) reports Claude Fable 5 and GPT-6 Astra above 93% across 12 models, with the best open-weight model about 12 points behind. Part of the data is gated on Hugging Face and part must be rebuilt from the publishers' PDFs.

Page last updated 2026-10-18 · Evals Review

Editor note

A Japanese finance benchmark of 321 examples taken from the 2026 editions of six Japanese certification exams in accounting, tax, financial planning and banking, with multiple-choice, computed answers, answer forms, essays and a few image questions. A human-in-the-loop agentic pipeline turns the exam PDFs into the dataset, so it can be refreshed with each new exam edition to limit contamination. It can be run with no tools or with a shell and web access through a Harbor adapter. The paper (arXiv:2611.18130) reports Claude Fable 5 and GPT-6 Astra above 93% across 12 models, with the best open-weight model about 12 points behind. Part of the data is gated on Hugging Face and part must be rebuilt from the publishers' PDFs.

What it tests
Japan-specific financial knowledge and exam-style reasoning, with and without tool use, on a refreshable test set.
Original source
github.com/SakanaAI/JFinBench
Last updated
arXiv:2611.18130; repo last push 2026-11-16 (GitHub API, read 17 Nov 2026)
License
Code Apache-2.0 (GitHub API); dataset under the publishers' terms (HF tag: other)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related multilingual & indian languages evals

  • IndicGenBench — Text generation quality in Indic languages.
  • MILU — Multi-subject understanding in Indian languages and culture.
  • Global-MMLU — Knowledge across languages and cultural sensitivity.
  • Samiksha — Culturally grounded health chatbot answers in Indian languages.

All Multilingual & Indian languages evals

Discussion

No one has discussed JFinBench yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions