JFinBench
by Sakana AI (Issa Sugiura, Taro Makino, Takashi Ishida)
What is JFinBench? JFinBench is a multilingual & indian languages eval made by Sakana AI (Issa Sugiura, Taro Makino, Takashi Ishida). A Japanese finance benchmark of 321 examples taken from the 2026 editions of six Japanese certification exams in accounting, tax, financial planning and banking, with multiple-choice, computed answers, answer forms, essays and a few image questions. A human-in-the-loop agentic pipeline turns the exam PDFs into the dataset, so it can be refreshed with each new exam edition to limit contamination. It can be run with no tools or with a shell and web access through a Harbor adapter. The paper (arXiv:2611.18130) reports Claude Fable 5 and GPT-6 Astra above 93% across 12 models, with the best open-weight model about 12 points behind. Part of the data is gated on Hugging Face and part must be rebuilt from the publishers' PDFs.
Editor note
A Japanese finance benchmark of 321 examples taken from the 2026 editions of six Japanese certification exams in accounting, tax, financial planning and banking, with multiple-choice, computed answers, answer forms, essays and a few image questions. A human-in-the-loop agentic pipeline turns the exam PDFs into the dataset, so it can be refreshed with each new exam edition to limit contamination. It can be run with no tools or with a shell and web access through a Harbor adapter. The paper (arXiv:2611.18130) reports Claude Fable 5 and GPT-6 Astra above 93% across 12 models, with the best open-weight model about 12 points behind. Part of the data is gated on Hugging Face and part must be rebuilt from the publishers' PDFs.
- What it tests
- Japan-specific financial knowledge and exam-style reasoning, with and without tool use, on a refreshable test set.
- Original source
- github.com/SakanaAI/JFinBench
- Last updated
- arXiv:2611.18130; repo last push 2026-11-16 (GitHub API, read 17 Nov 2026)
- License
- Code Apache-2.0 (GitHub API); dataset under the publishers' terms (HF tag: other)
Related multilingual & indian languages evals
- IndicGenBench — Text generation quality in Indic languages.
- MILU — Multi-subject understanding in Indian languages and culture.
- Global-MMLU — Knowledge across languages and cultural sensitivity.
- Samiksha — Culturally grounded health chatbot answers in Indian languages.
Discussion
No one has discussed JFinBench yet. Have you run it? Share what it caught, what it missed or how you set it up.