IdeaTaste
by Shanshan Zhong, Chen Xu, Jingjie Ning, Zhengzhong Liu, Aviral Kumar, Chenyan Xiong
What is IdeaTaste? IdeaTaste is a reasoning & knowledge eval made by Shanshan Zhong, Chen Xu, Jingjie Ning, Zhengzhong Liu, Aviral Kumar, Chenyan Xiong. A research-taste benchmark: LLM agents read a literature corpus frozen at the end of 2025 (about 263K papers across ML systems, code generation, world models and auto-research) and write research proposals, which are then matched against the roughly 11.6K papers those communities actually published in 2026. Scores cover citation matching, idea matching, citation diversity and idea diversity; the authors' run spans 6 models and over 90,000 proposals. Code, dataset (Hugging Face) and a leaderboard are public, but scoring calls paid LLM judges, so reruns cost API spend.
Editor note
A research-taste benchmark: LLM agents read a literature corpus frozen at the end of 2025 (about 263K papers across ML systems, code generation, world models and auto-research) and write research proposals, which are then matched against the roughly 11.6K papers those communities actually published in 2026. Scores cover citation matching, idea matching, citation diversity and idea diversity; the authors' run spans 6 models and over 90,000 proposals. Code, dataset (Hugging Face) and a leaderboard are public, but scoring calls paid LLM judges, so reruns cost API spend.
- What it tests
- Whether an LLM agent proposes the research directions and cites the prior work that a field later pursued.
- Original source
- github.com/cxcscmu/IdeaTaste
- Last updated
- Repo last push 2026-11-03
- License
- MIT
Related reasoning & knowledge evals
- GPQA — Expert-level science reasoning.
- Humanity's Last Exam — Frontier academic knowledge and reasoning.
- LiveBench — Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge) in v1; LiveBench 2.0 (14 Oct 2026) adds long-horizon real-work tasks.
- MMLU — World knowledge and problem solving across many school and professional subjects.
Discussion
No one has discussed IdeaTaste yet. Have you run it? Share what it caught, what it missed or how you set it up.