ViBench 2.0
by Replit (with Fireworks running the official results)
What is ViBench 2.0? ViBench 2.0 is a coding eval made by Replit (with Fireworks running the official results). A long-horizon software benchmark where the agent starts from an empty project, builds a first version of a product from a short spec, then adds up to 18 features on top of its own code. Version 2.0 (28 Oct 2026) has 17 products modelled on familiar apps, 241 feature requests and 544 test plans; a grading agent runs every plan in a real browser. Eight products are public in the Harbor format and nine are held out. Replit reports the top model passing every plan in 65% of builds.
Editor note
A long-horizon software benchmark where the agent starts from an empty project, builds a first version of a product from a short spec, then adds up to 18 features on top of its own code. Version 2.0 (28 Oct 2026) has 17 products modelled on familiar apps, 241 feature requests and 544 test plans; a grading agent runs every plan in a real browser. Eight products are public in the Harbor format and nine are held out. Replit reports the top model passing every plan in 65% of builds.
- What it tests
- Whether coding agents can grow a whole product feature by feature without breaking what they built earlier.
- Original source
- replit.com/blog/vibench-2-long-horizon-software-benchmark
- Last updated
- Version 2.0 released 2026-10-28; public repo last push 2026-10-28
- License
- Apache-2.0 (public repo ViBench/vibench-public)
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.
Discussion
No one has discussed ViBench 2.0 yet. Have you run it? Share what it caught, what it missed or how you set it up.