gBench
by Google DeepMind (Luciano Martins, Frank Li)
What is gBench? gBench is a rubric libraries & frameworks eval made by Google DeepMind (Luciano Martins, Frank Li). An open-source suite for benchmarking open LLMs and vision-language models on your own hardware, against any OpenAI-compatible endpoint or a local vLLM server it starts itself. It measures serving speed and load limits with repeatability checks, and capability in three tiers: 86 standard academic eval suites across knowledge, math, coding, long context, tool use and vision; 16 deterministic smoke tests, including pairs that tell a broken chat template apart from a weak model; and an agentic quality suite for multi-turn tool calling and session memory. Useful when you self-host a model and want speed and quality numbers from one tool.
Editor note
An open-source suite for benchmarking open LLMs and vision-language models on your own hardware, against any OpenAI-compatible endpoint or a local vLLM server it starts itself. It measures serving speed and load limits with repeatability checks, and capability in three tiers: 86 standard academic eval suites across knowledge, math, coding, long context, tool use and vision; 16 deterministic smoke tests, including pairs that tell a broken chat template apart from a weak model; and an agentic quality suite for multi-turn tool calling and session memory. Useful when you self-host a model and want speed and quality numbers from one tool.
- What it tests
- Serving performance and capability of self-hosted open models, from academic evals to quick smoke tests and agentic tool-use checks.
- Original source
- github.com/google-deepmind/gbench
- Last updated
- Repo created 2026-10-30; last push 2026-10-31
- License
- Apache-2.0
Related rubric libraries & frameworks evals
- Inspect Evals — Many benchmarks under one runner: coding, agents, safety, knowledge.
- OpenAI Evals — Building and running your own evals; community registry.
- LLMJury rubric library — LLM-as-judge rubrics for support, RAG, summaries, code and tone.
- Prompt Assay judge templates — Starter LLM-as-judge prompts.
Discussion
No one has discussed gBench yet. Have you run it? Share what it caught, what it missed or how you set it up.