Agents & tool use

WeightBench

by Aayush Marasini, Barsat Khadka and Nick Rahimi

What is WeightBench? WeightBench is an agents & tool use eval made by Aayush Marasini, Barsat Khadka and Nick Rahimi. Tests whether an agent can work directly on another neural network's weights, which the authors call weight-space programming. Its 42 tasks in five tracks ask the agent to read what a network does, fix a broken network, write a working network from scratch, spot edits that are impossible, and compress a model without data. Target networks range from about 2K-parameter control policies to 7B-parameter language models, the agent runs in a sandbox with a metered budget of forward passes, and scores come from a hash-chained log of its model calls. The paper (arXiv:2611.18553) reports agents doing well at reading small networks and writing control policies but not yet at patching language models; the README labels its leaderboard v0.1 and preliminary.

Page last updated 2026-10-18 · Evals Review

Editor note

Tests whether an agent can work directly on another neural network's weights, which the authors call weight-space programming. Its 42 tasks in five tracks ask the agent to read what a network does, fix a broken network, write a working network from scratch, spot edits that are impossible, and compress a model without data. Target networks range from about 2K-parameter control policies to 7B-parameter language models, the agent runs in a sandbox with a metered budget of forward passes, and scores come from a hash-chained log of its model calls. The paper (arXiv:2611.18553) reports agents doing well at reading small networks and writing control policies but not yet at patching language models; the README labels its leaderboard v0.1 and preliminary.

What it tests
Reading, repairing, writing and compressing neural-network weights directly, without gradient descent.
Original source
github.com/Aayush-Marasini/WeightBench
Last updated
Leaderboard v0.1 (preliminary); arXiv:2611.18553 (13 Nov 2026); repo last push 2026-11-16 (GitHub API, read 19 Nov 2026)
License
MIT (GitHub API, read 19 Nov 2026)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related agents & tool use evals

  • GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
  • WebArena — Autonomous web navigation and task completion.
  • OSWorld — Computer-use agents operating real operating systems and apps.
  • Terminal-Bench — Agents working in the shell end to end.

All Agents & tool use evals · Comparison table

Discussion

No one has discussed WeightBench yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions