DROP
by Dua et al. (AllenNLP & collaborators)
What is DROP? DROP (Discrete Reasoning Over Paragraphs) is an English reading-comprehension benchmark of about 96K crowdsourced, adversarially written questions. Answering requires resolving references across a paragraph and doing discrete operations such as addition, counting or sorting.
Editor note
96K crowdsourced, adversarially written reading questions where the model must locate several facts in a paragraph and then add, count or sort them. At release the best system reached 32.7% F1 against 96.0% for expert humans.
- What it tests
- Reading comprehension that needs arithmetic and counting over the text.
- Original source
- arxiv.org/abs/1903.00161
- Last updated
- Not checked yet
- License
- Not checked yet — see source
When to use DROP
- You want to test reading plus simple numerical reasoning over text.
- You compare against older model reports: OpenAI's simple-evals reports DROP as 3-shot F1.
Limitations
- It is old and public, so contamination is likely.
- Licence not verified: the Hugging Face card lists CC BY-SA 4.0, while OpenAI's simple-evals README links an Apache-2.0 licence. Check before redistributing.
How to run it
- Load the dataset from Hugging Face (ucinlp/drop) or the AllenNLP page.
- Score with F1 and exact match on the answers; simple-evals has a ready DROP eval.
Notable results
- Paper (2019): the best existing systems reached 32.7% F1 versus 96.0% for expert humans; the authors' numerical-reasoning model reached 47.0% F1. [Paper (arXiv)]
Sources
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related reasoning & knowledge evals
- GPQA — Expert-level science reasoning.
- Humanity's Last Exam — Frontier academic knowledge and reasoning.
- LiveBench — Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge).
- MMLU — World knowledge and problem solving across many school and professional subjects.