arxiv:2609.11115
Published on Sep 10
· Submitted by
Koutian Wu on Sep 14
Upvote
172
Authors:
,
,
,
,
Abstract
Benchmark Radar is a searchable living database and discovery engine for AI evaluation benchmarks that aggregates sources, score histories, and evidence to support benchmark selection and comparison.
Generated by thinkingmachines/Inkling-Small
Benchmark researchers and developers of large language models (LLMs) and other AI systems need to find relevant evaluations, locate their benchmark datasets and code, and understand the settings behind reported scores. We present Benchmark Radar, a living database and search engine for retrieval and discovery of AI benchmarks, covering LLM evaluation, agentic and tool-use benchmarks, coding, reasoning, safety, and domain-specific evaluations. The system combines daily discovery of benchmark papers, repositories, datasets, and releases with a searchable benchmark catalog, mentions in model cards and technical reports, and score histories. It retains source identities and citations so readers can inspect candidate benchmarks and their evaluation evidence. Daily discovery draws on 37 sources: 13 direct connectors and 24 first-party research and engineering feeds. The catalog contains 1,283 source records drawn from 4 benchmark catalogs and 12,916 numeric observations on 790 records. We describe collection and retrieval, audit the full catalog, and examine benchmark saturation, adoption trends, and the limits of score comparisons. A worked example walks through a complete prior-art search, showing how to query the catalog and inspect benchmark evidence when designing a new evaluation. We release the web dashboard with a benchmark leaderboard, a Pareto frontier view of score against measured use, saturation and trend views, daily feeds, downloadable evidence, a command-line interface (CLI) for offline queries, and reproducible analysis.
View arXiv page View PDF Project pageGitHub 244 Add to collection
Community
Paper author Paper submitter 9 days ago
•
edited 9 days ago
Benchmark radar is all you need when doing benchmark research! We kept running into new benchmarks while doing benchmark research, so we built a crawler that continuously collects benchmark-related signals from across the web. It pulls evidence from 37 public sources every day, and keeps updating. We also have a CLI tool, which can help you create LaTeX version related benchmark work in minutes from 12k+ benchmark/eval/dataset records.
·
8 days ago
Thanks for the excellent work!
9 days ago
Interesting and timely work
·
9 days ago
Thank you Charles!!!
8 days ago
This is an automated message from the ResearchStudio team.
We created an interactive ResearchStudio Reel for this paper. It includes a visual poster, a video, and a blog, all available for download in editable formats.
Open the ResearchStudio Reel →
Download all files from HF Mirror
Please give this comment a thumbs up if you find the Reel helpful!
Want to explore or create Reels for more papers? Visit the ResearchStudio demo.
8 days ago
Wow!! This Will definitely help my benchmark research
·
Paper author 8 days ago
Thanks Sada btw your name lol
8 days ago
Excellent work! I am using it as daily podcast of benchmarks
8 days ago
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
-
Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI (2026)
-
Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents (2026)
-
Cost-Effective Repository Exploration for Agentic Issue Localization (2026)
-
Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on HF Mirror checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
· Sign up or log in to comment
Upvote
172
Get this paper in your agent:
hf papers read 2609.11115
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2609.11115 in a model README.md to link it from this page.
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2609.11115 in a Space README.md to link it from this page.