Redefining Search Benchmarking: Why Static Datasets Are No Longer Enough
In the landscape of modern AI, evaluating search-capable agents presents a fundamental paradox: how do you measure performance when the agent has access to the very tools that can circumvent the evaluation? Traditional benchmarks often suffer from leakage, where static query sets and publicized “ground truth” answers are indexed by web-crawling agents, allowing them to cheat by bypassing the search process entirely. Keenable AI’s introduction of NEEDLE (News, Everyday, Expert, Deep-tail, and Legal Evaluation) offers a robust, open-source architectural response to this challenge.
The Architecture of a Living Benchmark
Unlike conventional evaluation suites that freeze in time, NEEDLE is designed to evolve. It functions not as a static repository of questions, but as a dynamic harness that regenerates its query stream hourly. By drawing from live RSS feeds and current market trends, NEEDLE forces agents to demonstrate real-time retrieval capabilities rather than relying on cached internal knowledge.
The technical architecture of NEEDLE relies on two primary command-line operations: generate and run. Built to integrate into CI pipelines, the tool is installed using uv sync and requires a single OpenRouter key for high-level qualitative assessment alongside individual API keys for each search engine under test.
Key Evaluation Verticals
NEEDLE segments its evaluation into five distinct intent-based verticals to stress-test specific agent capabilities:
- News: Queries derived from over 124 RSS feeds and Google Trends, updated hourly to prevent memorization.
- Finance: Fact-based retrieval from SEC XBRL (10-Q filings), Wikidata, and the Global Legal Entity Identifier Foundation (GLEIF).
- Scholar: A multi-modal query approach using title fragments, full-text details, natural language hints, and “tip-of-the-tongue” descriptions to measure academic retrieval precision.
- Deep-tail: Queries sampled directly from real-world agent trajectories (e.g., DeepResearchGym, OpenResearcher), focused on rare, high-entropy tokens.
- Legal: Recent federal court opinions from CourtListener and eCFR sections, ensuring the search engine can navigate dense, non-standardized sources.
The Oracle Ceiling: Measuring Relative Success
One of the most profound aspects of the NEEDLE framework is the ultimate score. For every query executed, the system pools the results from all 15 competing search engines into a single synthetic oracle. By ordering these results by relevance, NEEDLE establishes an empirical upper bound of what is discoverable within the current search landscape.
This allows developers to distinguish between two critical failures:
- Ranking Failures: The evidence was found by the field collectively, but the specific engine failed to prioritize it.
- Retrieval Failures: The evidence was not retrieved by any participant in the pool, indicating a systemic limitation of current indexing.
Infrastructure and Performance Insights
NEEDLE emphasizes consistency by applying a 2,000-character evidence cap to all participants and enforcing sequential execution to ensure latency metrics remain unpolluted by concurrent server load. The benchmark tracks both P50 and P95 latency, acknowledging that agents frequently trigger multiple search calls in a single task, making the tail-end latency distribution a primary factor in the end-user experience.
Recent data indicates significant performance clustering in the Finance vertical, where most engines achieve recall scores between 0.77 and 0.91. Conversely, in the Deep-tail category—a proxy for complex agentic workflows—even the most capable engine struggles to capture more than 0.557 of the collective ceiling, revealing a substantial opportunity for future innovation in search indexing and retrieval strategies.
NEEDLE is available as an MIT-licensed project, with artifacts published to Hugging Face, providing a necessary layer of transparency and reproducibility for the agentic AI community.
