I wonder if search engines linked to human-use-case engines (e.g. google/bing) start at a disadvantage because they have been historically incentivized to break themselves to support their business models? It seems reasonable to suppose that "good at selling ads" ≠ "good at finding results".
do you somehow control how non-trivial the queries are? The LLM generates them, right?
what if every engine returns garbage, or on the other hand, handles them too well?
building a benchmark like this in a genuinely fair way seems extremely hard to me. I’m very curious about the details, of course within what you can share.
It is hard to be fair I agree. We tried to be open about what we do here: github.com/keenableai/needle
The queries from what I can tell are not trivial. The actual github repo of the benchmark has a judgement/query browser where you can inspect the different query streams: https://keenableai.github.io/needle/
One of the authors here. We have been seeing lots of benchmaxxing and leakage in standard web search benchmarks such as BrowseComp.
We developed this live benchmark with daily/hourly sampled fresh queries matching real agentic search traffic to estimate actual search performance of different AI search providers.
It seems that the agentic search benchmarks have fallen victim to reward hacking, just like the coding benchmarks. It's good to see mitigation efforts being made to address this problem.
What do you foresee in the future releases and improvements to this?
so is this an independent search engine benchmark or a blog post from a search engine provider showing their search engine at the top of the benchmark?
i'm a little bit confused as at first when I was reading it i thought it was a search engine benchmark, but it seems that keenable is at the top which i assume is related to the web domain owner? i've never heard of keeenable
The judgements are available on huggingface so training is possible. But given that we evaluate on new queries daily there would need to be some generalization happening for this to show up in the benchmark.
The blog post appears to get confused and devotes its entire second half to pitching Keenable itself. If the idea is to build credibility for the new benchmark, this maybe was not the best choice.
Search engines with their own indexes can retrieve documents no one else has. That means your agents will be able to answer questions previously unanswerable.
Besides the clear AI smell, this nonsensical claim also plainly contradicts the methodology's key evaluation claim that the quality of an engine's results should be measured against how much it overlaps with the reranked aggregate of the other engines. The benchmark thus seemingly values an engine's ability to "answer unanswerable questions" at zero.
Search engines should improve from being used in a way where the quality improvement is not an engineering push but a property of the architecture. That's a learning machine, and it's the only kind of system that can adapt to agents.
This automatically optimizing for clicks using ML is the main way google and other "human" focused search engines have been improving for 20 years. Not sure what citation is needed here.
> Besides the clear AI smell, this nonsensical claim also plainly contradicts the methodology's key evaluation claim that the quality of an engine's results should be measured against how much it overlaps with the reranked aggregate of the other engines. The benchmark thus seemingly values an engine's ability to "answer unanswerable questions" at zero.
The engine itself is part of the reranked aggregate so if it finds something useful and everybody else does not it gets full credit.
I wonder if search engines linked to human-use-case engines (e.g. google/bing) start at a disadvantage because they have been historically incentivized to break themselves to support their business models? It seems reasonable to suppose that "good at selling ads" ≠ "good at finding results".
do you somehow control how non-trivial the queries are? The LLM generates them, right?
what if every engine returns garbage, or on the other hand, handles them too well?
building a benchmark like this in a genuinely fair way seems extremely hard to me. I’m very curious about the details, of course within what you can share.
It is hard to be fair I agree. We tried to be open about what we do here: github.com/keenableai/needle
The queries from what I can tell are not trivial. The actual github repo of the benchmark has a judgement/query browser where you can inspect the different query streams: https://keenableai.github.io/needle/
One of the authors here. We have been seeing lots of benchmaxxing and leakage in standard web search benchmarks such as BrowseComp.
We developed this live benchmark with daily/hourly sampled fresh queries matching real agentic search traffic to estimate actual search performance of different AI search providers.
It seems that the agentic search benchmarks have fallen victim to reward hacking, just like the coding benchmarks. It's good to see mitigation efforts being made to address this problem.
What do you foresee in the future releases and improvements to this?
so is this an independent search engine benchmark or a blog post from a search engine provider showing their search engine at the top of the benchmark?
i'm a little bit confused as at first when I was reading it i thought it was a search engine benchmark, but it seems that keenable is at the top which i assume is related to the web domain owner? i've never heard of keeenable
Do you think it will be possible to train on this bench?
The judgements are available on huggingface so training is possible. But given that we evaluate on new queries daily there would need to be some generalization happening for this to show up in the benchmark.
The blog post appears to get confused and devotes its entire second half to pitching Keenable itself. If the idea is to build credibility for the new benchmark, this maybe was not the best choice.
Besides the clear AI smell, this nonsensical claim also plainly contradicts the methodology's key evaluation claim that the quality of an engine's results should be measured against how much it overlaps with the reranked aggregate of the other engines. The benchmark thus seemingly values an engine's ability to "answer unanswerable questions" at zero.
Yeah? Care to cite anything for that?
> Yeah? Care to cite anything for that?
This automatically optimizing for clicks using ML is the main way google and other "human" focused search engines have been improving for 20 years. Not sure what citation is needed here.
> Besides the clear AI smell, this nonsensical claim also plainly contradicts the methodology's key evaluation claim that the quality of an engine's results should be measured against how much it overlaps with the reranked aggregate of the other engines. The benchmark thus seemingly values an engine's ability to "answer unanswerable questions" at zero.
The engine itself is part of the reranked aggregate so if it finds something useful and everybody else does not it gets full credit.