Ask HN: Has DuckDuckGo ever had it's own index?
They say they have their own crawler, but it must just be for instant answers because all of the “blue links” come from Bing. Did they ever have their own index which added to the results? What was DuckDuckGo like when it started?
Part of why I ask this is because Gabriel Weinberg (top dog at DDG) is quoted as saying, “[Y]ou can get to an 80/20 pretty quickly” in this court document: https://www.courtlistener.com/docket/18552824/1436/united-states-of-america-v-google-llc/
He is talking about the 80/20 problem, which is supposedly that it isn’t too hard to make a search engine that can answer 80% of queries, but it takes more effort to be able to answer the remaining 20%.
The problem is, if DuckDuckGo has never had it’s own index what would Gabriel know about the 80/20 problem? Seems like he would only know what he’s heard from other people.
It's not limited to the context of search engines, the 80/20 phenomenon is well documented in many places: https://en.wikipedia.org/wiki/Pareto_principle
Oh funny, thanks for that. I was experimenting with keyword recommendation the other day, and I told a friend that the keywords which appear in 19% of the results for a query seem like the most useful and relevant to the original query.
He said something like, “Oh, 80% comes from 20%.” I didn’t get it, but he must have been referencing this.