The organic search results have also been tweaked over the years to put a greater emphasis on domain "authority." Google SERPs are a very different animal than they were 5 or 10 years ago -- back then there was a greater emphasis on keyword relevancy. Nowadays, you get stuff that doesn't match your search very well, but it comes from a domain that is popular, old, has lots of links, and maybe has a little bit of secret Google blessing because it's a big brand...
People commonly lament that the Web has changed, it's dominated by big brands, it's not weird anymore, etc. I think this is a big part. Google has made these changes in an effort to address some legitimate problems around spam, black/gray hat SEO etc. But they likely have some less user-centric incentives as well ($$$) and either way, Google SERPs have become more boring and less relevant.
There might be an opportunity for disruption emerging because of this. Google knows its search engine isn't very good anymore! That is why they're increasingly trying to pitch their service as a recommendation engine with special AI sauce, not an index of the Web. But frankly as this article demonstrates, what Google wants to recommend comes from a smaller sandbox and often is not the most interesting stuff out there.
If someone can come up with a distribution mechanism which solves the problem of helping the user discover All The Content minus the spammers, scammers and fake news, they will be able to deliver more relevant information than search or social media can today. Google being large actually gives them an in-built disadvantage here because no matter what algorithm they develop, it's going to be everyone's first priority to reverse engineer and game.
> If someone can come up with a distribution mechanism which solves the problem of helping the user discover All The Content minus the spammers, scammers and fake news, they will be able to deliver more relevant information than search or social media can today. Google being large actually gives them an in-built disadvantage here because no matter what algorithm they develop, it's going to be everyone's first priority to reverse engineer and game.
I wish there was a way to detect people attempting to game the algorithm, and just ban them indiscriminately. That's the only way. It probably requires AGI, though.
Just like the old saw, "you're not stuck in traffic, you're the traffic", if you're doing SEO beyond making your page lightweight and useful to people, you are a part of the problem.
It won't solve fake news issue; people could still make nice and fast sites full of lies. But at least low-effort "content marketing" spam would hopefully disappear, not to mention comment spam and all the other content that exists solely to push useless sites higher in SERPs.
> if you're doing SEO beyond making your page lightweight and useful to people, you are a part of the problem.
This is, unfortunately, an overly simplistic model. Everything can be manipulative if done to an extreme extent.
There's no individual thing that you can easily draw the line at. The question is where do you draw the line, and what combination of factors is ban worthy.
Hi there! Search Quality engineer here (though I don’t actually work on the core ranking myself, know very little about the specifics of its evolution over the years, and am nowhere near the news team). Every change we launch to search (and there are a lot) has very stringent requirements to meet. We have to, first and foremost, improve things for our users. That means a radical focus on improving the relevance, quality, and accuracy with which Google responds to your query. We measure this in a lot of ways. We relentlessly evaluate every change based on live traffic experiments (in which our metrics continuously evolve alongside our understanding of how to measure a good experience for our users) and side-by-side comparisons of queries with and without the change by real humans (trained to evaluate things with an eye towards the ideal results page). In every way we can measure, our search engine is the best it’s ever been. We definitely aren’t perfect; edge cases at scale are a never-ceasing job, and core improvements to the experience of searching on google always show that there’s plenty of headroom to improve what we return to users. But I heartily push back against both of these statements:
> Google knows its search engine isn't very good anymore!
And
> That is why they're increasingly trying to pitch their service as a > recommendation engine with special AI sauce, not an index of the web
We provide some content discovery tools (Discover, News), but those are offshoots made possible by our core mission and competencies: organizing the web’s information and making it accessible to users (yes we really say that, though frequently with a wink and a nod). We don’t try to be an index of the web, but instead as the best place to find out anything.
For a high-level read (and some in-depth coverage, like the rater guidelines) on what I’m taking about, consult this page: https://www.google.com/search/howsearchworks/mission/web-use...
Just an example: recently I made a search about korean pilots. Nothing controversial. First page results were all big media. In particular I noticed an article on NYT which was a badly made summary of a blog post that perfectly answered the question in the original search and it was even linked as source. Yet this blog wasn't even in the first page of results, but it would have been years ago. And that's true for all kind of queries. We always get those pathetic summaries made by big media of much better blogs by small companies and normal people who it seems don't deserve the first page anymore (despite being the original creators of so much quality content thst get rewritten by the big media). And let's not start on how inaccurate results are when a search is even slighly controversial.
Google shifted from being a web search engine to a promoter for big media companies. They should start advertising the company as such instead of lying to consumers.
Actually, it probably would have been even worse in the old days of page rank rules all. I guarantee you that there is a lot more relinks to the NYT article than to its source.
Are all of the news article pointing to the blog source ? Yes (well not even that actually). But the rest of the web, the reddit, the facebooks, the twitter, the everything points to the news article on NYT, Fox, ...
>We have to, first and foremost, improve things for our users. That means a radical focus on improving the relevance, quality, and accuracy [...] //
You admitted you don't have a longitudinal view. Results really are less accurate, less relevant; we're not hating on G.
Google appear to have decided to target people who don't know what they're looking for (accuracy and relevance aren't so important), and target natural answers to natural questions rather than focused, targeted search (which was what made its name).
It's like if Nike switched to making clogs but they looked just like their traditional trainers.
Now, that's sorta fine, but it's tiresome as hell to have to alter search settings every.. single.. time.., why am I logged in again? And, still not find what you're looking for, but instead some curated set of mass media pages that only have a cursory relationship to your keywords.
You can read whatever guidelines you like, but "clogs" aren't going to work like "trainers".
> You admitted you don't have a longitudinal view. Results really are less accurate, less relevant; we're not hating on G.
Yeah, whatever their intentions, sometime in the '07-'09 time frame they very obviously changed things in a big way and appeared to simply give up trying to separate spam from small-time sites, and just down-ranked the lot. The search results for certain types of queries are somewhat better now, as a result. The search results for other types of queries have gone from "pretty much perfect" to "totally useless", though. Anyone who's been a heavy user of Google for a long time has noticed this.
But how do you attribute this to Google and not to the web exploding in size several magnitudes.
Finding a needle in a 3ft haystack is one thing, finding it in 3 miles of hay is another.
I feel it's not Google getting worse for some queries but the internet is just so much larger now, and the evidence is that the other major search engines don't bring back that same simplicity and ease of finding high quality technical information from smaller sources that I also remember from the late 2000's
Eh, it happened pretty quickly and in the middle of a bunch of "we're making our search experience better!" stuff around the same time. I only gave a range because I'm fuzzy on the exact year it went to hell. Notably, the failure mode isn't that it finds a bunch of spam sites that happen to contain the keywords I want or have great SEO for those keywords, which is what one might expect if it were a wheat/chaff thing. In fact the more obscure and specific I get the more Google's results seem to try to wrestle me back to the middle, and the more it shows me page after page of blogging-as-income and similar lowish-value sites that are wildly far from what I was trying to get, and are in no way trying to masquerade as anything like the content I was looking for.
I think the others don't go against this trend because they also don't want to spend more money fighting spam to try to salvage a use case / usage pattern that is at this point so old that a lot of Web users have never seen it work, and that many users even back then didn't use effectively.
Those of us who relied on it heavily disagree strongly with the notion that Google only improves their search, though. For me it's much worse today—to the point of being entirely useless, in fact—for a significant percentage of queries I'd like to try against it, and noticeably worse though still usable for a bunch of others. It is definitely better at sending me to Wikipedia and at finding local restaurants or whatever, though, which is the kind of thing that probably constitutes 95+% of all queries against it (throw in people typing "example.com" into search rather than the address bar and it's more like 99%), so I'm sure their metrics look great.
I use www.searx.me with Google, Bing and DDG results weighted, and in my opinion the results are flat out better than Google alone.
> In every way we can measure, our search engine is the best it’s ever been.
To take a quote from Jeff Bezos, "The thing I have noticed is when the anecdotes and the data disagree, the anecdotes are usually right. There's something wrong with the way you are measuring it." Metrics are important, but metrics definitely aren't everything.
This provoked a wry smile from me too.
"I don't find my search results very useful."
"Yes you do! Look at these numbers!"
There's a kernel of truth in there, but taken to its conclusion,Bezos is saying that days is irrelevant unless it's a cheaper way to collect anecdotes. I don't think he agrees with himself on that.
No, sorry, the grandparent post is absolutely right. The search has been getting worse and worse every year to the point where I can't find specific articles I _know_ exist anymore. I know the article exists, I know it's indexed by Google, I know what it's about and I still can't find it after 5-6 increasingly sophisticated queries.
I see the flip side of this as an admin of a small website. 10 years ago I used to get search engine traffic to articles based on queries about subjects of those articles. Today, this is just gone. I can't find my own damn website without explicitly using its domain name. Even if I search for a specific subject and go though multiple pages of results. Google thinks that one-paragraph summaries from content farms are more relevant that an edited multi-page essay from a website that existed for over a decade.
Additionally, the article mentions another thing I've noticed. Google is obsessed by how frequently the website is updated. So, what if I spend extra time to write, verify and edit all my content? From what I cant tell, this reliably gets you ranked lower than website that crank out garbage in huge quantities.
I've been using Google for nearly 20 years and anecdotally it HAS gotten worse precisely because it has become harder and harder to make precise queries.
> "Google knows its search engine isn't very good anymore!"
It is possible for this to be true and for the koolaid at the plex to be true - as the answer is highly dependant on what the end user is searching for.
I'd say that a decade or so ago, the mission stated was actually being done - somewhere around the panda and penguin updates, the results changed a lot - and in the war to destroy spam and seo, google also killed a lot of good sites by burying them in the results.
I get that around that time there was also a lot of pressure from multiple groups to censor the results even more.. and so veering from the mission of organizing the web and delivering what the user is looking for was somewhat hijacked by gov bodies, powerful groups like the mpaa and such - and maybe it was larry paige? that was supposed to be the adult pulling the reins?
Whatever the multiple causes, there was a time when people could find all kinds of things with google and even google shopping.. then things started to change and change a lot they have.
Today's google is a homogenized, censored and curated skewed view of the world that is damaging those who trust google to show them what is on the internet.
Not for all searches mind you. For many searches it is the best version of the best ever search engine. As far as being a yellow pages of the future, you guys have nailed it.
I have no idea what the wink and nod is supposed to mean ("organizing the web’s information and making it accessible to users (yes we really say that, though frequently with a wink and a nod)" - but if it means, wink wink - we used to show a lot of stuff and now use a bait and switch hiding being esoteric "algorythms" that curate via 'user testing' - results - blah blah.
Please. The algorythm has been made to censor in many ways. It may be too complicated to explain, and trade secrets may be the real reason you don't show it - but whatever.
Please, for people in the main areas of the inside - it is disingenuous to be spouting the kool aid like it's some kind of truth.
It may be mostly true, but the 10% or 15%? of what is hidden completely and the xx% that is downranked into 'might as well not exist" - can you imagine if a few people got together and convinced the world that they have the actual true holy book and yet it was selectively censored by 12% - it might be true, but not really true..
and yet we are here today where big G is trusted by millions - and they have already targeted the next billion to acquire - and it appears people on the inside are repeating this false mantra like they are some kind of prophets of perfection.
As things continue to be censored more and more, big G becomes more and more like the yellow pages, with some 'trusted sources' thrown in for added content.
You've crowdsources a yellow pages that can be updated daily instead of once a year. You've selected some trusted sources about a few topics and point people to them.
You do not make the web accessible to users, only some of it. You do not make "best place to find out anything." - you provide the best place to find some select things that other people have made or given you updated info about.
The worse part of this whole thing is that the trust built up years ago by the brand has gotten a lot of people to convince a lot of other people that "just google it" is the way to find truth on the web, and you have taken over default search and find on most devices hijacking other possibly less censored options. You have hampered the abilities of many to share by making ideas and speech disappear.
However, I am glad that someone who works at google has said something. The silence from the G men / women / ts is more mocking. I wish Matt C had never left the plex and communication channels he shared.
remember when google would put a notice at the bottom of some results with something like 'some results not shown due to dmca, we have been forced by law to hide some results, see this chilling effects page about how this hurts...
now google hides more things itself.
and yet the same old lines are spewed "we index and give users access to find things on the web".
Come on, how much of the web is actually available to the end users through google and how much is not?