I work at Alphabet and I recently went to an internal tech talk about deploying large language models like this at Google. As a disclaimer I'll first note that this is not my area of expertise, I just attended the tech talk because it sounded interesting.
Large language models like GPT are one of the biggest areas of active ML research at Google, and there's a ton of pretty obvious applications for how they can be used to answer queries, index information, etc. There is a huge budget at Google related to staffing people to work on these kinds of models and do the actual training, which is very expensive because it takes a ton of compute capacity to train these super huge language models. However what I gathered from the talk is the economics of actually using these kinds of language models in the biggest Google products (e.g. search, gmail) isn't quite there yet. It's one thing to put up a demo that interested nerds can play with, but it's quite another thing to try to integrate it deeply in a system that serves billions of requests a day when you take into account serving costs, added latency, and the fact that the average revenue on something like a Google search is close to infinitesimal already. I think I remember the presenter saying something like they'd want to reduce the costs by at least 10x before it would be feasible to integrate models like this in products like search. A 10x or even 100x improvement is obviously an attainable target in the next few years, so I think technology like this is coming in the next few years.
This is so true. Some folks in Ads also tried to explore using large language models (one example: LLM is going to be the ultimate solution for contextual targeting if it's properly done), but one of the major bottleneck is always its cost and latency. Even if you can afford cpu/gpu/tpu costs, you always have to play within a finite latency budget. Large language model often adds latency by order of seconds, not even milliseconds! This is simply not acceptable.
I think Pathways is one approach to tackle this issue at scale by making the network sparsely activated so the computation cost can be somehow bounded based on difficulty of each query. This effectively gives Google knobs for the axis across computational cost and the result quality by limiting the size of network to be activated. If it turns out to work well, then we might be able to see it incorporated to Search in a foreseeable future.
That's the thing though, Google doesn't have to release this with Search or in Chrome. It could be a separate product that they can gate access to (charging say $5/mo for 'x' queries a day)? Or, API the model behind GCP? But: Outside of DeepMind, there's nothing comparable from them (in terms of utility AI).
This is the first thing which came to my mind as well, pay per use!
This is the problem of Google, or almost every other big techs. Its infrastructures, products and businesses are designed to serve at least hundreds of millions of users. This works really well for established products but significantly elevates the launch bar for new products, even seemingly easy projects like "why not having this as a small experimental website?". I won't be surprised if someone in the research team actually tried to bring up a small demo site but immediately found a showstopper from product counsels or internal AI guidelines...
Launching a full-fledged paid product is even harder, I guess you'll need to secure at least 3~40 headcounts just to integrate this into many subsystems inside Google. And this needs some senior executives driving the project since this is a cross-organization project between research and products. This creates a structural problem, in that they usually expect bigger impacts from these kind of projects to justify the cost. It's possible to pursue without involving top-down decision makers, but usually that kinds of project tends to fail to create consensus since everyone has different priority.
So "a separate small, experimental product" is not going to work unless 1. the model becomes fully productionized, generally available inside the company so a single VP (or even director) can quickly build a prototype to demonstrate or 2. someone successfully proposes a convincing path to the major billion user product to draw senior executive's attention or 3. the research team decides to build their own product team from scratch and aggressively invest into the sub team.
I concur, but I thought there's a reason Area 120 exists?
I am already willing to pay 15-20 dollars a month for this. Google will fall behind as others give this out as a paid service.
Don’t need ads for something this useful.
Googles Ad revenue from US alone could be 100B. If there are 100M converting users, that's 1000$ per user. 200$ per month cannot get you got. Think more like 100$ per month
Why should serving only this model cost as much as all of Google ?
GPT3 costs something like 5 cents a query. At 20 dollars a month, that would be 400 queries a month. I don't know about you but I'm pretty sure I do at least an order of magnitude more Google searches than that.
I don’t think every query needs this
Way off. The pricing for gpt3 is $0.02 per 1000 tokens for the largest model. 1000 tokens is about 750 words.
Google will just acquire those startups, more than likely.
From my knowledge, the cost of large language model search engine will be closer to $150~200 subscription per month than $15~20 in the status quo if the implementation is done naively. The cost will go down rapidly, but it's just not there yet.
I would consider paying $150-$200 / month for chat.openai.com access, especially if it continued to improve. It is an astonishing tool for learning and productivity. December 1, 2022 will be a day we all remember.
I agree. I haven't felt this excited by a new technology since the WWW. I can already solve hard problems with it which otherwise would require hiring consultants or spending inordinate amounts of time doing research. It's absolutely game changing.
What hard problems? I'm only using it to clean up my writing as a non-native speake
You might. Most people won't, especially not with how easily it can give you a wrong answer.
(I am extremely bullish on probabilistic productivity tools)
I think this is the intermediate solution. A Google Search Plus until economies of scale kick in. Most users will still prefer free slightly shittier search results but you can capitalize on the minority willing to pay and make a decent business out of it.
How up to date are LLMs likely to be for search? chatGPT is behind by a year or more. How quickly can an LLM be updated with recent content? That would seem to favor Google for the latest stuff.
The idea that LLMs will kill Google seems to me completely absurd. Name any other organization as well-positioned to create and exploit LLMs.
All the existing social platforms could also implement ActivityPub and have it working in a week. Name any other organizations that are as well-positioned to make the Fediverse a reality.
They [don't] do it, because they have a business model. Same goes for Google. The problem for google is that apparently this other tool is already available, today, though the website is currently overloaded so I can't reach it.
But if that site starts working for me, later today, why would I ever ask Google anything again?
> All the existing social platforms could also implement ActivityPub and have it working in a week. Name any other organizations that are as well-positioned to make the Fediverse a reality.
That's not a good analogy. There are architectural reasons why AP/fediverse will never work, no matter how hard anyone tries. It is not business reasons that prevent, say, Facebook from adopting ActivityPub. They are prevented from adopting it by critical thinking.
Back in the 90s, when mcdonalds.com was owned by some yokel who thought to register it before anyone else, I used to say that they couldn't do capitalism on the internet, and look, they pulled it off! We only had to throw every shred of social cohesion out to make it happen, but hey, the attention economy is in full swing!
Rubbish, lad. These platforms manage accounts in their millions within the garden every day, and you're telling me that they can't manage to open up a hole in the API to let someone add and read posts that way, rather than through their sadistic web interfaces? After everything they've already done?
More to the point, ActivityPub is just the current popular prototype, the Bitcoin if you will, of the federated social space. We'll get it sorted just fine.
I'm also largely skeptical of the claim that Google is going to completely drop the ball here, but this is classic Innovator's Dilemma - sometimes a business can't effectively segment their existing customers enough to introduce a higher initial cost but ultimately better technology.
I think a Google Search Premium that cost $50/month would go over pretty poorly with Google's existing customer base (advertisers), but a startup can experiment with the business model with the right early adopters (e.g. Neeva).
I mean, they’re already using LLMs. Namely, BERT.
There is no exact definition but BERT would probably just be a LM, not a "large" LM.
Is the exact definition important? The point is, they developed a thing and integrated into their core product. BERT allows them to handle conversational queries much better than before.
I think it does, because LLMs allow things that LMs like BERT don't - like answering complex question on their own etc which is being discussed in the context of this thread.
I'm thrilled we got to play with ChatGPT long before Google tried putting something like it directly into search. As neat as it is, it's also wrong, insistent that it is right when it's wrong, and frequently extremely biased to the nature of "Google probably shouldn't have fired Timnit Gebru for being right".
Hopefully by the time Google gets this implemented based on cost and latency metrics, we'll have better controls to hold Google accountable for doing so.
> It's one thing to put up a demo that interested nerds can play with, but it's quite another thing to try to integrate it deeply in a system that serves billions of requests a day
What’s funny is that Google itself started out as a “demo that interested nerds could play with” — searching your own name on the internet was so squarely a nerd move in the late 90s.
Google’s disruption here does not lie in the “ChatGPT or LLMs wil kill Search” realm. Rather, the fact that there isn’t a small-scale, dozen users product leveraging LLMs coming from Google, oh idk maybe something like Quora or Google Answers, or maybe the “Google Flights experience with Assistant”
When google started search was a big business with a number of competitors. The most popular, Yahoo, was one of the biggest sites! There was even a meta-search engine (dogpile, I think?) that would run searches against several of the top contenders, but personally I mostly used AltaVista until google killed them.
Google did have a key product insight that you didn't need the "web portal" cruft -- just good search was enough.
> and the fact that the average revenue on something like a Google search is close to infinitesimal already
Isn't Search ad revenue >$100Bn per year?
Isn't that >$0.07 per search?
From what I can tell, yearly Search ad revenue is in the neighborhood of $104 billion [0], and the number of yearly searches served by Google is somewhere in the neighborhood of 3.1 trillion [1]. This brings the revenue per search to somewhere between 3 and 3.5 cents.
[0] https://abc.xyz/investor/static/pdf/20210203_alphabet_10K.pd... (page 33)
[1] https://www.oberlo.com/blog/google-search-statistics
The problem for Google isn't just technical, it's organizational.
The entire organization and all its products are built around ads. If a new product comes along that drastically reduces the number of pages a user views, what happens to the ad revenue?
Right now, every click, every query is an impression. But if there's an all-knowing AI answering all my questions accurately, what incentive do I, as a user, have to search again, scroll through pages, and look around multiple pages?
Google will have to adopt a radically new business model and there's organizational inertia in doing that.
They could still serve AdSense-style ads inside the generated content.
Yeah, but those ad impressions and clicks go down drastically. Is Google ready to live in a world with 50% lower revenue?
> Right now, every click, every query is an impression.
The vast majority (maybe 95%?) of my searches don't show ads.
Wow. Very different from my experience. If I'm not using Brave, often the first 5ish results are ads.
> The entire organization and all its products are built around ads.
Citation?
I assume ads are a big part of Google but I suspect it’s not organized around ads.
Eg I assume the GCP teams don’t report to any ad teams.
I bet Gmail team -which does show ads- is primarily managed to optimize for the paid enterprise customers and they just have an ads guy shove ad boxes in where they can.
I bet no one at Nest reports to an ads team, and they’re organized around making money on a per-device basis instead.
Is Google good at adopting new successful business models? Ask stadia. But I bet there’s plenty of organizational clarity that alternative revenue streams are important.
Disclaimer: I don’t know the internal structure of these teams
> I assume ads are a big part of Google but I suspect it’s not organized around ads.
Other than GCP, how many products can you name that are not monetized by ads?
Advertising is nearly 80% of their revenue. It has remained stubbornly near that mark despite the massive list of products they keep releasing (and shutting down early).
Large organizations tend to coagulate around their profit centers. Google isn't any more immune to it than, say, IBM.
> I bet Gmail team -which does show ads-
How many people even experience ads in gmail? They aren't there on Workspace or EDU. They aren't there on gmail.com unless you are using the tabbed inbox with the Promotions tab enabled, and you visit that tab. Which, honestly, who would?
> How many people even experience ads in gmail?
I'll be honest, I was under the impression they stopped showing ads on Gmail when I was writing this. I actually google'd it but couldn't find any news source citing that google stripped ads off gmail so I omitted to mention it.
I agree with others that this would be something that was worth paying a monthly fee for, iff it was good.