It is suspicious that OpenAI decided to generate 300 billion output tokens from a model still in training, right after learning there was a credible chance that a major math proof was in that model’s training data. Obviously there are reasonably plausible explanations for each step, but it does sort of feel like parallel construction.
I think people are focusing on the training data issue too much. If the data was contaminated, I can still blame that on negligence.
But, at least with the Navier-Stokes solution, it's clear [^1] that they learned that Alpöge and Buckmaster were getting close to a solution and learned of the general approach they were taking. Only after learning the secret to cracking the problem did they send the first prompt.
What makes this worse to me is the intention. They intentionally threw $15 million in compute at the problem in order to scoop the result. They intentionally left Buckmaster and Alpöge out of the citations.
Data contamination should be enough to disqualify them from the prize, but I can believe it to be accidental. On the other hand, someone made an intentional decision to scoop the result by throwing money at the problem. That's so much worse.
[^1]: That's the timeline claimed by Buckmaster, and no one from OAI has disputed it.
> the secret
So such thing existed. In fact, what they learnt was some progress existed, not what the specific progress was.
I think you’re overlooking what I’m implying here. It’s not that they knew contamination was possible but they went ahead anyway. To spell it out just a little bit more: learning the answer might be in model X’s training data made them believe that model X specifically might be able to solve the question, and they were able to very quickly find enough certainty about the former to commit millions of dollars to the latter.
> and learned of the general approach they were taking. Only after learning the secret to cracking the problem did they send the first prompt.
Do you have any evidence of this? They don't dispute the timeline, but they never said they knew what Levant/Buckmaster were doing.
It's in OpenAI's first announcement that they had solved the problem.
> Only after learning the secret to cracking the problem did they send the first prompt.
Which quote in the announcement post provides evidence for the above quote?
“ On Tuesday, September 1, we heard rumors that two Millennium Prize problems had been resolved. Inspired by these rumors and by the step change in performance of our internal model, we launched an effort to evaluate it on all open Millennium Prize problems and a few other high-impact problems.”
- https://openai.com/index/navier-stokes-solution/
They do not explicitly admit to knowing about NS specifically, but are extremely explicit that they tried to scoop some potential millennium prize winners.
So then they DIDN'T "learn the secret to cracking the problem". They simply knew that part of the problem was solved. Knowing a problem can be solved and knowing the solution are not the same thing.
Yeah, and suckers are born every day...
I like how the comment below summarizes it:
> learning the answer might be in model X’s training data made them believe that model X specifically might be able to solve the question, and they were able to very quickly find enough certainty about the former to commit millions of dollars to the latter.
They don’t need to know, because their IP stealing machine knows for them. They just have to buy enough compute, and someone else’s work is theirs.
I said “very quickly find enough certainty” to suggest hypothetical situations like “someone searches the conversation logs, confirms for themselves the solution is present, then shares the confidence gained from this knowledge without explicitly sharing the knowledge itself”. That person could recuse themselves from the project so the project can still legally make claims like “conversation data was not used” in the announcement, while also knowing that they are guaranteed to get there if they just pull the lever enough.
(Naturally, I have far too much respect for OpenAI’s legal team to suggest this is what happened in their project.)
The claim that OpenAI somehow used the mathematicians' ideas to leapfrog them seems unsupported at this time and IMHO it was irresponsible to bring it up because credulous people will immediately believe that narrative.
And from my perspective, if some math folks typing in a few questions to OpenAI provides sufficient training data for OpenAI to solve a big problem... that's amazing! A few conversations/prompts out of the billions that OpenAI trains on lead to this result- that means there is an awful lot of low-hanging fruit that could be exploited cheaply.
I'm curious how many other 300 billion output tokens OpenAI has "paid for" that have resulted in no breakthroughs.
Either they had a pretty good idea that investing this type of money in that compute on a model in training would lead to these specific results, or they gambled with other people's money.
I want to hear about the gambles and expenditures they don't brag about. In America's energy economy, there's finite resources to expend.
The (unprovable, yes, without OpenAI being willingly transparent) argument is that openAI constructed a prompt to scoop them using some inside knowledge about the approach, which they allude to in the announcement.
In the transcripts, Brubeck is very cagey and evasive about the prompt, when it was supplied, and its contents.
So because they didn’t admit to it they didn’t do it?
> but I can believe it to be accidental
What accident is it when the system is designed to function that way?
Their claim is that training on their solution is "unlikely but possible".
Consider this scenario.
Has a google crawler read my new novel, which I may or may not have posted on my blog, page by page, as I wrote it?
Can you, without knowledge of what I have actually done, claim that the google crawler has not seen the novel?
Without any evidence that I have posted the novel online, it might be tempting to say that the crawler has not seen the novel, but what if I were in an adversarial position against Google on this topic and were challenging them to make that claim. You would wonder if I were hoping Google to overreach by making a definitive claim without taking into account some action that they had no knowledge of. It becomes difficult to use the scientific expression "There is no evidence for this" when there is an accusation of malfeasance because it can be so easily be conflated as "You can't prove we did it". It seems like the best you could say would be 'Unlikely, but possible'
I'm not taking them at their word, sorry. Genuinely, there is no reason to.
> They intentionally threw $15 million in compute at the problem
what? really?
Yes. Maybe much more:
> Such intensive use of AI doesn't come cheap. In a post on X, LisanBench, an LLM benchmark evaluator, estimated that the output tokens alone would cost about $6.5 million at OpenAI's average consumer price. Including the far larger volume of input tokens, the post estimated the total could reach $10 million to $40 million.
https://www.businessinsider.com/openai-math-problem-solved-t...
But think of all the IPO Monopoly money they just generated.
That's their API pricing. There's no way they actually paid $15M in compute. I'd say much more likely it's in the order of $1M.
Who are you who is so wise in the ways of a private company's internal cost accounting
When I worked at Google, we spent $100M in power on protein folding and drug discovery (this was long before AlphaFold). Never underestimate the willingness of smart rich people to invest in speculative science.
> They intentionally left Buckmaster and Alpöge out of the citations.
No, they asked if they could do a joint publish.
No, they asked one guy to do a joint publish conditioned on leaving the other collaborator out, with veiled threats. The joint publish part smells awfully like admission of guilt given there’s absolutely no reason to do it if you believe you independently arrived at the result using only public prior work. The leaving out collaborator part is outright academic malpractice. Disclosure: I was an academic once.
To add: with a requirement that he rewrite the proof to credit OpenAI.
This would be a new publication with OpenAI's novel solution. Buckmaster and Alpöge did not arrive at the NS solution, they used an approach that also happened to be used in OpenAI's proof.
> there’s absolutely no reason to do it if you believe you independently arrived at the result using only public prior work
Unless the goal is to assuage the existential pain felt by many mathematicians with respect to what may feel like increasingly inconsequential efforts. It also seemed like a way to build good will amongst knowledge workers dealing with similar issues, with the goal of showing they are dedicated to easing the transition for them as society parades into the future. This has clearly backfired.
> The leaving out collaborator part is outright academic malpractice.
They would not be leaving a collaborator out. Again, this would be a new publication. Here's the quote from Buckmaster's post [1]:
"Two proposals were offered to me. The first was that we post our Euler result, and that OpenAI post its Navier-Stokes result the next day. The second was that, after posting Euler, I alone write a paper presenting the Navier-Stokes result, acknowledging that an internal OpenAI model had resolved it."
[1] https://cims.nyu.edu/~tristanb/statement.pdf
These are utter bullshit that breaks every norm, but may be convincing enough for some non-academic OpenAI cheerleaders who post dozens of comments on these topics.