root_axis 1 hour ago

What value is there in just dumping something into an LLM and saying it produced errors?

edit: to all the people responding that LLMs are good at math, I'm going to paste a reply I wrote downthread...

I'm not arguing that LLMs aren't useful tools for mathematicians, I'm questioning the quality of discourse that amounts to "I pasted this into the LLM and it said there are errors"

  • markasoftware 1 hour ago

    Have you tried doing this recently? For my work on software at least, the latest LLMs have a very low false positive rate for bugs. They almost never hallucinate issues. My understanding is the same applies to math.

    • root_axis 1 hour ago

      I'm not arguing that LLMs aren't useful tools for mathematicians, I'm questioning the quality of discourse that amounts to "I pasted this into the LLM and it said there are errors".

      • fasterik 1 hour ago

        There's value in knowing whether a claim is true or false. Plugging things into an LLM doesn't necessarily give 100% certainty or tell us why something is true or false in a human-comprehensible way, but it can move our credences and suggest directions for investigation.

        • mtlmtlmtlmtl 24 minutes ago

          If you're not even gonna share what the supposed errors were, the value is pretty much infinitesimal.

          Surely, GPT said something more elaborate than "nuh uh".

      • whimsicalism 1 hour ago

        classic HN isolated demand for rigor. i found it interesting

      • jiggawatts 27 minutes ago

        Think of it as a (fuzzy) compiler.

        Now think about complaining that someone "merely pasted my code into a compiler and said that there are errors in my code."

        That's nuts, right? If a compiler -- a kind of proof engine -- finds errors, there are errors. Fix them! Don't complain about the usage of a compiler, the errors were real, they're there, they need to be fixed.

        The fuzziness part is both useful and a liability, but the error rate is dropping.

        Some numbers for you:

        A year ago, I ran a huge legacy codebase through Gemini Pro 2.5 and it spat out several thousand bugs, but manual inspection showed that 80% were false positives and only 20% were worth fixing.

        I did a similar exercise with GPT 5.6 more recently, and the hit rate was 80% valid and 20% false positive. The human developer got into an argument saying that they thought all 100% were false positives. He lost. The AI could easily produce "repro" snippets that triggered every issue (mostly security vulnerabilities, but some correctness ones too).

        These days, if you're publishing anything with verifiable claims in it, and not running it through an AI first, it's the direct equivalent of uploading open source code to GitHub without bothering to check that it compiles first.

        It's sloppy.

    • shmoil 1 hour ago

      Pre-chatgpt 5.5 it hallucinated quiet a bit on hard math. Starting from 5.6 it is incredibly good.

    • mahogany 1 hour ago

      I would guess that LLMs are much worse at math without having a theorem prover to validate approaches with. Since this draft is not formalized, an LLM is operating purely on the text which is more tenuous.

    • gregorygoc 1 hour ago

      I’m doing it daily and it finds a bunch of issues which are not issues in practice.

    • simplyluke 1 hour ago

      I suspect LLMs are much better trained and more reliable at bug identification in react apps than draft-state proofs on the bleeding edge of math, on problems that are specifically highlighted as extremely difficult to solve without error in their training.

    • SubmarineClub 1 hour ago

      Cool story.

      I have the opposite experience in my line of work.

      • chaps 1 hour ago

        What's your line of work?

    • acedTrex 45 minutes ago

      > the latest LLMs have a very low false positive rate for bugs.

      Man people live in different worlds, astra for me finds 25/6 bugs all the time.

  • arcastroe 1 hour ago

    It's being hurriedly announced as fully human work. I wonder if the author had carefully avoided ever running the full argument through frontier LLMs (OpenAI/Anthropic) for fear of it being stolen or incorporated into AI training.

    This may be the first time this proof is being proof-read by frontier models

    (this is pure speculation, but provides an answer to your question)

    • hkmaxpro 1 hour ago

      I also suspect the author actively tried to avoid using AI at all.

      Another possible reason: AI will correct your mistakes by using ideas of others. A proof will unlikely remain human-only, even if you just use AI to check for your errors.

  • mtlmtlmtlmtl 29 minutes ago

    Seriously. At least be constructive. Share what the errors were. Make an effort to understand them, anything. That tweet just amounts to low effort FUD.

  • zone411 12 minutes ago

    Very high value. The LLMs are excellent at auditing math proofs.

JoshTriplett 1 hour ago

Can confirm that current GPT (with no guidance) took 5 minutes to find what it claims are errors with counterexamples. (I have nowhere near the level of state-of-the-art mathematical knowledge and experience needed to validate them. I'm posting this solely as additional confirmation of "there may be issues with this proof that an actual mathematician together with LLM assistance would be able to properly evaluate".)

  • croes 1 hour ago

    And that is the problem.

    AI claims but too few people can check.

    Won‘t get better from now on

    • gigabyts 1 hour ago

      I mean I guess there is incentive for some human curiosity and trying to learn more than ever, small bit but still to get a standard for verification.

  • amritananda 1 hour ago

    I wouldn't be surprised if these models were aligned to extreme skepticism (to the point of hallucination) when e.g. a user proposes a solution to a Millennium Prize Problem.

    • JoshTriplett 1 hour ago

      1) There's a meaningful difference between "I'm not buying it based on my priors" and "I'm claiming to have a counterexample". I'm looking forward to seeing analysis by other mathematicians.

      2) Any model with properly updated priors will have less skepticism of a Millennium Prize Problem being solved now.

  • tappio 48 minutes ago

    Did you try the same with the other LLM assisted proofs? I would be curious what your GPT model says about them. I fed Alpöge and Anthropic statement and details to Gemini 4 hydrogen, and it said that there are "major problems and red flags in both the text and the current verification status", and that it is impossible to verify the 250k pages of slop, and there are there are "Extraordinary, unsubstantiated mathematical leaps in the sketch". So I would take these surface level reviews by LLMs with a grain of salt.

    Anyhow, I hope that they are both right!

tasuki 36 minutes ago

> but either way, alpoge/anthropic certainly has the claim to the first proof

Is that a thing? Don't people care more about the proof's elegance and insight than its timing?