> However, this result made it clear that the reliability of state-of-the-art LLMs is fundamentally limited: If they need to complete every step correctly in order to solve a task, after a certain number of steps they will almost surely fail as a result of an underlying propensity to make errors, even when the answer should be obvious. While an error rate of 1-in-1,000 seems low, and would be great on a traditional LLM benchmark, on a task that requires successful execution of thousands of steps in a row, such a system results in inevitable failure.
What a relief to see an obvious problem actually acknowledged. I can't even guess how many times I've been shouted down about this exact topic in the reasoning debates on HN, or seen papers just kind of glossing over it as if it were a non-issue.
The next really natural question is.. if you're committed to decomposing a problem into tons of microsteps and voting.. why aren't we just embracing hybrid symbolic systems? The decomposition step kind of implies you're in a problem domain where variables separate out somewhat cleanly and that this should be doable. As far as I can tell the "voting" discussed in the paper is about candidate outputs, i.e. solutions to subproblems? If you switch to hybrid symbolic systems, then you can vote on candidate inputs to solvers and at least be damned sure that their output is always correct.
Also the success of chain-of-code compared with chain-of-thought approaches could actually imply that having no real solver is maybe not the obstacle you'd expect! Maybe you can invent a semiformal logic just in time that appears to be expressive enough to encapsulate the problem domain, and have the LLM emulate a nonexistent solver. If the error rate with this sort of approach is still too high, then at least you know concretely what solver or formal-language you need to implement in order to improve.
> While an error rate of 1-in-1,000 seems low, [...], on a task that requires successful execution of thousands of steps in a row, such a system results in inevitable failure.
This is also why (edit: non-LIDAR) FSD cars are an illusion.
FSD isn't, and never was, a sensor problem. It's an AI problem. Always was. Always will be.
Humans drive around with two mid-tier cameras on a pivot mount. Which means that any sufficiently advanced AI can do the same.
When a FSD car gets into an avoidable collision, you dump the blackbox data, and what do you see? You see that the cameras had it. All the information the car needed to avoid a collision was right there in the visual stream. The car had every bit of information it needed to make the right call, and didn't make the right call.
You can acknowledge that, and focus on building better AIs. Or you can neglect AI altogether, and have a car with 6 LIDARs drag a pedestrian, because it had all the sensor coverage but zero object permanence.
In simple terms, the LIDAR sensor will allow you to do "If object at X, don't go to X". But obviously, you need more than that. Old school Kalman filters for object tracking etc.
LIDAR doesn’t stop them bumping into anything. LIDAR is a sensor, it doesn’t recognise anything or make decisions about steering, acceleration, or braking.
You need more than the sensor, obviously. The point is that you don't need any AI to make a system this way that is substantially safer than a system based only on camera feeds and AI.
Briefly, the idea is recursively to decompose tasks into the simplest possible steps, recursively call (relatively small) LLMs as agents to execute one step at a time, and using a clever voting scheme to choose how to execute each step. The authors use this technique to get a relatively small LLM to solve Towers of Hanoi with 20 rings (1M steps). All of it using natural language.
The most obvious question is whether other tasks, more interesting -- less "rote" -- than Towers of Hanoi, can similarly be recursively decomposed into simple steps. I'm not sure that's always possible.
> into the simplest possible steps, recursively call (relatively small) LLMs as agents to execute one step at a time, and using a clever voting scheme to choose how to execute each step.
One issue I often run into with this stuff is the tightly coupled nature of things in the real world. I’ll fashion an example:
Let’s say you break a job down into 3 tasks: A, B and C. Doing one of those tasks is too much for an LLM to accomplish in one turn (this is something you learn intuitively through experience), but an LLM could break each task into 3 subtasks. So you do that, and start by having the LLM break task A into subtasks A1, A2 and A3. And B into B1, B2 and B3. But when you break down task C, the LLM (which needs to start with a fresh context each time since each “breakdown” uses 60-70% of the context) doesn’t know the details of task A, and thus writes a prompt for C1 that is incompatible with “the world where A1 has been completed”.
This sort of “tunnel vision” is currently an issue with scaling 2025 agents. As useful context lengths get longer it’ll get easier, but figuring out how to pack exactly the right info into a context is tough, especially when the tool you’d reach for to automate it (LLMs) are the same tool that suffers from these context limitations.
None of this means big things aren’t possible, just that the fussyness of these systems increases with the size of the task, and that fussyness leads to more requirements of “human review” in the process.
I've been experimenting with this with a custom /plan slash command for claude code, available here: https://github.com/atomCAD/agents
Planning is definitely still something that requires a human in the loop, but I have been able to avoid the problem you are describing. It does require some trickery (not yet represented in the /plan command) when the overall plan exceeds reasonable context window size (~20k tokens). You basically have to start having the AI consider combinatorially many batches of the plan compared with each other, to discover and correct these dependency issues.
>the LLM (which needs to start with a fresh context each time since each “breakdown” uses 60-70% of the context) doesn’t know the details of task A, and thus writes a prompt for C1 that is incompatible with “the world where A1 has been completed”.
Can't that be solved with sub agents? The main agents oversees on combines code and calls sub agents for each tasks.
Reasoning by analogy is great for intuition, but doesn’t guarantee real results hold. Consider “voltage is like water pressure in pipes, so if there’s a cut in my wire’s insulation, the device won’t get enough voltage” — clearly this is not true, even though it relies on an analogy that’s generally useful.
I really like that analogy, thank you for it. Also applies to “it’s overvoltage, so I just need to poke a little hole in it to let the excess bleed out”…
> "If there’s a cut in my wire’s insulation, the device won’t get enough voltage" doesn't follow from: "voltage is like water pressure in pipes"
I absolutely agree! In the same way, "an LLM can solve complex problems if it breaks them into subtasks" doesn't follow from "NASA breaks large projects into smaller parts"
IBM tried that with CMM (capability maturity model), it didn't work, the problem is NASA knows what they're building, rockets and satellites don't have any grey areas and everything is specified. Other things are less well defined, and the people specifying aren't rocket scientists.
This is a really good analogy because the complex intersections between multiple groups independently working and trying to collaborate together into a collaborative hierarchy towards one large goal was one of the things that hid a lot of the problems that led to the Challenger disaster, according to Feynmen.
The space shuttle’s design was also deeply flawed to the point it failed to do the core objective, significantly lowering costs. Instead the core mission was sacrificed to meet some arbitrary design goals such as being able to de-orbit heavy objects.
That’s the core issue with decomposition of tasks, you aren’t communicating back up the chain and finding globally optimal solutions unless the task is simple enough to be completely understood.
I'm pretty sure the problem with the shuttle was that it had too many (possibly conflicting) goals instead of one large goal.
It's manned, even though most launches probably could be done without crew. The deadly Challenger launch was risking human crew for something as mundane as launching two satellites into space.
Because it's manned, it has to be able to land at airports, because retrieving astronauts at sea is an unreasonable complication for launching a satellite. Damage to the wings will cause loss of the entire aircraft, something that is unlikely to happen to a capsule.
Because it is a horizontal landing system, the aerodynamics favor putting the shuttle on the same level as the external fuel tank, which exposes the wing to debris from the top of the external fuel tank. If you try building a vertical shuttle in KSP, you will notice that the wings give you too much control authority during launch. Fins are best placed near the bottom of the rocket.
It's reusable, which means wear and tear can secretly accumulate without you noticing. This significantly increases the design requirements for the critical components, like the SRB that had a poor "tang" design, which, as it turns out, was definitively not fit for reuse.
Hmm...
The key is to successfully decompose a big, hard problem into easier atomic sub-problems. However, the decomposition process itself is difficult, and this paper is not about that. They decompose a task using a human-written prompt.
> The approach relies on an extreme decomposition of a task into subtasks, each of which can be tackled by focused microagents. The high level of modularity resulting from the decomposition allows error correction to be applied at each step through an efficient multi-agent voting scheme.
Big if that the decomposition and the voting happen accurately for anything other than toy problems
The approach in the paper specifically addresses the case where an LLM can usually solve a task when it requires few steps, but fails for the same kind of task with more steps because it randomly gets a step in the middle wrong and then derails. It can't do anything for tasks that the LLM can't solve even when there's just a few steps.
In other words, it compensates for random error, not systematic error.
I was just thinking "these guys will talk about this graph for the rest of their lives", it's the best graph you could ever hope to put into a paper. Loved it.
In case you want to know what’s going on in the left side of that chart, they gave a log scale in appendix a. I was thinking it was silly to not just use that version on the top, but I guess log scales make big differences ’feel’ smaller.
A log scale is actually appropriate in this context from a first-principles perspective. Per scaling laws (and also general behavior of epsilon-probability of failure multiplied N times), you would generally expect more vs. less effective techniques to have multiplicatively greater or fewer steps until failure, not additively greater/fewer. Figure 1 is comical, but the appendix figure is the more scientifically appropriate one.
Especially since it's a recursive problem so each step is naturally broken up into subtasks. And the algorithm of what subtasks to break it up in to is public. This makes it much easier for it to get down to a case that the LLM can reliable solve.
I guess that the subtask decomposition of many (sub)problems is known and in the training distribution. How many real-world problems are resistant to divide-and-conquer? Presumably most/all of the unsolved mathematics conjectures. What else?
This has seemed to me to be the natural next step to turn LLMs into more deterministic tools. Pushing the frontier is nice, but I think LLMs have a whole different gear when they are able to self-decompose in a reliable way. Most of my success creating reusable LLM products came from determining where requirements/outputs need to be "hard" vs. "soft".
state = init_state()
while state is not complete:
state = LLM("You are a helpful assistant. The rules and format of the game is [...]. The correct strategy to use at each step is [...]. The current state is [...]. Output the state after making the next move")
The problem is how to even define a task using the English language and make sure there is enough entropy to infer the detailed intent. For it to be later split into zillions of small steps which can be executed over time by an LLM.
one issue I see is when steps in a plan depend on one another, when you cannot know all the next steps exactly before seeing the results of the previous ones, when you may have to backtrack sometimes
Right, it’s kind of like solving systems of linear equations. Some can be solved just by reordering, but most need you to handle all the constraints at once.
No mention of MoE. One would think this is a logical evolution of that but not a mention (that I saw). Its own rubric for the task, Towers of Hanoi, was admittedly weak.
LLM papers are starting to look like the last decade of JS frameworks and Tools. Only with less code and more academics, and thats disappointing, because I think a lack of pragmatism and grounding is now holding the field back...
> However, this result made it clear that the reliability of state-of-the-art LLMs is fundamentally limited: If they need to complete every step correctly in order to solve a task, after a certain number of steps they will almost surely fail as a result of an underlying propensity to make errors, even when the answer should be obvious. While an error rate of 1-in-1,000 seems low, and would be great on a traditional LLM benchmark, on a task that requires successful execution of thousands of steps in a row, such a system results in inevitable failure.
What a relief to see an obvious problem actually acknowledged. I can't even guess how many times I've been shouted down about this exact topic in the reasoning debates on HN, or seen papers just kind of glossing over it as if it were a non-issue.
The next really natural question is.. if you're committed to decomposing a problem into tons of microsteps and voting.. why aren't we just embracing hybrid symbolic systems? The decomposition step kind of implies you're in a problem domain where variables separate out somewhat cleanly and that this should be doable. As far as I can tell the "voting" discussed in the paper is about candidate outputs, i.e. solutions to subproblems? If you switch to hybrid symbolic systems, then you can vote on candidate inputs to solvers and at least be damned sure that their output is always correct.
Also the success of chain-of-code compared with chain-of-thought approaches could actually imply that having no real solver is maybe not the obstacle you'd expect! Maybe you can invent a semiformal logic just in time that appears to be expressive enough to encapsulate the problem domain, and have the LLM emulate a nonexistent solver. If the error rate with this sort of approach is still too high, then at least you know concretely what solver or formal-language you need to implement in order to improve.
> While an error rate of 1-in-1,000 seems low, [...], on a task that requires successful execution of thousands of steps in a row, such a system results in inevitable failure.
This is also why (edit: non-LIDAR) FSD cars are an illusion.
FSD isn't, and never was, a sensor problem. It's an AI problem. Always was. Always will be.
Humans drive around with two mid-tier cameras on a pivot mount. Which means that any sufficiently advanced AI can do the same.
When a FSD car gets into an avoidable collision, you dump the blackbox data, and what do you see? You see that the cameras had it. All the information the car needed to avoid a collision was right there in the visual stream. The car had every bit of information it needed to make the right call, and didn't make the right call.
You can acknowledge that, and focus on building better AIs. Or you can neglect AI altogether, and have a car with 6 LIDARs drag a pedestrian, because it had all the sensor coverage but zero object permanence.
> Or you can neglect AI altogether, and have a car with 6 LIDARs drag a pedestrian, because it had all the sensor coverage but zero object permanence.
False dichotomy, much?
> Humans drive around with two mid-tier cameras on a pivot mount. Which means that any sufficiently advanced AI can do the same.
Yes, if we can get their error rate below 0.000001%. Until we get there, sensors + old school computer-vision provide safety.
What makes you think that "sensors + old school computer-vision" gives you an error rate better than "completely fucked"?
In simple terms, the LIDAR sensor will allow you to do "If object at X, don't go to X". But obviously, you need more than that. Old school Kalman filters for object tracking etc.
Waymo has driven a hundred million miles and is far safer than human drivers.
Yes because they use non-AI techniques (LIDAR) to prevent them from bumping into things.
I should have said non-LIDAR in my comment, yes.
LIDAR doesn’t stop them bumping into anything. LIDAR is a sensor, it doesn’t recognise anything or make decisions about steering, acceleration, or braking.
You need more than the sensor, obviously. The point is that you don't need any AI to make a system this way that is substantially safer than a system based only on camera feeds and AI.
I think you're still in the edit window, FYI. (At least for a few more minutes.)
Nice!
Briefly, the idea is recursively to decompose tasks into the simplest possible steps, recursively call (relatively small) LLMs as agents to execute one step at a time, and using a clever voting scheme to choose how to execute each step. The authors use this technique to get a relatively small LLM to solve Towers of Hanoi with 20 rings (1M steps). All of it using natural language.
The most obvious question is whether other tasks, more interesting -- less "rote" -- than Towers of Hanoi, can similarly be recursively decomposed into simple steps. I'm not sure that's always possible.
This works because a problem could be broken down to a prompt which rarely hallucinates.
Most real world prompts can't be reduced to something so consistent and reliable.
Their key finding was that the number of votes grows linearly with number of prompts you are trying to chain.
However the issue is that the number of votes you need will grow exponentially with hallucination rate.
Its LLMs all the way down :-)
This can't be scaled to more generalised tasks. If you solve that then you've solved the hallucination issue.
> into the simplest possible steps, recursively call (relatively small) LLMs as agents to execute one step at a time, and using a clever voting scheme to choose how to execute each step.
It's like humans! Everything old is new again :)
Why not? That's basically how NASA manages large projects.
One issue I often run into with this stuff is the tightly coupled nature of things in the real world. I’ll fashion an example:
Let’s say you break a job down into 3 tasks: A, B and C. Doing one of those tasks is too much for an LLM to accomplish in one turn (this is something you learn intuitively through experience), but an LLM could break each task into 3 subtasks. So you do that, and start by having the LLM break task A into subtasks A1, A2 and A3. And B into B1, B2 and B3. But when you break down task C, the LLM (which needs to start with a fresh context each time since each “breakdown” uses 60-70% of the context) doesn’t know the details of task A, and thus writes a prompt for C1 that is incompatible with “the world where A1 has been completed”.
This sort of “tunnel vision” is currently an issue with scaling 2025 agents. As useful context lengths get longer it’ll get easier, but figuring out how to pack exactly the right info into a context is tough, especially when the tool you’d reach for to automate it (LLMs) are the same tool that suffers from these context limitations.
None of this means big things aren’t possible, just that the fussyness of these systems increases with the size of the task, and that fussyness leads to more requirements of “human review” in the process.
I've been experimenting with this with a custom /plan slash command for claude code, available here: https://github.com/atomCAD/agents
Planning is definitely still something that requires a human in the loop, but I have been able to avoid the problem you are describing. It does require some trickery (not yet represented in the /plan command) when the overall plan exceeds reasonable context window size (~20k tokens). You basically have to start having the AI consider combinatorially many batches of the plan compared with each other, to discover and correct these dependency issues.
>the LLM (which needs to start with a fresh context each time since each “breakdown” uses 60-70% of the context) doesn’t know the details of task A, and thus writes a prompt for C1 that is incompatible with “the world where A1 has been completed”.
Can't that be solved with sub agents? The main agents oversees on combines code and calls sub agents for each tasks.
Reasoning by analogy is great for intuition, but doesn’t guarantee real results hold. Consider “voltage is like water pressure in pipes, so if there’s a cut in my wire’s insulation, the device won’t get enough voltage” — clearly this is not true, even though it relies on an analogy that’s generally useful.
I really like that analogy, thank you for it. Also applies to “it’s overvoltage, so I just need to poke a little hole in it to let the excess bleed out”…
That one can work, briefly, depending on how conductive your tool is.
If air was highly conductive that analogy would totally hold.
"If there’s a cut in my wire’s insulation, the device won’t get enough voltage" doesn't follow from: "voltage is like water pressure in pipes"
So I don't really get your point.
> "If there’s a cut in my wire’s insulation, the device won’t get enough voltage" doesn't follow from: "voltage is like water pressure in pipes"
I absolutely agree! In the same way, "an LLM can solve complex problems if it breaks them into subtasks" doesn't follow from "NASA breaks large projects into smaller parts"
Well, corona losses are a thing, after all.
IBM tried that with CMM (capability maturity model), it didn't work, the problem is NASA knows what they're building, rockets and satellites don't have any grey areas and everything is specified. Other things are less well defined, and the people specifying aren't rocket scientists.
"basically" is doing a lot of work in this sentence.
I could imagine that even a small task at NASA might involve more knowledge and logic than the smallest task for a Hanoi's tower problem.
Depends on what is considered as small enough for the LLM to be resolved with a high confidence.
This is a really good analogy because the complex intersections between multiple groups independently working and trying to collaborate together into a collaborative hierarchy towards one large goal was one of the things that hid a lot of the problems that led to the Challenger disaster, according to Feynmen.
It is also what made the space shuttle possible in the first place, so I'd be careful about generalizing too much from that observation.
The space shuttle’s design was also deeply flawed to the point it failed to do the core objective, significantly lowering costs. Instead the core mission was sacrificed to meet some arbitrary design goals such as being able to de-orbit heavy objects.
That’s the core issue with decomposition of tasks, you aren’t communicating back up the chain and finding globally optimal solutions unless the task is simple enough to be completely understood.
I'm pretty sure the problem with the shuttle was that it had too many (possibly conflicting) goals instead of one large goal.
It's manned, even though most launches probably could be done without crew. The deadly Challenger launch was risking human crew for something as mundane as launching two satellites into space.
Because it's manned, it has to be able to land at airports, because retrieving astronauts at sea is an unreasonable complication for launching a satellite. Damage to the wings will cause loss of the entire aircraft, something that is unlikely to happen to a capsule.
Because it is a horizontal landing system, the aerodynamics favor putting the shuttle on the same level as the external fuel tank, which exposes the wing to debris from the top of the external fuel tank. If you try building a vertical shuttle in KSP, you will notice that the wings give you too much control authority during launch. Fins are best placed near the bottom of the rocket.
It's reusable, which means wear and tear can secretly accumulate without you noticing. This significantly increases the design requirements for the critical components, like the SRB that had a poor "tang" design, which, as it turns out, was definitively not fit for reuse.
NASA has done a lot of amazing things but I wouldn’t bet on them winning a Super Bowl.
They'd have a 50% chance of winning one on Mars, since it would just be NASA vs China
Every year NASA has a 50% chance of winning the Superbowl- even on Earth!
Either they win or don't. /s
It seems like this could be implemented by any harness.
> All of it using natural language.
Combining this with those approaches that recursively reason in latent space would be interesting.
Hmm... The key is to successfully decompose a big, hard problem into easier atomic sub-problems. However, the decomposition process itself is difficult, and this paper is not about that. They decompose a task using a human-written prompt.
> The approach relies on an extreme decomposition of a task into subtasks, each of which can be tackled by focused microagents. The high level of modularity resulting from the decomposition allows error correction to be applied at each step through an efficient multi-agent voting scheme.
Big if that the decomposition and the voting happen accurately for anything other than toy problems
The approach in the paper specifically addresses the case where an LLM can usually solve a task when it requires few steps, but fails for the same kind of task with more steps because it randomly gets a step in the middle wrong and then derails. It can't do anything for tasks that the LLM can't solve even when there's just a few steps.
In other words, it compensates for random error, not systematic error.
Worth opening the pdf just for the graph on page 1.
A striking example of how not to present data. If the Cognizant AI team is here: please can you fix it in the next version of the paper?
I think it's a brilliant example of how to use data to make a point.
https://xkcd.com/1162/
Except on figure 1 they're all at 0, making it look like the authors didn't know how to use the models or deliberately made them do nothing.
http://www.vibechart.net
I was just thinking "these guys will talk about this graph for the rest of their lives", it's the best graph you could ever hope to put into a paper. Loved it.
In case you want to know what’s going on in the left side of that chart, they gave a log scale in appendix a. I was thinking it was silly to not just use that version on the top, but I guess log scales make big differences ’feel’ smaller.
A log scale is actually appropriate in this context from a first-principles perspective. Per scaling laws (and also general behavior of epsilon-probability of failure multiplied N times), you would generally expect more vs. less effective techniques to have multiplicatively greater or fewer steps until failure, not additively greater/fewer. Figure 1 is comical, but the appendix figure is the more scientifically appropriate one.
At that rate, they might as well have gone one step further and made the x axis exponential scale to make it feel even bigger.
The dashed lines on top of the data points and labels is making me wince
Really seems like the reason logarithmic scales were invented..
I dunno, even though the authors address its use, making the task Tower of Hanoi doesn't meet the excitement of the title.
Especially since it's a recursive problem so each step is naturally broken up into subtasks. And the algorithm of what subtasks to break it up in to is public. This makes it much easier for it to get down to a case that the LLM can reliable solve.
I guess that the subtask decomposition of many (sub)problems is known and in the training distribution. How many real-world problems are resistant to divide-and-conquer? Presumably most/all of the unsolved mathematics conjectures. What else?
And yet the reverse paper was posted ad nauseam, covered by every news slop site, and overblown with really negative takes.
I have ADHD and the same approach works for me. (In fact, most days it is essential!)
do you have an algorithm for breaking down, organizing, and scheduling the small tasks, though? can it also be broken down?
This has seemed to me to be the natural next step to turn LLMs into more deterministic tools. Pushing the frontier is nice, but I think LLMs have a whole different gear when they are able to self-decompose in a reliable way. Most of my success creating reusable LLM products came from determining where requirements/outputs need to be "hard" vs. "soft".
Here is the pseudocode of MAKER:
The problem is how to even define a task using the English language and make sure there is enough entropy to infer the detailed intent. For it to be later split into zillions of small steps which can be executed over time by an LLM.
In English, that's hard, but there are programming languages ... specialized in breaking a complex task down for computers to understand.
...
one issue I see is when steps in a plan depend on one another, when you cannot know all the next steps exactly before seeing the results of the previous ones, when you may have to backtrack sometimes
This is actually good insight and worded in a simple way that clicked in my brain, thanks!
And you can decompose the proof of Fermat's last theorem into logical combinators.
The meat is in decomposing the difficult problem into steps
Some real life problems cannot be decomposed or cannot be decomposed with ease by an LLM.
Also, if we decompose a big task into many tasks, some might be solved in an incompatible way with the rest of the tasks and you can not combine them.
Right, it’s kind of like solving systems of linear equations. Some can be solved just by reordering, but most need you to handle all the constraints at once.
On the surface this is an interesting concept...
The paper however, meh...
No mention of MoE. One would think this is a logical evolution of that but not a mention (that I saw). Its own rubric for the task, Towers of Hanoi, was admittedly weak.
LLM papers are starting to look like the last decade of JS frameworks and Tools. Only with less code and more academics, and thats disappointing, because I think a lack of pragmatism and grounding is now holding the field back...