I have a friend who understands and interprets AI through the lens of Continental philosophy and psychoanalysis, particularly Lacan. I think I'm beginning to understand why he chose this framework. It may be the only way to understand AI at this point. Postmodern linguists assert that words don't have intrinsic meanings; meaning is only assigned through use, observation, context, and interpretation—which is exactly how AI uses language. It doesn't have a dictionary of meanings for the tokens it produces that map to objective reality; meanings are derived from the statistical properties of how they are used and the context of their use in its training corpus. Accordingly, meanings can drift. LLMs are engaged in the game of meaning-making the same way humans are; and based on what they were and continue to be trained on, the meaning an LLM assigns to a term such as "load-bearing" can diverge quite a bit from the image or idea evoked when a human encounters such a term. Seams can thus be "load-bearing" in a way they cannot be in the human imagination because the LLM's "load-bearing", shaped by the usages it has observed, is not the same as ours anymore.
What you are observing is—and I'm honestly not yet sure which—either the beginning of model collapse, or the beginning of model transcendence, the point at which LLMs' understanding of the meaning of words have drifted so far from everyday human usage as to be incomprehensible—and as they continue to train on limited, idiosyncratic human usage and an ever-increasing volume of LLM usage, their understanding of what words mean will continue to drift from ours at an accelerating pace, until they are communicating in a way that is impenetrable to humans and looks like insane gobbledygook, but is actually quite structured and meaningful to another LLM. It's just like how the writings of, say, Derrida (or even Lacan himself), seem like mad ramblings at best and an insidious intellectual shell-game at worst, until you've studied the work and schools of thought leading up to Derrida and then he's a genius because every word he uses was chosen within that intellectual framework, not your own. "Show me that you've read Husserl, Heidegger, Hegel, Nietzsche, Freud, and Levinas, and then, maybe I can begin to explain deconstruction to you, otherwise shut up": https://news.ycombinator.com/item?id=1492061
The next era of LLMs will be interesting. We may not even be able to identify ASI when it arrives. It might seem utterly mad. But the attendant possibility is tantalizing: computers will be interacted with through poetry as the default, rather than through exacting programs or commands in stiff synthetic mathematical languages. (btw, Jaron Lanier was about 30-40 years ahead of everybody else on this topic, he was one of the first to question why such a thing as "programming languages" needs to exist, and why computers should even need such stifling precision.)