LLMs largely live in the world of pure language and related tokens - something humans invented late in their evolution. Human intelligence comes - at least partially - from more fundamental physical experience. Look at examples of intelligent animals that lack language.

Basically there's something missing with AI. Its conception of the physical world is limited by our ability to describe it - either linguistically or mathematically. I'm not sure what this means for AGI, but I suspect that LLM intelligence is fundamentally not the same as human or animal intelligence at the moment as a result.