I have the same feeling. I'm not worried about superintelligent AI because we are only training them on human level intelligence. By what mechanism does our current AI technology take the leap to technologies that humans have never conceived of?
Our current AI is more like a fancy Google search than some kind of machine God.
"Human level intelligence" is not some sort of hard ceiling. We already can create AIs that are vastly superhuman in narrow domains. It would be the height of hubris to claim that a broadly superhuman AI is impossible - that would require human brain to be the pinnacle of general intelligence.
How do we get to ASI? That's what recursive self-improvement is about.
If AGI is reachable, then we can make AI that, in turn, makes improved successor AIs. The performance goes up. It's not bounded by human intelligence - it's bounded by how much the previous generation of AI could improve upon itself.
We don't have a stable recipe for RSI yet, but AI development is already AI-assisted. It's just that the "improvement" loops of today are long, and require plenty of human input. Betting against RSI is betting that it'll stay that way forever - that tightening the loop and removing humans from it is fundamentally impossible.
How is this better than training the next Google by having the bots do Google searches?
I'm not saying that it is impossible to surpass human intelligence, all I'm saying is the AI has the same set of working data that humanity does. Unless Plato was right all along it's going to be hard for the AI to discover too much more from that data than humanity has already discovered. Sure there are some less well explored niches that the AI can help fill in, but the part where it makes the next step above humanity seems unlikely given the constraints.
Do we expect the AIs to develop entirely new branches of mathematics? To discover new physical phenomena? Come up with an entirely new way of thinking? That seems to be what these AI companies are promising and I'm skeptical.
AlphaGo beat humans at go partly by studying human games but then they made MuZero that can learn games in general just through self play and became better than humans at chess, go, shogi and many others. That kind of approach my be doable for general intelligence rather than just board games.
Centuries of physical experimentation, observation, and testing of hypothesis. Developing new branches of mathematics to deal with anomalies in observed test data. Developing entire branches of language to help organize and transmit concepts to other humans.
Our current AI is more like a fancy Google search than some kind of machine God.