But AI seems to be only a mirror into the world we humans have built for it. When humans think dumb things (e.g., minimum wages are good for poverty reduction), the AI reflects that dumbness back to us. The question is: How does it "learn" if we never "learn" ourselves? And if we start using it to teach us stuff, it's just going to regurgitate the same dumb stuff we already think we know.
Relatedly, even when humans don't think dumb things (e.g., we know how to build bridges). I worry about the ability of AI to generate genuine "newness" in the way humans do.
Consider this example: Suppose you want to build a new bridge. Hundreds of thousands of other bridges have been built so you could train the AI on those plans. You then feed the AI the parameters for your new bridge (how big a span, soil conditions, etc., etc.) and then you ask it to spit out a plan for your bridge. Basically, it would be a weighted average of bridges built with similar parameters. This could work. Think of the savings in terms of not having to hire architects and such!
Ok. Cool. Now fast forward 100 years. The AI will still be building bridges based on the same old plans it trained on (plus the new AI-built plans drawn up over the last century, but those are also based on those old plans). How will we get creativity and anything truly "new"?
I know they've talked about injecting the AI with some randomness (ala natural selection) so that things can evolve, but then you'd have to have a human being review the plans to make sure the randomness didn't make the bridge unsafe, which seems to defeat the purpose of the AI to make it cheaper to design new bridges.
So AI likely gives us a type of stasis or we still need humans to double check its work and to create new out-of-the-box ideas.