Agency
But are humans really so great at exhibiting agency? After all, Cluely managed to raise tens of millions of dollars with a product that promises to take decision-making out of our hands. AI can't function without instructions from humans, but an increasing number of humans seem incapable of functioning without AI. There are people who can't order at a restaurant without having an AI scan the menu and tell them what to eat; people who no longer know how to talk to their friends and family and get ChatGPT to do it instead. For Alexander, this is a kind of Sartrean mauvaise foi. "It's terrifying to ask someone out," he said. "What you want is to have the dating site that tells you that algorithmically you've been matched with this person, and then magically you have permission to talk to them. I think there's something similar going on here with AI. Many of these people are smart enough that they could answer their own questions, but they want someone else to do it, because then they don't have to have this terrifying encounter with their own humanity." His best-case scenario for AI is essentially the antithesis of Roy's: superintelligence that will actively refuse to give us everything we want, for the sake of preserving our humanity. "If we ever get AI that is strong enough to basically be God and solve all of our problems, it will need to use the same techniques that the actual God uses in terms of maintaining some distance. I do think it's possible that the AI will be like, Now I am God. I've concluded that the actual God made exactly the right decision on how much evil to permit in the universe. Therefore I refuse to change anything."
Doozey of a piece.