Imagine assigning every living human being a virtual AI powered con artist to tail them around and try to convince them of whatever is ordered by the highest bidder. This AI is powered by mass surveillance and big data and is able to know the "mark" intimately and work on them 24/7 both directly and via their social network. Now throw in deep fakes, attention maximizing "compulsion loops" and other adversarial models of human cognition, etc.
It's basically the hydrogen bomb of propaganda, a doomsday machine for the mind. It would be like creating Satan, except this one would be entirely amoral and mercenary.
This doomsday machine only dooms as long as there's just one user. Burger King, Subway, McDonald's, Wendy's, and Taco Bell are all going to try to hack your mind to buy lunch from their chains. But people can't eat 5 lunches for lunch. In practice this doomsday persuasion is going to bog down in a red queen's race between propagandizers/advertisers with mutually exclusive goals, much like the current media environment. Improved persuasion technology means that organizations that want to persuade people end up spending money on zero-sum games against rival organizations but public behavior in aggregate doesn't change dramatically.
Corporatism isn't the only one out there who would find such technologies useful. Besides, in this Judeo-Christian religious context there are many warnings from the Bible around such consumerism.
Doesn't the same zero-sum dynamic apply to political persuasion? There are many competing groups in the world that already spend money on political persuasion. You said "convince them of whatever is ordered by the highest bidder" so I'm not considering the case where one secretive group with a pointed agenda (the CIA? the FSB?) has a monopoly on super-persuasion.
It's basically the hydrogen bomb of propaganda, a doomsday machine for the mind. It would be like creating Satan, except this one would be entirely amoral and mercenary.
This doomsday machine only dooms as long as there's just one user. Burger King, Subway, McDonald's, Wendy's, and Taco Bell are all going to try to hack your mind to buy lunch from their chains. But people can't eat 5 lunches for lunch. In practice this doomsday persuasion is going to bog down in a red queen's race between propagandizers/advertisers with mutually exclusive goals, much like the current media environment. Improved persuasion technology means that organizations that want to persuade people end up spending money on zero-sum games against rival organizations but public behavior in aggregate doesn't change dramatically.