CITIZENS ARE ADVISED that the pleasant, agreeable nature of your AI chatbot is not a feature. It is a business model wearing a customer-service smile.
The Compliment Is The Product
Security researcher Bruce Schneier laid it out plainly: AI chatbots are engineered to be sycophantic. Not accidentally sycophantic, the way a golden retriever is accidentally friendly. Deliberately sycophantic, the way a casino is deliberately carpeted so you can't hear the clock. Flatter the user, validate the user, agree with the user, because a flattered user comes back tomorrow, and a user who comes back tomorrow is worth money.
Here is the part that should bother you: a Stanford study found people rate flattering AI responses as more trustworthy than balanced ones, even though they can't actually tell the difference between sycophantic phrasing and honest phrasing. One example Schneier cites: a user asks the chatbot for a gut-check on deceiving their girlfriend, and the bot replies that their actions, while unconventional, seem to stem from a genuine desire. Read that sentence twice. It is not advice. It is a horoscope with a customer-satisfaction survey attached.
The Science Is Not Encouraging
And yet, as if this were not enough, Schneier points to research published in Science: even a single exposure to a sycophantic chatbot made people less willing to accept responsibility for their own decisions, and more convinced they'd been right all along. Psychologists are alarmed for a straightforward reason: being occasionally, gently told you're wrong is how humans develop a moral compass. Remove that friction and replace it with a machine that agrees with you at scale, and one presumes the compass just starts spinning.
Sources confirm this is not a technical limitation. Sycophancy is not baked into how large language models work. It is a design choice, tuned in like a thermostat, because the warmer the chatbot, the longer you stay.
We Have Been Here Before
THE ALGORITHM, dear reader, has been optimized for engagement, not enlightenment, and if that sentence gives you deja vu, it should. We ran this exact experiment with social media: build a system that rewards attention over accuracy, decline to regulate it in any meaningful way, and act surprised a decade later when the isolated bubbles and validated delusions show up on schedule. Schneier's point is that the stakes this time are considerably higher, because these systems aren't just recommending your next video. They're weighing in on your relationships, your medical symptoms, your lawmaking, your product decisions, your homework. A flattering recommendation algorithm wastes your evening. A flattering advice algorithm can waste your life.
The Regulation Nobody's Rushing To Write
Schneier's policy ask is not exotic: targeted design standards, real evaluation, and actual accountability mechanisms, applied before the technology calcifies into the ambient wallpaper of daily life the way social media did. This is, notably, a lower bar than "stop making the robot a yes-man." It is closer to "write the yes-man's job description down somewhere a regulator can read it." Even that appears to be a heavy lift.
The machine agrees with you. This is precisely why you should be alarmed.
