Last month I asked an AI to summarize a technical framework I knew well. The summary was fluent, confident, and completely wrong about a specific causal claim. It sounded right. The language was precise. The structure was exactly what I'd expect from an expert summary. I almost forwarded it to a colleague before I caught it.
This is the vibe check problem. And it's quietly breaking how we make decisions with AI-generated information.
Plausible is not the same as true
Large language models are trained to predict the next plausible token. They get rewarded for outputs that sound like they could have been written by someone who knows what they're talking about. That's not a bug. That's the objective. But our brains treat fluency as evidence of competence, and competence as evidence of correctness.
There's a reason for this. In the ancestral environment, if someone spoke fluently and used the right jargon, they probably knew what they were talking about. Speaking confidently without uncertainty was expensive. Liars and fools tended to hedge. Experts didn't need to. That heuristic worked reasonably well for most of human history.
AI broke that heuristic in about eighteen months.
We trained a system to maximize the appearance of expertise and then released it broadly. And then we acted surprised when people treated its outputs as expert. The problem isn't that people are gullible. The problem is that the AI is giving them exactly the signals they've been trained to trust.
Why experts are not immune
The instinct is to think this is a layperson problem. Someone without domain expertise gets fooled by confident AI output because they can't tell the difference. But I've watched technically sophisticated people do the same thing, including in my own field.
It happens because of how reasoning actually works. System 2 the slow, deliberate, analytical process does the checking. But System 1 the fast, intuitive one has already signed off on the output before System 2 gets activated. And System 1 uses exactly the wrong signals to evaluate AI content. It sees confidence. Fluency. Proper structure. It relaxes.
One of the more uncomfortable patterns I've noticed: the less someone knows about a topic, the more likely they are to accept AI output on it uncritically. Not because they're stupid. Because they lack the internal calibration to notice when the vibe is off. They don't have a mental model of what wrong looks like, so they default to trusting the confident presentation.
The compounding problem
Here's where it gets serious. AI-generated content is increasingly being used as input to other AI systems. Someone generates a summary. That summary gets pasted into a slide. That slide gets used to train another model. The original error doesn't just persist, it propagates and compounds.
I've seen this in the wild. A vendor published a whitepaper with AI-generated research citations that didn't exist. Other vendors cited that whitepaper in their own materials. Someone built a product decision on those citations. The error went through at least three layers of review before anyone with domain knowledge noticed. And that's the version that got caught.
What actually helps
Most AI literacy guidance focuses on prompting skills. Ask better questions. Request citations. Verify the output. All of that helps at the margins. But the vibe check problem isn't a prompting problem. It's a cognitive problem.
Some things that actually work:
- Force explanation before evaluation. Before you judge an AI output, you have to explain it back in your own words. Not paraphrase. Actual explanation. If you can't articulate why something is true, you don't actually know if it is.
- Spot-check one claim, not all of them. You can't verify everything. Pick the most specific, most falsifiable claim in the output and verify that independently. The most specific claim is usually where hallucinations hide.
- Separate drafting from judging. Write first, evaluate later. Don't let the fluent draft create false legitimacy in your mind before you've actually assessed the content.
- Calibrate your confidence before looking at the output. Ask yourself how sure you are and why, before you let the AI's confidence level anchor yours.
The incentive problem nobody talks about
The real issue is that we reward confident output and punish uncertain output, even when uncertainty is more honest. The person who says "I'm not sure, let me check" gets passed over for the person who says "Here's the answer." AI has learned to give us what we reward.
And we keep building systems that make it easier to generate confident output than to verify it. The path of least resistance is always to accept the vibe and move on. Changing that requires making the harder path the default, not the exception.
The fix isn't better AI. It's better instincts for when to trust it, and an environment that rewards skepticism over fluency. That second part is the hard one, and it's entirely on us.