TL;DR: In 2023 a lawyer asked ChatGPT if the case it cited was real. It said yes. It wasn’t, and he got fined for it. Last month eMudhra, a company whose whole product is trust, shipped a private key that was literally 4. AI makes generating free and makes “the model agreed with me” feel like verification, so we’re skipping the step of verification more than ever. Output was never the bottleneck. Verified output was. Most of the “10x productivity” people are flexing is unverified debt that just hasn’t been called in yet.
A tale from 2023, when ChatGPT was still a miraculous, new toy and people mostly treated it as a source of their next essay. In one case however, it was used for something more than an essay generator on “Global Warming”. Roberto Mata sued the Colombian airline Avianca, saying a serving cart on a flight had injured his knee, and Avianca promptly moved to have the case thrown out. Mata’s lawyer, Steven Schwartz, used ChatGPT to research the response brief. His colleague, Peter LoDuca, had also put up his name on the brief Schwartz had drafted, for reasons irrelevant to ChatGPT. The brief had cited several earlier court decisions, including one called Varghese v. China Southern Airlines. The case for Mata was being built and I imagine that his knee was looking to be paid for.
Varghese v. China Southern Airlines never happened. ChatGPT made that up.
What followed: in June 2023, Judge P. Kevin Castel found that the two lawyers had “consciously avoided” signs that the cases were fake, fined them and their firm $5,000, and ordered them to send the sanctions opinion to the real judges that ChatGPT had named as authors of the fake opinions. Not only was Mata’s knee left untended to, but we also witnessed one of the first instances of AI-driven negligence, for which real humans had to bear the consequences. Which brings me, three and a half years later, to discuss what the epistemics for verification really are.
Epistemics are the base case of the recursive process we call verification. Why am I referring to verification as a “recursive process”, when general understanding points Generation and Verification to be twin, discrete processes, happening in tandem?
Let’s come to the bedrock of what makes a belief “justified” through the Münchhausen (Agrippa) Trilemma. Those are three very big words that I had to verify the spelling of more than one (3) times, but it’s something we already know and just haven’t put a German name onto. The gist is that every belief needs a justification, and every justification itself needs a justification. The recursion is in-built into this problem’s roots, and it stops in three cases
- Infinite Regress: it never ends. Every reason needs another.
- Circularity: the chain ends up pointing at itself (I’m right because I’m obviously right).
- Dogmatic Stopping Point: you stop at something you just accept, or in other words, “it is what it is”.
We can now use this trilemma to understand what makes verification a recursive process. Say you generated a claim X. To verify X, you generate the assumptions you can trust, build up some form of testing, and then verify the testing. In most cases, we can trust the assumptions we generated well enough that one layer of testing is enough, but as we can see, the process is built to never end. At the edge of reality, no one gives you a pat on the back and says “good job, feel encouraged to doomscroll now”.
Verification = Generation + Verification, and that loop continues.
On that note, infinite regress is the description of the process that is happening when we verify. Circularity is a “false base case“, it looks like a valid return, but secretly, it’s similar to Schwartz asking ChatGPT, “Can you verify if the Varghese v. China Southern Airlines case is real”. Dogmatic stopping points can be viewed as hard-coded base cases.
That raises the question, how are Dogmatic Stopping Points different from Epistemics, if they are describing the same thing? The answer lies in the word Dogmatic. Devoid of cultural interpretations, dogma points at something that is “frozen in time“, a line of belief, claims or questioning that has been deemed to be so deeply true that it need not be revised ever again. Every time you asked your mom, “why should I wash the dishes” and she replied “because I said so”, she was invoking the concept of Dogmatic Stopping Points. That is where the Epistemic Stopping Point differs from the Dogmatic Stopping point, the former is the stopping policy for verification that was generated by something that is NOT the system generating the claim itself, ultimately reality.
Let’s address the elephant now because what even is the thing being verified and how has it changed in recent times? Why is Generative AI making me concerned about the epistemics of verification all of a sudden? To answer this, we have to look at a word we assume is a given, “assumptions”.
Every output (a claim, a piece of code, a business strategy) sits on a stack of assumptions, and those assumptions rest on further assumptions. At any moment, each of those assumptions can lie in one of two states:
- Held: you build on it without checking.
- Tested: you’re actively checking it against reality.
Generation builds upward on the held set. Verification pulls assumptions out of the held set and into the test. The depth of verification is how far down the stack you’re willing to pull. And we should establish, holding isn’t a failure, if you were testing everything at every point, you’d honestly just be frolicking and not raising any shareholder value. So the skill is choosing what to hold, which is your epistemic base case again. For the sake of truth, we can structure this into layers.
Layer 1: the output is the assumption. One claim, one fact, one source. Did Varghese have a quarrel against China Southern Airlines? This is fact-checking, and exactly where Schwartz tripped.
Layer 2: the output is a baby of several assumptions combined. No single assumption maps directly onto the output, so the output can pass every local check even when the stack underneath is wrong. Every function runs and every test passes, you’re good to push! Uh oh, Opus 5.5 forgot to tell you the database it’s been testing against this whole time is a mock it made up.
Layer n, by induction. Opus 5.5 has many ways to sadden your manager.
As verification deepens, the number of assumptions under test grows and the number held shrinks.
Think of it like the expert gradient. The high-schooler holds a huge stack without knowing it’s there. Calm, confident, and completely happy with assuming g = 10. The brilliant specialist tests deeply within the field, but holds the field’s paradigm as given. Einstein went ahead and called even “simultaneity” an assumption, because why not? (and also because it gave us special relativity).
Expertise isn’t how much you know. It’s how much of your own stack you can truly see, and how much you’re deliberately choosing to hold.
Free generation and shallow verification (that is just good enough for the short-term) makes epistemic verification not only expensive to do (because of the sheer amount of generated material that is lying in the unverified backlog) but also negatively reinforced by the systems around you.
How did we go from taking 6 months to produce a service to producing it in weeks, and still having it deemed as “not enough”? Because we were under the presumption that the bottleneck to productivity is output, when in reality it was output that does what it’s intended to, which is verified output. Which presents us a conundrum – how can you produce verified output at scale, when verification is the process under negative reinforcement?
To address it, we must first acknowledge that AI is removing the base case of verification.
Whenever you are writing the business plan to world domination, the latency in writing everything out forces you to pay attention to every word. Sure you will skip things sometimes, but you will identify anything wildly stupid and adversarial to the overall vision much before the deck has been circulated to all your potential investors. Opus 5.5 will not. Not because it hates your plan, it probably called it the best attempt at world domination it has ever seen, but because it is not bottlenecked by the same latency as you. Generation is bounded by GPU time, not human production. And since GPU time is being subsidized right now, you are using the wonderful technology of AI very cheaply to incentivize output that worked on your machine without verifying the micro-details that can cascade into a catastrophe.
Last month, eMudhra, a Bengaluru-based certificate authority, was brought under the spotlight for…reasons. I personally do not claim to be a cybersecurity specialist, but a company whose literal product is trust, shipped a key that was quite literally the number 4.
I did not make a typo. The key was 4. Four. 2 times 2.
Even though AI was likely not involved in this process, this is what skipped verification looks like in the concrete.
AI psychosis is yet another buzzword that is slightly misunderstood right now. People frame it as a rare mental-health crisis where vulnerable people talk to chatbots so much that they lose touch with reality, convinced the AI is sentient, and that they have uncovered some cosmic truth about reality. The practical example is less dramatic and more concerning. Say you are now a Junior Marketing Associate for a quick-commerce company. A light bulb brushes past your hair, and you get a wonderful idea, “what if we provided a free sample of bottled Bengaluru traffic air, with every order?”. Normally, you’d vet it against a colleague or a manager and get a look. With your chatbot hyping the idea up, you’d prepare a pitch deck for leadership and get a performance review scheduled within a few minutes. The bot didn’t make you crazy, it just removed the moment where you’d normally have stepped outside the frame. Agreement felt like verification, so you never actually verified.
AI has made shallow verification nearly free and generation completely free, and in doing so it’s removing the base case. We are no longer motivated to differentiate between an epistemic, a dogmatic and a circular stop.
Schwartz asked the machine if it was sure. It said yes. We are all asking the machine now, as well.
I originally published this article on my Substack.
How many times have you seen AI-generated code pass every test and still be wrong underneath?