AI Incompetence as Dark Pattern
This is just a short note to elevate two sources that provide useful learnings that we can glean from the Claude Code … source code.
- This stuff is wildly incompetent
- The incompetence is part of the dopamine addiction machinery
Let me elaborate.
LLMs are wildly incompetent
The main reason I’m writing this note, is to document the absolutely brilliant takedown of the Claude Code source code by Jonny at neuromatch.social.
In case you’re not in the loop: “Claude Code” is a piece of software, built by Anthropic, that software developers can use to do “agentic software development”. That is: it’s a toolbox for the Claude AI, that makes Claude more powerful when working on software.
Jonny highlights, how truly awful this code is from a software engineering perspective. Like, how a lot of the prompting is to “not do bad things”. How calls to tools are not invoked directly, but via prompting the LLM.
this shit sucks so bad they can’t even CALL THEIR OWN CODE FROM INSIDE THEIR OWN CODE.
How things you should actually use an LLM for, bypass the underlying LLM.
They REALLY ARE using REGEX to detect if a prompt is negative emotion. dogs you are LITERALLY RIDING ON A LANGUAGE MODEL what are you even DOING
Since the quality of Claude Code is what you get, when you develop with Claude Code, the shambolic state and dismal quality of this source code is a damning verdict on not only Claude Code, but on the whole idea of “vibe coding”.
reminder that anthropic ran (and is still running) an ENTIRE AD CAMPAIGN around “Claude code is written with claude code” and after the source was leaked that has got to be the funniest self-own in the history of advertising because OH BOY IT SHOWS.
The incompetence is part of the dopamine addiction machinery
It’s bad enough, that the produced software code is superficially brilliant, but architecturally made of quicksand.
one thing that is clear from reading a lot of LLM code - and this is obvious from the nature of the models and their application - is that it is big on the form of what it loves to call “architecture” even if in toto it makes no fucking sense.
What is worse, is that this architectural deficiency powers a dark pattern, in which each new iteration trashing the previous iteration feels as progress, even when as a developer you’re running in circles and accumulating technical debt.
but what the LLMs do is in one narrow slice of time implement the “is member of set {pdf}” version robustly one time, and then they implement the regex pattern version flexibly another time, and then they implement the any str.endswith() version modularly another time, and so on. Of course usually in-place, and different file naming patterns are part of the architecture when it’s feeling a little too spicy to stay in place.
This is an important feature of the gambling addiction formulation of these tools: only the margin matters, the last generation. it carefully regulates what it shows you to create a space of potential reward and closes the gap. It’s episodic TV, gameshows for code: someone wins every week, but we get cycles in cycles of seeming progression that always leave one stone conspicuously unturned. The intermediate comments from the LLM where it discovers prior structure and boldly decides to forge ahead brand new are also part of the reward cycle: we are going up, forever. cleaning up after ourselves is down there.
Tech debt is when you have banked a lot of story hours and are finally due for a big cathartic shift and set the LLM loose for “the big cleanup.” this is also very similar to the tools that scam mobile games use (for those who don’t know me, i spent roughly six months with daily scheduled (carefully titrated lmao) time playing the worst scam mobile chum games i could find to try and experience what the grip of that addition is like without uh losing a bunch of money).
Accumulate tech debt by new features, release that with “cleanup,” transition to “security audit.” Each is actually the same, but the present themselves as the continuation of and solution to the others.
A series of flashing lights and pictures, set membership, regex, green checks, the feeling of going very fast but never making it anywhere. except in code you do make it somewhere, it’s just that the horizon falls away behind you and the places you were before disappear. and sooner or later only anthropic can really afford to keep the agents running 24/7 tending to the slop heap - the house always wins.
David Gerard points out, that there’s an underlying theory for this.
There’s a book on this — Hooked: How to Build Habit-Forming Products by Nir Eyal, from 2014. This is the how-to on getting people addicted to your mobile app.
Hooked builds on research papers about the mechanisms of gambling addiction — a life-destroying problem — to explain how you can use the same psychological manipulation techniques in your app. This book is on the bookshelf at every startup.
Eyal said he wrote Hooked to promote healthy habits, not addiction — but from the outside, you’ll be hard pressed to tell the difference. Because the model is, literally, how to design a poker machine. Keep the lab rats pulling the lever.
You’ll see the Hooked model in every mobile game. You’ll see it in dating apps, where the business model is to string you along and not connect with someone nice, but keep you spending money on the dating app.
This is what we call dark pattern design. Instead of using psychology insights to develop better user experiences, neuropsychological mechanisms are abused to degrade user’s agency.
What this means, it that as an AI user you feel productive. But you’re actually running in circles. And here’s the thing: you’re burning tokens while running in circles, feeding the Silicon Valley venture capital machine.