I Know It When I See It
· AI & Machine Learning

I Know It When I See It

Autonomous agents don't develop taste. They execute the taste someone was disciplined enough to write down. A worked example from a real pipeline.

By Geordie Everitt

Curated from CIqSi (ciqsi.net)
To illustrate the word *destroy*, I wanted a child knocking down a tower of blocks. Every four-year-old on Earth has done this; it is arguably the purest expression of the concept available to a human being. The image model refused. A small figure, an act of demolition, no matter how gleeful — it read the scene as a child committing violence and declined to render it. So the "destroy" card in a language-learning episode ended up depicting a sandcastle succumbing to a wave, which is fine, and also a lie about what destruction feels like when you are four. This is where we actually are with the machines. Not marvelling that the dog can talk. Expecting the dog to exercise judgement — to decide, unsupervised, whether a child knocking over blocks is pedagogy or assault, and to be *right*. That is the whole promise of the word "agentic": a system that understands why its content works well enough to make the call itself. We have moved the goalposts from competence to taste. And taste is a much stranger thing to put in a machine than competence ever was. ## The one agentic corner of an unglamorous pipeline [LinguaMama](https://linguamama.ai), my language-learning project, runs a content pipeline that is mostly not agentic at all. It is a traditional workflow — script, translate, caption, illustrate, stitch — that happens to use generative models as the labor at each station. There is exactly one station where something resembling judgement has to happen, and it exists for a boring, budgetary reason. The images come from a cheap diffusion model, Cloudflare's `flux-2-klein-9b`, running about a tenth of the cost of the model I would use if money were no object. At that price you accept a certain failure rate. Roughly one image in ten comes out wrong in some structurally interesting way: a car drawn from outside when the scene called for the passenger seat, a hand with six fingers, a face half-melted. Someone — or something — has to catch these before they ship to someone learning the word for "car." So after each image is generated, a second model looks at it and decides whether it passed. A vision model inspects the picture and returns a verdict: clean, or a list of specific defects, each with a severity. If the verdict is bad enough, a third step rewrites the prompt to target the diagnosed flaw, re-renders, and inspects again — a bounded loop, three attempts, fail loud on any error. That loop is the agentic corner. It looks, it judges, it acts, it looks again. Except the judgement isn't where you'd think. ## The taste is in the list The vision model is not sitting there being tasteful. It is handed a catalogue — ten named failure modes, the clown car and the extra fingers and the melted face and the rest — and told to check the image against each and nothing else. I didn't type that catalogue; I taught it, feeding a stronger model a pile of the cheap one's failures and having it distil the wreckage into rules. The taste was mine — I chose which images counted as broken — and the wording is a stronger machine's account of that choice. The system prompt gives the whole game away in a single line. It instructs the inspector: **do not comment on artistic taste, mood, composition, or colour beyond what the catalogue describes.** The machine is forbidden its own taste. It may only check the image against a list of things a human singled out, in advance, as unacceptable, and report the matches. What looks like machine judgement is human judgement, externalized into a taxonomy and handed to a tireless clerk. That catalogue is the judgement; the model merely executes it. And this is the shape of almost every "agentic" system I have built: the autonomy is real, the loop runs without me, but the taste it enforces is mine — written down, made explicit enough that a machine can apply it a thousand times without getting bored or getting ideas. ## Potter Stewart never wrote his list In 1964, faced with defining hard-core pornography, Justice Potter Stewart declined. He offered instead the most quoted evasion in American law: "I know it when I see it." It is, when you sit with it, an infuriating thing for a judge to say — a man whose entire job is to convert judgement into written, appealable, portable standards, announcing that on this particular question he would supply a verdict and withhold the reasoning. Trust the vibe. It knows. The [LinguaMama](https://linguamama.ai) QA loop is the exact opposite gesture, and that is the whole point of it. It refuses to let the judgement stay a vibe. It forces "this image is wrong" to become "this image matches failure mode `object-orientation`, severity egregious, note: the car is perpendicular to the road." A finding you can read, argue with, port to another episode, and file a bug against. The discipline of building the thing *is* the discipline of forcing the judgement into writing — of dragging every "I know it when I see it" into a sentence a stranger could apply and get the same answer. The Hays Code was a written list. The FCC's seven dirty words were a written list. Every content standard that has ever survived contact with an appeal has been a written list, precisely because "I know it when I see it" cannot be appealed. You cannot argue with a feeling. This anxiety about who judges, and by what standard, is old — the novelty is only that we are now handing the clerk's job to a model. ## The bill comes due at the guardrail Which brings me back to the child and the blocks. The diffusion model's refusal to render a demolished tower is judgement too — but it is Potter Stewart's kind. Somewhere in that model's training is an unwritten conviction that a small figure plus an act of destruction equals violence, and it applies that conviction with total confidence and no catalogue you can inspect. It is the same reflex that bleeps the word "kill" out of a documentary, shadow-bans a video because bronze nude statues stood in the background of a park in Spain, and insists that an episode about *body parts* feature everyone fully clothed. The machine knows obscenity when it sees it. It will not tell you what it saw. You cannot debug a vibe. That is the entire difference between the two systems, and it has nothing to do with carbon versus silicon. My taxonomy is contestable because it is written; if I decide a toppling tower is pedagogy, I edit line one and the clerk complies. The platform's guardrail is not contestable, because it was never written down — it is a conviction wearing the costume of a rule. When it is wrong, and it is often wrong, there is no line to edit and no bug to file. There is only the shadow-ban, and the silence, and the fully-clothed anatomy lesson. So as the models gain judgement — and they are gaining it — the question worth asking is not whether they have taste. Of course they do; a tenth of my images prove it, badly. The question is whether the taste they act on is legible: written, attributed, editable, wrong in ways you can name and fix. Steering AI, in the end, is the unglamorous discipline of turning "I know it when I see it" into a list somebody else can read. The systems that do it will be arguable. The ones that don't will just know things about you, and never say what. --- Published under the name **Geordie**.