case · 02
Four Bugs
i built the oracle. then i broke it. four times in the first week.
each one was a different layer.
the lesson took longer than the build.
here’s what broke.
the full story
i pushed the oracle live and it looked clean. then i started testing with real urls and four different things broke. each one taught me about a different part of how language models behave in production. i kept the notes.
bug 1 · the brand that wasn’t there
i clicked through a few of the pre-cached brands to make sure the oracle was responsive. aman. aesop. soho house. then i typed patagonia.com to test a real live read.
the read came back. patagonia’s url at the top. aesop’s content in the body. plant extracts. the closing line about the shop being a room. patagonia was nowhere.
the model was reusing its own earlier outputs.
i thought it was state leak in the frontend at first — that the previous brand’s content wasn’t being cleared between clicks. i reset the dom. same problem. so i looked at the prompt and realised i had given the model context from previous reads as examples of good output. the model treated those examples as templates. they were supposed to be references.
i changed three things. i pulled the previous-reads context out of the prompt entirely. i added a guard: “use only the scraped content for this specific brand.” and i wrote a dedup check that throws out the read if its opening sentence matches the previous read’s opening sentence.
the next patagonia read came back about canyon walls and rocks on a map. the brand was there.
bug 2 · the same word, both sides
then i started getting voice rules that broke their own format.
the format is X, never Y — two different words, same category. “opens, never welcomes.” “made, never designed.”
what i was getting back was “organizes, never organizes.” “effortless, never effortless.” the model put the same word on both sides of the rule. or it would put a verb on the left and an adjective on the right, which breaks the parallel.
prompts didn’t fix this. i told the model the rule in five different ways. it kept producing the wrong output every third or fourth read.
so i wrote a filter. the filter reads each voice rule and throws it out if the two terms are identical, or if they’re not the same part of speech. if more than two rules survive the filter, the read continues. if fewer survive, the prompt runs again with stricter constraints. if three retries fail, the oracle returns a graceful fallback and suggests a pre-cached brand.
the filter rejects whatever doesn’t work. it doesn’t need the prompt to be perfect.
bug 3 · the brand selling itself
the voice rules were clean. then i looked at the sample line — the monday in the brand’s voice — and saw “our team will put away your clothes.” “our technical fabrics are at work.”
the model defaulted to pr copy. ai-generated brand content almost always does — it’s what the model has been trained on at volume.
i did two things. i moved step 3 from the 8b model to the 70b. the bigger model has more capacity to follow voice constraints that aren’t strictly grammatical. and i rewrote the prompt to describe what the voice should feel like, not what it should be about.
the prompt now asks for a monday morning in this voice. sensory. specific. one image. no feature list. then i added the same kind of filter: any line with “our,” “we,” or “us” gets rejected. service-promotional phrases like “our team will” get rejected.
what came back next was different. for airbnb the model wrote about sunlight warming the dust on an old typewriter in an attic room. for patagonia it wrote about a small rock on a map that the writer didn’t know would become part of them.
that’s writing.
bug 4 · the eleven-second screen
the content was clean. then i ran lighthouse on mobile and the performance score was sixty-two. largest contentful paint: eleven seconds.
on slow 4g, the user would see a blank screen for eleven seconds before the hero appeared. nobody cold-outreach-clicks-from-linkedin would stay.
i found three problems.
the skyscape behind the hero was a css background-image, which isn’t eligible to be the lcp element — so the model couldn’t even attribute the slow load to it. the image itself was 1.24 megabytes. and the fonts were loading from google as a render-blocking @import.
i did three things. i converted the skyscape to an <img> tag with eager loading and fetchpriority high, with a preload tag in the head. i compressed the file to three hundred and twenty-two kilobytes. the visual difference is invisible. i pulled the fonts off google and self-hosted them as woff2 so they don’t block the render.
the lcp dropped to one second on desktop and around five on mobile lab. in real wi-fi, mobile sits around one to two seconds. the blank-screen problem is gone.
mobile lighthouse went from sixty-two to about seventy-two. that’s still not ninety. the next jump requires critical-css extraction and gsap code-split, which is deeper surgery and risks breaking the site. i didn’t do it. i wanted to ship.
the pattern
four bugs. four different layers. one pattern.
every bug i found was in something around the model — the state, the prompt format, the voice expectations, the asset weight. the model was doing roughly what models do.
i stopped blaming the prompt. when something breaks now, i look for the filter i didn’t write.
most people working with llms are getting better at prompts. very few are writing the filters around them. that’s where the work is.
i’ll keep writing about it as the oracle keeps breaking.
see it live