I Used Less AI and the Feature Got Better

Here's a pitch that sells itself: an AI feature, but with less AI in it. Say that in a meeting and watch people's faces.

Okay, let me back up.

I built a message router at work. It looks at an inbound message and decides what kind of response it needs. The obvious move, the one every tutorial nudges you toward, is to send it to the model, let the model figure out the category, done. One call, minimal code, feels very "AI-native."

I didn't do that. Not because I'm smarter than the tutorial. Because I've been burned by that pattern before, and I didn't feel like getting burned again.

Most messages aren't hard

They're the same handful of things, phrased slightly differently, that you've already seen a thousand times. Calling a language model to work that out is like hiring a consultant to tell you the sky is blue. Technically it'll get the right answer. It'll also take longer, cost more, and occasionally, inexplicably, say the sky is "kind of blue, contextually."

And it's not one call. It's every message, forever. Paying full price for a decision a lookup could have made.

So the system checks cheap things first

Cheapest to most expensive, bailing out the moment something is confident enough to answer.

First, a gibberish check. Before you ask what a message means, you have to ask whether it means anything at all. Typos, keyboard mashing, a stray emoji sent by accident. That's its own small rabbit hole and honestly deserves its own post, so I'll leave it there for now. Just know that a chunk of traffic gets filtered out before anyone, human or model, has to think about it.

Then keyword matching. The boring, deterministic kind. Exact or near-exact phrasing you've catalogued a hundred times over. Fast, free, no ambiguity. Deeply unfashionable, does a lot of the work.

Then vector similarity. Embed the message, compare it against a bank of things you've already handled, look at the nearest neighbours. If the top match clears the threshold, and clears it by a wide enough margin over whatever came second, you treat it as basically the same question you've answered before. No reasoning required, just "we've been here."

The margin turned out to matter more than the raw score. A message that sits roughly equally close to two different intents isn't a match. It's a coin flip wearing a number.

Only what survives all three goes to the model. Genuinely unclear, genuinely new.

The LLM isn't the front door here. It's the last person you ask, after you've already tried the easy ways.

Disagreements are free tuning data

This is the part I didn't plan and now like the most.

Sometimes I run the cheap layers and the model on the same message. Not always, just a sample. Most of the time they agree, which is boring, and reassuring, and tells me nothing.

The interesting cases are the ones where they don't.

If keyword matching says one thing and the model says another, one of them is wrong, and either answer is worth having. Model's right? Then I've got a rule that's too greedy, or a phrase I catalogued badly. Cheap layer's right? Then the model got clever about something that didn't need cleverness.

Either way I now have a labelled example sitting exactly on the boundary I care about, and I didn't have to write it, imagine it, or pay anyone to annotate it. The system is just telling me where it's confused, unprompted, for free.

So the bank grows. The rules get tighter. More traffic resolves before it ever reaches a model. The thing gets cheaper and faster while also getting better, which is not usually how this goes.

That's it

None of this is clever. "Check cheap things first" isn't an insight, it's most of computer science.

But there's a real pull right now toward making the model the answer to everything, strong enough that "just don't call it" reads like a design philosophy instead of a Tuesday.

Anyway. Most messages never touch a model now, and the ones that do are the ones actually worth paying for.