How Our Spam Filter Decides Over 78% of Messages Before Calling an LLM

The simplest way to build an AI spam filter today is to send every message to a large language model and ask, “Is this spam?” It works. It is also the most expensive and slowest way to do it, and the cost never goes down: the ten-thousandth copy of the same spam costs as much to classify as the first.

SPAM Escudo, our privacy-first spam filter for website contact forms, takes a different approach. An LLM is part of the pipeline, but it is the last of five stages, and every decision it makes teaches the cheaper stages to handle more on their own. On top of that, a weekly AI-assisted review looks at everything that still reached the LLM and asks how it could have been caught earlier. Here is how it works, and what that loop has done to the numbers.

Five stages, cheapest first

The first four stages are fast and close to free. The LLM is the only stage with a meaningful cost per message.

The LLM is accessed through AWS Bedrock, where we chose the most cost-effective model that separates spam from genuine messages reliably. Because the easy decisions never reach it, it only has to handle the harder ones.

Where we started

Here is how 972 messages from our first round of live-feed testing were decided:

972 messages across two test accounts: 949 spam (97.6%), 23 genuine (2.4%). “User feedback” covers messages a user reclassified themselves.

Two numbers stand out. First, almost all contact-form traffic is spam: 97.6% in this test. Any approach that pays for an LLM call on every message is mostly paying to read spam. Second, even in this first round, the honeypot and domain rules alone decided over 40% of messages, at practically no cost. Overall, 56% of messages were decided before any LLM call. That was the starting point for the improvements described below.

Design decision 1: the cheap stages only ever say “spam”

The early stages are deliberately one-directional. A honeypot hit or a blacklisted domain is strong evidence of spam. But nothing cheap is strong enough evidence that a message is genuine, so the cheap stages never make that call. Anything that might be genuine always goes to the LLM for final verification.

That is exactly what the data shows: in the first round, 22 of the 23 genuine messages were confirmed at the LLM stage, and one by embeddings. The expensive model is reserved for the decision where a mistake hurts most, blocking a real customer’s enquiry, and since genuine messages are a small minority, this costs very little.

Design decision 2: every LLM verdict trains the cheap stages

Almost every spam message wants the reader to do something: visit a link, call a number or reply to an email address. That call to action is the spammer’s weak point, because they have to reuse it.

• When the LLM identifies a message as spam, its call-to-action values are blacklisted: URLs, phone numbers and email addresses.
• If the same sending domain keeps appearing in spam, the domain itself is blacklisted automatically.
• Blacklist entries expire after 14 days, unless another offending message arrives, which puts them back on the list. The rules stay current without growing forever or permanently punishing a domain that was briefly abused.

The next time that spammer tries, the rules catch the message without calling the LLM, and the rationale is recorded so anyone reviewing it can see exactly why:

This is what makes the cost curve bend. As the blacklists and the library of known spam grow, more volume is decided by rules and embeddings, and the share of messages that needs the LLM falls. The system gets cheaper to run the longer it runs.

Design decision 3: a weekly loop that targets what the LLM still sees

Automatic blacklisting learns from individual spam messages. It does not spot new patterns. So every week, the messages that fell through to the LLM are reviewed using an AI skill we have built and evolved for the job. It looks at what those messages have in common, proposes ways to decide them at an earlier, cheaper stage, and tests each idea against a fixed benchmark of real messages.

The log matters as much as the wins. Ideas that did not help are recorded too, so the same dead ends are not explored twice, and every production change can be traced back to the evidence that justified it.

One rule is not negotiable: a change is only promoted if it keeps the false-positive rate on genuine messages at zero. Catching more spam is worthless if it costs a real customer enquiry.

The results so far

These figures come from replaying the same 956 real messages (937 spam, 19 genuine) through each version of the pipeline, so the versions can be compared directly:

Rules and domain reputation alone took pre-LLM decisions from 54.9% to 78.1%, more than halving the number of LLM calls. Adding embedding similarity search, using pgvector, takes the full pre-LLM stack past 90% on the same messages, leaving fewer than one in ten for the LLM. That stack is now running in production, with results to match. Since the LLM is the only stage with a real cost per message, LLM spend falls roughly in line with the calls, by around 80%. A message decided before the LLM is also answered in milliseconds rather than seconds.

And throughout, the one number that must not move has not moved: none of the 19 genuine messages was wrongly flagged by any version.

Mistakes are cheap to fix

No classifier is perfect, so the recovery path matters as much as the accuracy. Messages flagged as spam are stored rather than deleted. If a user finds a genuine message among them, they mark it as genuine, and the sender and related details are whitelisted so the same mistake doesn’t happen again. Corrections work in the other direction too: in the first round of testing, users reclassified 13 messages, and in every case the final verdict was spam.

Privacy by design

Contact forms collect exactly the personal data that GDPR and CCPA protect: names, email addresses and message content. SPAM Escudo is built on that assumption. Submissions are encrypted at rest, and personal data is minimised and not kept longer than a spam decision needs. It integrates with a JavaScript snippet for static sites or a native WordPress plugin, so the same pipeline protects both. You can find out more and try it at spamescudo.com.

What’s next

• More of the same loop. Each week’s fall-through is the input for the next round of improvements, and the remaining LLM share keeps shrinking.
• Labelling and routing. Once a message is known to be genuine, the same pipeline can decide what it is about and route it to the right person to respond: sales enquiries to sales, support requests to support.

The general lesson

If your AI workflow sends every input to an LLM, ask two questions. Which of these decisions could something cheaper make with confidence? And what can each LLM decision teach the cheaper stages? The answers are usually “many” and “a lot”, and together they turn a fixed cost per decision into one that falls over time.

Then make the improvement a habit rather than a project. A regular, measured look at what still reaches the expensive model, with a strict gate on what gets promoted, is how 55% becomes 78%, and 78% becomes 90%.