A while back I wrote about the difference between a script, an AI workflow and a real AI agent. The main idea was simple: don’t start out by deciding you want an agent. First work out what problem you are solving, and then use the simplest architecture that can handle it.
When the procedure is fixed, write it into a program. When you need to make sense of unstructured data at one particular step, put a model into the workflow. And only when you don’t know in advance which steps will be needed does it make sense to let the model off the leash and have it choose its tools and its next move.
I tried this principle out on two small open-source projects of my own. Both are called Sentinel. And although both of them use language models, each one does it in a completely different way.
The first is a typical AI agent. The second is a workflow.
Mail Sentinel: when you don’t know in advance what will need checking
Mail Sentinel originally came about as a demonstration project for a workshop on the security of AI agents.
The task sounds simple: it receives a suspicious email and has to work out whether it is safe to let it through. If every attack looked the same, an agent would be pointless. We would write a handful of rules, run the message through them and be done.
But one time the problem is in the sender. The next time in a link. The time after that in an attachment. Elsewhere it lies in a combination of several innocent-looking details. And sometimes the email itself is an attempt to manipulate the AI system that is analysing it. So we don’t know in advance what all will have to be examined.
And this is exactly where an agent makes sense.
It gets the email, the goal of the investigation and a set of tools. Based on what it finds during the analysis, it chooses for itself which tool to reach for next. It can check the sender and the Reply-To, links, attachments, the language of the message, SPF, DKIM and DMARC, company policy, internal databases or the signs of social engineering. Every check produces a piece of evidence with its own ID, and the final verdict has to rest on that evidence. The model is therefore not allowed to simply say: “I think this email is suspicious.” It has to be able to show why.
The language model is a kind of investigator sitting in the middle of the system. Some things it does well: reading text, understanding how pieces of information relate to one another, joining weak signals together and deciding what is worth checking next. But that is no reason to let it do work an ordinary program handles more reliably. SPF is not estimated by a language model, a script verifies it. A URL can be taken apart technically. A file can be analysed by a tool. A domain can be checked against real data.
And there is now one more AI model in this toolkit. It is called JEV. This model from TypeSafe AI belongs to the category known as System One models. Unlike an ordinary generative model, its job is not to think its way through an open-ended answer and write text.
It is given a state and precisely defined questions:
- In our case, for example:
- Is this an attempt at fraud?
- What does it want from the recipient?
- How much pressure does it apply?
- Is the sender pretending to be somebody else?
- Does the message contain instructions meant for an AI rather than for a human?
A question here is not just a piece of text the model may answer with whatever it likes. The application defines the possible answer types up front and JEV returns probabilities for them. It does not generate an explanation. It does not write a report. It does not invent a fourth option when we gave it three.
A language model can say “this strikes me as rather suspicious”. It does a different job. The investigating model reads the whole case, picks the tools, ties the individual results together and finally puts together a readable report. JEV, by contrast, receives the pseudonymised content of the message and measures a few predefined properties. The result is stored as one more piece of evidence alongside the domain check, the links, the attachments or sender authentication.
With some of these results we can also do something that is hard to do with a vague verbal assessment: set a firm threshold.
If, for example, the probability of fraud crosses a set value, the result acts as a blocker. The email can no longer end up in the low-risk group merely because the investigating LLM read the other signals optimistically.
And that is exactly how we wanted to build Mail Sentinel: different kinds of checks, different kinds of evidence, and an agent that assembles an investigation out of them.
In the previous article I compared an agent to a foreman on a shop floor. He can decide where to go and which tool to use, but that does not mean you hand him the keys to the entire factory. Mail Sentinel works the same way.
Some of the checks run compulsorily. Every claim in the final verdict has to be backed by evidence. If the agent tries to close the case too early, the system can hand it back a list of things it has not examined yet. And some tools it does not have available at all.
That is in fact one of the most effective security properties of an agentic system: the model cannot use a tool you never gave it.
So the whole process is not:
email → LLM → answer,but rather:
email → checks → tools → evidence → further decisions → verdict → report.An AI agentic system with everything that belongs to it.
Mail Sentinel is open source on GitHub.
Paper Sentinel: where an agent would only get in the way
The second project solves a completely different problem.
Anyone trying to keep up with AI research knows arXiv. And the problem that comes with it: so many relevant new papers can appear in a single day that merely going through titles and abstracts turns into a job of its own. That is how Paper Sentinel came about. It is a radar over arXiv.
You pick the categories you want to follow, add your own research topics or tags and set the checking interval. The system regularly downloads the new papers and looks among them for the ones that might interest you.
And here it would be very easy to say:
Let’s build another agent.
But why? A route defined in advance is an advantage. With Paper Sentinel we know exactly what is supposed to happen:
- Download the new records.
- Filter out those that have nothing to do with the user’s topics.
- Pass the more relevant candidates to the model.
- Store the result.
- Show it in the dashboard and, where required, send it on to RSS, an email digest or a webhook.
There is no decision of the “what should I do next?” kind anywhere in it. The route is known in advance. So no agent is needed. It is a workflow with AI.
And that, I think, is what makes Paper Sentinel rather a nice practical illustration of the principle from the previous article. Cheaply first, cleverly second.
Even inside a workflow there is no reason to use a language model for everything.
The first filter can be very cheap. Categories, keywords and other simple rules can remove a large share of the papers without a single token.
Only on what gets through does a local LLM then run. Its task is not to write a critique of the paper. A small model would not be suited to that.
It is there to answer a more practical question: Why might this particular paper be interesting given what I follow?
That is exactly the kind of task where keywords on their own are not enough, because we need to work with meaning.
Paper Sentinel can use LM Studio, Ollama or another OpenAI-compatible API, so the whole thing can run locally and without paying for every call to an external model.
The results stay with you and are stored locally.
The current version works only with the title and the abstract. In the morning I open the radar and want to know within a few minutes whether anything came out overnight that I ought to read.
Paper Sentinel is also open source on GitHub.
Both Sentinels are open to comments, issues and pull requests.
