The safety filter wasn't protecting my data.
It was deleting it.

The AI memory system I use stores what we discuss as records: who did what, about which topic, then links them into a knowledge graph. It normally works quietly in the background, until one day I asked it about something I knew we had just covered, and it said there was nothing on file.

At first I assumed I had misremembered. But after it happened several times I opened the logs. What I saw stopped me. Over four days, a line reading "dropped" had appeared 732 times. Those records were never written to the graph, and the part that scared me was this: not a single error had been raised.

When I looked at what the missing records had in common, the answer was obvious. They were the ones with personal or slightly sensitive content, exactly the kind of thing a memory system has to be able to store. It is not forbidden material. Yet it was precisely the group that had vanished.

Here is the arc of the story: first why the data disappeared without a sound, then the fix that worked without rebuilding the whole system, and finally the lesson you can apply to your own work, even if you never touch Gemini.

Part 1The filter that deletes data quietly

The root cause is in the extraction step

A memory system like this does not just store raw text. Every time something new comes in, it hands the text to a language model first and asks it to pull out who is involved, what happened, and how it all connects. This is the extraction step, and its result has to be structured output the system can turn into a graph.

The model I first assigned to this job was a small Gemini (flash-lite): fast and cheap. The problem is that before Gemini returns anything, its answer passes through its own safety filter. If the content lands in a category it guards, it blocks that response, as Google's own docs describe: content is categorized and blocked by a probability threshold.

So when a response gets blocked, what my system receives is emptiness, not an error saying "this was filtered." An empty result makes the extraction step fail for that whole record, and the record then drops with no warning at all.

Why "gone quietly" is worse than "failed loudly"

If a system breaks and throws a bright red error, that is annoying but fine, because you know right away that something is wrong and can go fix it. Silent data loss is the opposite. Everything looks healthy. The numbers on screen do not report a gap. The graph keeps growing.

You only find out when you reach for something and it is not there, which might be weeks later, and you will never fully know what all was lost, because it was never recorded. This is the point that matters most: a memory system you can't trust is worse than no memory system, because it lulls you into relying on something incomplete.

A filter that blocks and tells you is a safety feature. A filter that blocks and stays quiet is a data-loss bug wearing a safety costume.

Setting safety to its loosest still isn't enough

The first thing everyone reaches for is to set safety settings as permissive as possible (BLOCK_NONE). That does help for some categories, but it is not a reliable answer for data that must not be lost, for two reasons.

1. Some categories cannot be turned all the way down, and the filtering thresholds are something the provider can change whenever they like. Set it well today, and a change on their side tomorrow starts dropping data again without you touching anything.
2. Even if you cover every category, you are still staking the integrity of your entire archive on someone else's filter, one you do not control. For data that genuinely has to be complete, that is a risk not worth carrying.

Seen this way, the problem changes shape. It goes from "how do I make Gemini stop filtering" to "why is our extraction step tied to a filter we don't control in the first place."

Part 2The fix: swap the engine, don't rebuild the car

Separate the job from the provider

The key to the fix is to treat the extraction step as a socket, not something welded to Gemini. The system around it does not need to know who is working inside. It just sends text in and receives structured output back. Whoever sits in that socket can be swapped. This is the principle of a fixed interface with a swappable engine inside.

Designed this way, moving off Gemini is not a rebuild. We just point the extraction step at a new model through OpenRouter (a broker that lets you call many providers' models behind one interface). The model we chose is qwen3-235b, an open model with no filtering layer that cuts our ordinary content away.

An open model isn't plug-and-play

Here is the part I want to tell straight, because the swap did not finish in one line. Once we changed the engine, a new kind of dropped record started appearing, not because the model was bad, but because our old system had been written to fit Gemini's output shape a little too tightly.

The small Gemini returned fairly simple output. The larger open model returns more structured detail, which is a good thing, but our old pipeline had not allowed for that variety. So we adjusted three points to let it accept a wider range of output.

• When writing to the graph, nested values (not simple scalars) had to be converted into a form the database accepts first, otherwise the whole record was rejected.
• The output-validation step used to throw away a whole record over one missing field. We changed it to keep what it did get, instead of discarding the entire thing over a single miss.
• The data schema used to require certain fields always be present. When the open model occasionally left one out, whole records dropped again. We fixed it by letting those fields be optional, while keeping the correctness rules for values that are supplied.

With all three in place, the drop rate that had swung between roughly 15 and 25 percent fell to zero across six runs against the live writer. Not reduced. Gone.

What about cost

People assume a bigger model must cost more, but for our workload it turned out cheaper. Measured on real usage it runs about $0.0024 per record, roughly $2 per month at ~750 records a month, and because it runs on OpenRouter's cloud it adds no load to our own machine at all.

One more thing we did at the same time: give this memory system its own API key, with its own spending cap, separate from everything else. If one day something goes wrong, costs spike, or a key leaks, the blast radius stays inside one box instead of spreading across the whole house.

Part 3Applying it to your own work

The lesson that stuck the most

This is not really about Gemini alone. Every commercial model has its own filtering layer, and those layers are designed for "safety" in the provider's sense, not for the completeness of your data. Those two things are not the same, and sometimes they are in direct conflict.

If you take one thing from this piece, let it be this: a filter that deletes your data without telling you is not a safety feature, it is data loss. And for work where data must not be lost, failing loudly where you notice beats disappearing quietly while you assume everything is fine.

Check whether your system is losing data quietly

1. Count the drops, not just the successes. For every model call, log both the count going in and the count coming out. If 100 go in and 92 come out, there must be somewhere that shows where the 8 went, rather than letting it slide.
2. Feed in things you know should pass. Periodically inject samples you know are valid. If they disappear, a filter is eating good data (this technique is called a positive control).
3. Tell "failed" apart from "filtered." An empty result from a block often looks like ordinary failure, which makes it hard to separate. Watch for the signals specific to filtering too.

When to swap the engine

Not every workload needs to leave commercial models. If yours is general content that rarely brushes sensitive categories, tuning safety settings may be enough. But if any of these are true, take separating the extraction step into a genuinely swappable engine seriously.

• Your data must not be lost, especially work where being auditable is critical.
• The content you process routinely touches sensitive categories, such as health, finance, or personal data.
• You want to control your own cost and data privacy, rather than leaving it to someone else's thresholds.

Where to start

You do not need to change models on day one. Start by making yourself able to see, because the hardest silent-data-loss problem to fix is the one you don't yet know exists.

1. Take one step that calls a model and add logging for the count in and the count out on each run.
2. Run real data through it for a while, then check whether the two numbers match. When they diverge, note which kind of content it lines up with.
3. If you find it losing data systematically for a certain group of content, then decide whether to tune settings or swap the engine.

One round of putting the in and out numbers side by side is enough to show you how easily this stuff slips away in silence, and once you have seen it, you will stop trusting a pretty number on a screen for quite a while.