You ask a question inside a company archive of thousands of support tickets: "Which ports need to be open for the RSync transfer to work?" The AI in front of you does not know. It cannot know — that fact appears in no training data. It sits in the company's own records.
There is only one thing it can do: find the right page first, then read it and answer. And that is where the whole thing is decided. The quality of the answer depends far less on how clever the model is than on which pages it was handed.
Search first, answer second
The method is called retrieval-augmented generation. The model does not answer the question directly. The system searches the archive, picks a few passages, and hands those to the model along with the question. The model builds its answer on top of them.
"A few passages" is a harder limit than it sounds. In practice between three and seven go to the model. If the right page is not on that list, no amount of intelligence rescues the answer. The quality of the search sets the ceiling for everything that follows.
Two kinds of question, two kinds of search
People ask in two different ways. Some questions carry something exact: a ticket number, an error message, a project name, a file name. Others describe: "how do we do this?"
Two different families of search answer these two kinds.
The first is word search. It checks whether the words in your question appear on the page. It weighs three things: how often the word appears on that page, how rare it is across the whole archive, and how long the page is. The method is called BM25 and it dates from the nineties.
The second is meaning search. It turns every text into a list of coordinates. Nearby coordinates mean nearby meanings. The question and the page can find each other without sharing a single word.
Why a rare word is worth more
The word "system" appears on nearly every page; it tells you nothing about which page you want. But if "ML5664" appears in exactly one record, then the page containing it is almost certainly the page you are looking for. Word search measures that difference directly: the rarer a word, the stronger the evidence when it matches.
Word search has one obvious weakness. You write "fault" and the record says "anomaly". You mean the same thing, but not a single word lines up, so the page never surfaces. Meaning search closes that gap.
The old method beat the new one
The expectation was straightforward: meaning search, built on neural networks, would comfortably beat a thirty-year-old statistical method. A measurement over a hundred real questions showed the opposite.
Word search brought back 74% of the pages it was looking for within its first seven results. Meaning search managed only 65%.
The reason lies in the nature of the archive. These records are full of codes: ticket numbers, production lot identifiers, error strings, file names. Meaning search blurs exactly those, because in a coordinate space the nearest neighbour of one number is another number.
Meaning search weakens precisely where the word itself is what matters. A ticket number has no "meaning". It only has itself.
Finding and ordering are different jobs
Since each is good at a different kind of question, combining them is the obvious move. But they cannot be combined directly: the score a word search produces and the closeness a meaning search produces are not on the same scale. Adding them is meaningless.
The fix is to ignore the scores entirely and take only the rank order from each list. A page that lands high on both lists rises. A page that lands very high on just one list is not thrown away either; it survives.
Combining raised coverage to 81%. But it left one thing untouched: the order. The right page enters the list, yet sits fifth or sixth. Since only the first few passages reach the model, being on the list is not enough. It has to be near the top.
A second reading
So one more step is added: a model that reads the candidate pages from scratch and reorders them.
The difference is subtle and decisive. The first search compares summaries of the question and the page that were made separately. The second model reads the question and the page side by side and answers a single question: does this page actually answer this one?
The cost is plain: every question-page pair needs its own reading. It cannot be run across a whole archive. So it only works on the sixty candidates the first search brought back.
The result shows the two steps doing different work. Combining gets the right page onto the list; reordering carries it to the top.
| Method | Pages found | Time |
|---|---|---|
| Meaning search alone | 65% | 0.43 s |
| Word search alone | 74% | 0.72 s |
| Both combined | 81% | 1.24 s |
| Combined + reordering | 95% | 1.43 s |
In ninety-six of a hundred questions, the page being sought appeared within the first seven results. Usually near the front of them.
Why does precision fall?
The same measurement has another face, and at first glance it looks bad. When the system returns three pages, 36% of them are right. When it returns seven, only 18% are.
That is not a failure; it is a flaw in the measure. The reference list usually holds one or two correct pages per question. If you return seven and only two can possibly be right, the ceiling is already 29%. The number measures the reference list as much as it measures the system.
It would have been easy to leave that out. Knowing what a measure actually says is worth more than making it say more than it does.
Relevant does not mean you are allowed to read it
The most instructive finding of the work appears in no metric at all.
A page can match a question perfectly and still be a page that person should never see. The search layer knows nothing about permissions; it knows only about similarity. Unless results are filtered by the projects someone can actually reach, an innocent question can come back with a paragraph from a project they were never given access to.
An AI assistant either inherits an organisation's permission structure or breaches it. There is no third option.
This is the difference between treating the problem as a scientific one and treating it as a working system. The metrics never catch it. The first real user does.
How far does this hold?
What was measured is which pages were found. Whether the answer the model finally wrote was correct, clear or useful was not measured. That is a separate question.
The numbers belong to this archive too. Another organisation's records could easily reverse the gap between word search and meaning search. But two general lessons look portable: words and meaning complement each other rather than replace each other, and finding something is a different job from putting it in the right place.
What is this piece based on?
This piece draws on Nadjib Atamnia's master's thesis, written at Université d'Évry Paris-Saclay (2025–2026). The work was carried out on an assistant called RAGmine, developed in the IT department of Murata Integrated Passive Solutions; the measurements run over a hundred questions written by the team itself.
- Robertson, Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond, 2009.
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS, 2020.
- Cormack, Clarke, Büttcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods, SIGIR, 2009.
- Nogueira, Cho, Passage Re-ranking with BERT, 2019.
- Thakur et al., BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models, NeurIPS, 2021.
Nadjib Atamnia
Works on autonomous systems; co-author on the continual learning and multi-agent simulation projects.
See all their articles
