An organisation switches on an assistant that can search the intranet. The intent is modest: let people ask about leave policy without digging through folders. It works straight away; questions that used to land with HR are now answered in thirty seconds, with a link to the policy. Two weeks later somebody asks about pay bands and receives a passage from a document that was opened up to the whole company by mistake three years earlier.
The assistant did exactly what it was meant to do: search everything the person asking could reach. In doing so, it surfaced something that had been too widely shared for three years. That says a lot about how RAG works, and about what to sort out before switching it on.
How does RAG work?
RAG stands for retrieval-augmented generation. In plain terms: search first, then answer.
Four steps:
- Indexing. Your documents are split into chunks and converted into a form that supports searching by meaning rather than exact words. This happens once per document, and the result is usually stored in a vector database.
- Retrieval. When someone asks a question, the question is converted the same way and the closest-matching chunks are pulled back.
- Assembly. Those chunks are attached to the question, along with an instruction to answer from them.
- Generation. The model produces the answer, usually with a citation back to the source document.
What does not happen is training. The model receives the text with each query and forgets it afterwards, exactly like any other prompt.
What does it give you?
It solves two problems at once, which is why nearly every vendor now offers it.
Stale knowledge. A model knows nothing after its training cutoff and nothing about your organisation. With RAG it does not need to: the knowledge comes from a source you keep current.
Fabrication. Because real text accompanies the question, there is something to point at. That substantially reduces the problem described in why AI makes things up, without eliminating it: the model can still misread the retrieved passage or assert a connection the text does not support.
And one practical advantage that gets overlooked: you can remove a document from the index and it stops appearing in answers. With a model fine-tuned on your data, you cannot.
Where does the risk move to?
Without RAG the question is: what does an employee type in? That is a behavioural question, with a moment and a person attached.
With RAG the question is: what can the assistant reach? That is an authorisation question, and it was already answered years ago by whoever set permissions on those folders and sites.
Those two questions have different owners and different fixes. The second is also the more uncomfortable one, because the honest answer is usually that nobody is entirely sure.
Over-broad permissions become findable. A site readable by "everyone in the organisation" was protected in practice by nobody knowing it existed. Searching took effort. An assistant searches everywhere at once and summarises. The same thing happens with Copilot and SharePoint. The assistant does not break anything. It surfaces what was already broken.
The index ages differently. A deleted document may persist in the index until the next crawl. An edited document can keep surfacing in its old form. That is a new category for your retention policy, and it rarely gets one.
Retrieved text goes to the vendor. This is often skipped in assessments. The passages pulled from your own documents are attached to the prompt and therefore sent to the model. Everything that applies to ordinary prompts (storage, retention, the possibility of human review) applies to those passages too.
What do you sort out before switching it on?
- Which sources are in the index? Not the ones you intended, but the ones actually there.
- Whose permissions does the search run under? Usually the user's. Sometimes a service account, in which case every user gets the same answer regardless of what they were entitled to see.
- Are those permissions still correct? This is the real work and it has nothing to do with AI. It is also why Microsoft puts remediating oversharing first in its own deployment guidance.
- How does a document leave the index? On request, automatically or only at the next full crawl?
- What does the vendor do with retrieved text? The same question as for any prompt.
RAG moves the conversation from the employee to the configuration, and that is good news: configuration is something you can get right once. Start with a small, well-defined source, such as the staff handbook or one department's procedures. Check the permissions on exactly that source, switch the assistant on and only widen it once that works well. Keep in mind that people also paste their own text into the same chat window, from this morning's inbox for instance. With that text, they still need to take care with sensitive data.
