AI Knowledge Bases

A language model on its own does not know your company. It knows nothing about your products, processes or contracts and, when in doubt, makes up an answer. Retrieval-augmented generation, or RAG, gets around that. The system first searches the documents for the relevant passages and only then lets the model answer, citing its sources.

This pays off when knowledge is spread across many systems and nobody knows where anything is. Typical cases are support and sales teams answering the same questions from manuals again and again, or new hires who spend a long time looking for rules and procedures. We build and roll out such knowledge bases for internal policies, technical documentation, customer service or contracts.

Preparation matters more than the model

How good the answers get depends less on the language model than on what the search finds. That is why the work starts with the sources: file shares with policies and contracts, wikis with process descriptions, ticketing systems with solved cases, and business systems as far as their data exists as text.

Documents are split into sections that make sense on their own. Pieces that are too small lose their context, pieces that are too large make the search imprecise, and headings help with the split. Metadata such as document type, department, validity, version or language lets the search narrow down and exclude outdated versions. Headers, duplicates and tables of contents are removed. Tables and scanned pages get separate treatment, because they are harder for the search to capture than running text.

Search: combining meaning and keywords

Embeddings turn pieces of text into number sequences that represent their meaning. This lets the search also find passages that make the same point in different words. For product numbers, technical terms, names or dates, classic keyword search is more precise.

Most solutions therefore combine both into a hybrid search, whose results are merged into one ranked list. A reranking step then re-sorts the best hits with a more precise method before the language model gets to see them. Which combination fits, we decide on test questions from your company.

Permissions and freshness

The assistant may only show what the person asking is allowed to see in the source system. There are two ways to do that. Either the permissions are carried into the index along with the content and checked on every search, or the system fetches the data live from the source system at query time on behalf of the person.

Both depend on clean permissions. An assistant makes visible what hardly anyone found before, and folders and pages shared too generously should be cleaned up before launch. As for freshness, answers are at best as current as the index. New, changed and deleted documents must reach it along with their permissions, continuously or at fixed intervals depending on the source.

How we check answers, and where they still go wrong

Every statement should point to a document and section that can be opened directly. If the system finds nothing suitable, it should say so instead of guessing. Whether that works shows in a question catalog of real questions from the business teams, each with a known source and expected answer. We check separately whether the search finds the right passages and whether the answer is based on them. Common evaluation methods distinguish, for example, retrieval precision, groundedness in the sources and relevance of the answer. A new model, a different prompt or changed preparation can make results worse, so the tests run again every time. In operation, thumbs up or down and unanswered questions show where content is missing or outdated.

Some limits remain. The knowledge base only knows what is in the sources, and old versions lead to wrong answers if nobody marks or removes them. When two documents state different things, the system can show that but not decide. Metadata such as validity and version helps but does not replace people responsible for the content. Tasks like “compare all contracts from this year” are only partly solved by a simple search. And even with a source cited, an answer can be wrong; for important decisions a look at the document is still necessary.

Off-the-shelf or custom

Not every knowledge base has to be built from scratch. If the content mostly sits in Microsoft 365, Microsoft Copilot is the obvious route. Connectors can bring in further sources such as wikis or ticketing systems, and their permissions are respected. For many different source systems there are also products with prebuilt connectors and permission checks.

A custom RAG application makes sense for unusual document types, fixed workflows, strict hosting requirements, or when the answers are to be embedded in a business application or customer portal. The effort is higher, but search, prompts and interface can be shaped freely. Ready-made products are quicker to deploy, custom applications more flexible, and a combination often fits. We compare the options by sources, number of users and data protection requirements. What to consider when using language models in general is covered in the topic on generative AI.