I build RAG systems for clients, and the failure modes are consistent enough to be predictable. Here are the ones that show up most often, with the fixes that actually hold up once real users and real data get involved.

1. Your chunks are the problem, not your model Most RAG failures trace back to retrieval, and most retrieval failures trace back to chunking. Split a document into fixed 500 character blocks and you slice sentences in half, separate a heading from the paragraph it introduces, and strip away the context a chunk needs to stand on its own. The embedding model never had a chance.

The fix is to chunk on structure, not on raw character count, and to add overlap so meaning survives the boundaries.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(

chunk_size=800,

chunk_overlap=150,

separators=["\n\n", "\n", ". ", " ", ""],

)

chunks = splitter.split_text(document)

The overlap matters more than people expect. A fact that lands right on a chunk boundary is effectively invisible to retrieval without it. For structured documents such as legal text, technical manuals, or anything with clear sections, split on headings first and then sub-split inside each section, so every chunk stays semantically whole. 2. Retrieval returns "relevant" chunks that never answer the question Vector similarity finds text that is topically close to the query. Topically close is not the same as actually answering the question. Ask "what is the maximum allowable duct temperature" and pure semantic search will happily return three paragraphs about ducts that never mention a temperature at all.

Two fixes stack well here.

Hybrid search combines keyword search (BM25) with vector search, so exact terms like part numbers, codes, and specific values are not lost to fuzzy semantic matching. This alone fixes a surprising share of "the answer was in the docs but it never showed up" complaints.

Reranking takes the top 20 or so retrieved chunks and reorders them with a cross encoder that reads the query and each chunk together, then passes only the best few to the model.

retrieve wide, rerank narrow

candidates = vector_store.similarity_search(query, k=20)

reranked = reranker.rank(query, candidates)

context = reranked[:4]

Retrieving wide and reranking narrow is one of the highest leverage changes you can make. It costs a little latency and buys a large jump in answer quality, because the model finally gets the chunk that contains the answer instead of four chunks that are merely on the topic. 3. The model hallucinates because you let it When retrieval is weak, a language model does what it was trained to do: it fills the gap with something plausible. If you never told it what to do when the context is insufficient, a fabricated answer is the default outcome, not an edge case.

Grounding is a prompt design problem as much as a retrieval problem. Three instructions do most of the work:

Tell the model to answer only from the provided context. Tell it to say clearly when the context does not contain the answer, rather than guessing. Ask it to cite which chunk each claim came from, which makes hallucinations visible instead of hidden.

system_prompt = """Answer the question using only the context below.

If the context does not contain the answer, say you don't have

enough information. Do not use outside knowledge. Cite the source

number for each claim."""

An honest "I don't have enough information to answer that" is a feature. In most business settings a wrong answer delivered confidently is far more expensive than a system that admits its limits and routes the question to a human. 4. No evaluation means no idea whether it works Plenty of RAG systems ship on vibes. Someone types five questions, the answers look good, and it goes to production. Then it fails on the sixth kind of question nobody tested, and there is no way to tell whether a change made things better or worse.

You need an evaluation set: a fixed list of real questions paired with the answers or the source chunks they should retrieve. With that in place you can measure the two things that actually matter.

Retrieval hit rate asks whether the correct chunk appeared in the top results at all. If it did not, no amount of prompt tuning will save the answer. Faithfulness asks whether the generated answer is actually supported by the retrieved context, or whether the model drifted off into invention.

Measure both before and after every change. Without an eval set you are not improving a system, you are guessing at one. 5. It worked at launch and quietly rotted A RAG system is not static. The knowledge base grows, documents get updated, policies change, and the vector index slowly falls out of sync with reality. Nobody notices, because the system keeps answering. It just starts answering from stale information.

Production RAG needs an ingestion pipeline, not a one time embedding run. When a source document changes, its chunks need to be re-embedded and the old vectors removed, otherwise you serve outdated answers with full confidence. Version your index, track when each document was last ingested, and set up a re-embedding job that runs on a schedule or on document change.

This is exactly the kind of unglamorous plumbing that separates a demo from a system a business can rely on. Most of my generative AI development work ends up being this layer rather than the model itself. The short version If a RAG pipeline is misbehaving, work down this list before touching the model:

Chunk on structure with overlap, not on fixed character counts. Add hybrid search so exact terms are not lost to semantic fuzz. Retrieve wide, then rerank down to the few best chunks. Instruct the model to answer only from context and to admit when it cannot. Build an eval set and measure retrieval hit rate and faithfulness on every change. Treat ingestion as an ongoing pipeline so the index never goes stale.

Almost none of this is about the language model. RAG lives or dies on retrieval quality and the boring pipeline around it, and that is good news, because every item on that list is something you control. FAQ Does a bigger context window remove the need for good retrieval? No. Stuffing more chunks into a large context window tends to lower answer quality, because the model has to find the signal among more noise. Precise retrieval of a few strong chunks beats dumping in fifty mediocre ones.

Should I fine tune a model instead of using RAG? For keeping answers current and grounded in a changing knowledge base, RAG is usually the better tool, since you update documents instead of retraining. Fine tuning is better for teaching style, format, or a narrow task, and the two are often combined rather than treated as either or.

What is the single highest impact fix? Reranking. Retrieve a wide set of candidates and rerank them down to the best few. It is a small code change that consistently produces the largest quality gain in the systems I work on.

Written by Shreyans Padmani, a freelance AI and ML developer with a 100% Upwork job success score and 12 published case studies covering production AI systems. More at shreyans.tech.