Designing Scalable Retrieval-Augmented Generation Pipelines

Opening: Why does the retrieval step feel like a bottleneck?

Anyone who has tried to stitch a vector store to a language model knows the moment when the query latency jumps from a few hundred milliseconds to a few seconds. The question that keeps coming up in Slack channels and coffee breaks is: how can we keep the retrieval fast while the knowledge base grows to millions of passages? The answer is not a single setting but a collection of trade‑offs that become visible only after you have run the pipeline in production. In this note I walk through the concrete lessons I have gathered while iterating on a retrieval‑augmented generation (RAG) service that now handles several hundred requests per minute.

Background: Why scalability matters now

Large language models have become the default answer engine for many internal tools, chat‑bots and support assistants. Companies are feeding them with proprietary documentation, product catalogs, and legal contracts. As the corpus expands, the naive approach of re‑indexing the whole store every night becomes untenable. Users expect sub‑second answers, especially when the UI feels like a search box. At the same time, cost pressure pushes us to reuse cheap hardware wherever possible. The sweet spot sits at the intersection of three forces: vector‑store efficiency, query‑time filtering, and LLM prompting that respects token limits.

Technical deep dive: Building a pipeline that scales

Below is a minimal yet production‑ready example built with Python 3.11, LangChain 0.0.300, FAISS 1.7.4, and OpenAI’s gpt‑3.5‑turbo model (v1). The code demonstrates three ideas that keep the system from blowing up as the document count rises:

• Separate indexing and query phases – the index lives on disk and is memory‑mapped only when needed.
• Chunk documents into 500‑token pieces and store the chunk metadata for fast filtering.
• Use a two‑step prompt: first retrieve, then generate, so that the LLM never sees more than 2 k tokens.

# Install required packages
# pip install "langchain[all]==0.0.300" faiss-cpu openai

import os
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import FAISS
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.llms import OpenAI
from langchain.chains import RetrievalQA

# 1. Load raw docs (could be from a DB, S3, etc.)
raw_documents = [
    {"content": open(path, "r", encoding="utf-8").read(), "source": path}
    for path in os.listdir("./docs") if path.endswith(".txt")
]

# 2. Split into manageable chunks
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = []
for doc in raw_documents:
    for chunk in splitter.split_text(doc["content"]):
        chunks.append({"text": chunk, "source": doc["source"]})

# 3. Create embeddings (cached on disk)
embeddings = OpenAIEmbeddings(model="text-embedding-ada-002")
faiss_index = FAISS.from_texts([c["text"] for c in chunks], embeddings)
faiss_index.save_local("./faiss_index")

# 4. Load index for querying (memory‑mapped, no full load)
retriever = FAISS.load_local("./faiss_index", embeddings).as_retriever(search_kwargs={"k": 5})

# 5. Build the RAG chain – separate LLM and retriever
llm = OpenAI(model_name="gpt-3.5-turbo", temperature=0)
qa_chain = RetrievalQA.from_chain_type(llm=llm, retriever=retriever, return_source_documents=True)

# 6. Simple query function used by the API layer
def answer_question(query: str):
    result = qa_chain({"question": query})
    answer = result["result"]
    sources = [doc.metadata["source"] for doc in result["source_documents"]]
    return {"answer": answer, "sources": sources}

# Example call
if __name__ == "__main__":
    print(answer_question("How does our refund policy handle digital goods?"))

The code follows a clear separation of concerns. Indexing happens once, and the FAISS.load_local call loads only the index metadata into memory, leaving the heavy vectors on disk. The search_kwargs parameter lets you experiment with different k values without touching the LLM side. By returning the source documents you can add a simple citation UI, which also helps debugging when the answer drifts.

Common pitfalls: What tends to go wrong at scale

Even with a clean code base, several issues surface once traffic climbs:

• Vector drift. When you add new documents you must update the index incrementally. Re‑building the whole FAISS index daily is wasteful and can cause a temporary loss of queries if the index file is replaced mid‑flight. The fix is to use add_texts on the existing index and persist a new version atomically.
• Token overflow. The LLM has a hard limit on input tokens. If you retrieve too many chunks, the prompt will be truncated and the answer becomes meaningless. Guard this by counting tokens before concatenation and falling back to a smaller k if needed.
• Cold‑start latency. The first query after a deployment often triggers a disk read for the FAISS index, causing a spike. Warm‑up the service by issuing a dummy query during start‑up, or keep the index in a RAM‑disk if memory permits.
• Unbalanced chunk size. Overly large chunks reduce retrieval precision, while very small chunks inflate the number of vectors and increase search time. In practice a 500‑to‑1 000 token range works well for most English text.
• Missing metadata filters. When you have multi‑tenant data, forgetting to add a tenant filter in search_kwargs can leak information across customers. Always attach a metadata field and enforce it at query time.

Practical implementation guide: Step‑by‑step decisions

Below is a checklist you can follow when you start a new RAG service:

1. Define the document ingestion pipeline. Choose a source (Git, S3, DB), decide on a chunking strategy, and store both the chunk text and metadata (source, tenant, timestamp).
2. Select an embedding model. For most English corpora text-embedding-ada-002 gives a good balance of speed and quality. Pin the model version to avoid silent drift.
3. Pick a vector store. FAISS works well on a single node, while Milvus or Pinecone are better for multi‑node setups. The API is similar, so you can swap later.
4. Implement incremental indexing. Use the store’s add_texts method after each ingestion batch. Persist the new index file under a versioned path and update a symbolic link atomically.
5. Configure the retriever. Set k based on your latency budget (e.g., 5 for <200 ms, 10 for <400 ms). Add filter arguments for tenant or date constraints.
6. Build the LLM prompt template. Keep the system message short, include a placeholder for retrieved snippets, and limit the total token count to model_max_tokens - 500 to leave room for the answer.
7. Instrument monitoring. Track query latency, vector search time, and LLM response time separately. Alert when any component exceeds 80 % of its SLA.
8. Test failure modes. Simulate missing vectors, corrupted index files, and rate‑limited LLM responses. Your API should return a graceful fallback message rather than a stack trace.

Following this list lets you move from a prototype to a service that can survive traffic spikes and data growth without a major rewrite.

Closing thoughts: Keep the system honest

Scalability is not a one‑off configuration but a habit of checking assumptions as the data and traffic evolve. The most reliable pipelines are the ones where every piece – chunking, embedding, vector store, and prompting – can be swapped independently. When you notice latency creeping up, look first at the vector store’s disk I/O, then at the number of retrieved chunks, and finally at the LLM’s token usage. A disciplined approach to monitoring and incremental indexing will keep the retrieval step from becoming the weak link.

Take the next step

If you are ready to try a ready‑made starter kit or need help tailoring a RAG pipeline for your own data, visit akkistech.com and drop me a line. I’m happy to share more code samples and discuss architecture choices over a virtual coffee.