Designing a Scalable RAG Pipeline with Vector DBs and LLMs

Opening

Imagine a user typing a question into a chat widget and expecting an answer within a second, even though the answer has to be stitched together from a knowledge base that is terabytes in size. The latency feels like a deal‑breaker, but the underlying system also has to stay within a reasonable monthly bill. This tension between speed, consistency, and cost is the everyday reality for teams that want to serve LLM‑augmented answers at scale.

In this post I walk through the decisions I make when I need a Retrieval‑Augmented Generation (RAG) pipeline that can handle millions of queries per month without blowing up the budget. We will look at vector database selection, index partitioning strategies, and the way the retrieval step talks to the LLM inference layer. The goal is not a perfect recipe, but a map of trade‑offs that lets you pick the right point on the latency‑cost curve for your own constraints.

Why this matters now

LLMs have moved from research demos to production APIs that can be called from any service. At the same time, vector search engines have matured to the point where they can store billions of embeddings and serve them with sub‑second response times. When you combine the two, you get a system that can answer questions that are out‑of‑scope for a vanilla LLM, because the retrieval step brings in up‑to‑date facts.

Senior developers are asked to design these pipelines for internal knowledge bases, customer support bots, or even compliance‑driven document search. The design space is wide: you can choose a managed SaaS vector store, spin up your own cluster, decide whether to shard by document type or by time, and pick between synchronous or asynchronous LLM calls. Each choice ripples through latency, consistency guarantees, and the monthly invoice.

Technical deep dive

Below is a stripped‑down version of the data flow that I use in production. The steps are:

• Chunk raw documents and generate embeddings with an LLM encoder.
• Write embeddings to a vector store, optionally partitioned.
• On a query, embed the user prompt, retrieve top‑k vectors, and pass the retrieved texts to a generation model.
• Return the generated answer together with source citations.

Each step has its own latency profile and cost profile. The biggest impact comes from the vector store configuration. I compare three common setups:

1. Managed SaaS (e.g., Pinecone, Typesense Cloud) – you pay per request and per stored vector. No ops overhead, but you are locked into their indexing algorithm and pricing tiers.
2. Self‑hosted open‑source (e.g., Milvus, Qdrant) – you control hardware, you can tune index parameters, but you have to manage scaling, backups, and monitoring yourself.
3. Hybrid approach – a small managed tier for hot data, backed by a larger self‑hosted cluster for cold data. This adds complexity but can cut costs dramatically.

The code example shows how to ingest data into a self‑hosted Milvus instance and then query it. The same pattern works for other stores, you just swap the client library.

import os
import uuid
from pymilvus import Collection, CollectionSchema, FieldSchema, DataType, connections
from sentence_transformers import SentenceTransformer

# 1️⃣ Connect to Milvus (adjust host/port as needed)
connections.connect(host="localhost", port="19530")

# 2️⃣ Define a collection schema – one vector field and one string field
vector_field = FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=384)
text_field   = FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=65535)
id_field     = FieldSchema(name="id", dtype=DataType.VARCHAR, max_length=36, is_primary=True, auto_id=False)
schema = CollectionSchema(fields=[id_field, vector_field, text_field], description="RAG collection")

# 3️⃣ Create the collection (if it does not exist)
coll_name = "rag_docs"
if coll_name not in [c.name for c in Collection.list_collections()]:
    coll = Collection(name=coll_name, schema=schema)
    # Index for IVF_FLAT with 256 partitions – a good default for moderate latency
    coll.create_index(field_name="embedding", index_params={"index_type": "IVF_FLAT", "metric_type": "L2", "params": {"nlist": 256}})
else:
    coll = Collection(name=coll_name)

# 4️⃣ Encode a batch of texts
model = SentenceTransformer("all-MiniLM-L6-v2")
texts = ["Your first document text.", "Second document with more details."]
embeddings = model.encode(texts).tolist()
ids = [str(uuid.uuid4()) for _ in texts]

# 5️⃣ Insert into Milvus
mr = coll.insert([ids, embeddings, texts])
print(f"Inserted {mr.primary_keys}" )

# 6️⃣ Query – embed the user question and retrieve top‑3 matches
query = "How does the hybrid approach work?"
query_vec = model.encode([query]).tolist()[0]
search_params = {"metric_type": "L2", "params": {"nprobe": 10}}
results = coll.search(data=[query_vec],
                     anns_field="embedding",
                     param=search_params,
                     limit=3,
                     output_fields=["text", "id"])
for hits in results:
    for hit in hits:
        print(f"Score: {hit.distance:.4f}, ID: {hit.id}, Text: {hit.entity.get('text')}")

Notice a few things that affect latency and cost:

• Index type: IVF_FLAT gives fast build time and decent recall, while HNSW offers higher recall at the price of more memory.
• nlist / nprobe: Larger nlist creates more partitions, which can improve recall but increases query time. nprobe controls how many partitions are searched at query time; higher values mean slower but more accurate results.
• Batch size for ingestion: Writing in batches of 1 000–10 000 vectors reduces network overhead and improves throughput.

On the LLM side, the inference service can be a hosted API (e.g., OpenAI) or a self‑hosted model behind a GPU server. When you call the model synchronously after retrieval, the total response time is roughly t_retrieval + t_generation. If you need sub‑second answers, you either push the generation model to the edge (costly) or keep the retrieval set tiny (which may hurt answer quality).

Common pitfalls

Many teams run into the same roadblocks when they try to scale a RAG pipeline.

• Over‑indexing: Adding too many partitions or using a high‑dimensional index can balloon memory usage without delivering noticeable latency gains. Monitor RAM and CPU usage after each index tweak.
• Stale embeddings: When source documents change, the old vectors stay in the store unless you have a clean‑up job. Inconsistent results are confusing for users and can cause downstream errors.
• Cold‑start latency: If your vector store spins down after inactivity (common in managed SaaS free tiers), the first query after a pause can take seconds. Warm‑up scripts or a small always‑on replica solve this.
• Cost surprise from query volume: Retrieval requests are cheap per call, but when you multiply by millions of queries the price adds up. Keep an eye on the query * top_k metric; reducing top_k from 10 to 5 can cut costs by half with a modest impact on answer quality.
• Network bottlenecks: If the vector store and LLM inference service live in different regions, the round‑trip adds latency that you cannot hide with better indexes.

Practical implementation guide

Below is a step‑by‑step checklist you can follow when you start a new RAG project.

1. Define service level goals: Decide on a target 95th‑percentile latency (e.g., 800 ms) and a maximum monthly cost. This anchors all later decisions.
2. Pick a vector store:
   • If you need zero ops and can afford per‑query pricing, start with a managed SaaS that offers automatic scaling.
   • If you have predictable traffic and want to control hardware, spin up a self‑hosted Milvus or Qdrant cluster on Kubernetes.
   • Consider a hybrid: keep the most recent 10 % of vectors in the managed tier for hot queries, push the rest to the self‑hosted tier.
3. Choose an index configuration:
   • Start with IVF_FLAT, nlist=256, nprobe=10. Measure recall on a validation set.
   • If recall is below 85 %, switch to HNSW with M=32, efConstruction=200. Expect higher memory usage.
   • Record the memory footprint and adjust nlist or M until you stay within node limits.
4. Set up ingestion pipeline:
   • Chunk documents to 200‑300 word pieces.
   • Generate embeddings with a lightweight encoder (e.g., MiniLM) to keep CPU cost low.
   • Batch writes in groups of 5 000 vectors.
   • Tag each vector with a version identifier for easy re‑indexing.
5. Integrate retrieval with generation:
   • Expose a single HTTP endpoint that accepts a user prompt.
   • Inside the handler: embed the prompt, query the vector store, collect top‑k texts, and build a prompt for the LLM.
   • Use streaming responses from the LLM if your client can render partial answers.
6. Instrument and monitor:
   • Log t_retrieval, t_generation, and total latency per request.
   • Set alerts for 99th‑percentile latency spikes and for vector store CPU > 80 %.
   • Track queries * top_k to forecast cost.
7. Iterate:
   • If latency exceeds the goal, try reducing top_k, lowering nprobe, or moving the LLM closer to the vector store.
   • If answer quality suffers, increase top_k or switch to a higher‑recall index.
   • If cost is too high, evaluate a smaller embedding model or a cheaper managed tier.

Following this checklist gives you a repeatable loop: measure → adjust → re‑measure. The loop keeps the system balanced between speed, freshness, and spend.

Closing thoughts

Scalable RAG is less about a single magic component and more about the glue that holds retrieval and generation together. The vector database you choose determines how much you have to tune indexes, how you handle partitioning, and where the biggest cost levers sit. The LLM service you call decides whether you can afford to run inference at the edge or need to batch requests.

Remember that latency, consistency, and cost are three sides of the same triangle. Pulling one corner down inevitably lifts the others. The sweet spot for your product will be unique, but the decision framework outlined here should help you find it faster.

Next steps

Ready to try it out? Grab the starter repo at akkistech.com and spin a Milvus instance with one click. The repo contains the ingestion script, the query API, and a simple Docker‑compose file for a local LLM server. Give it a run, measure your numbers, and iterate from there.