This is the fourth post in our OpenAI API fundamentals series, building on context and cost management.
Ask a model a question about your company's internal policy document, your product's actual API reference, or anything else it never saw during training, and it will still often produce a confident, plausible-sounding, and completely fabricated answer — because generating plausible text is exactly what it's designed to do, whether or not it actually has the relevant facts. Retrieval-Augmented Generation fixes this by finding the actually-relevant text first, and explicitly handing it to the model as context before asking it to answer. Because RAG is genuinely the pattern most real production LLM applications are built around, it's worth covering the full pipeline in depth, including the parts that only matter once a system moves past a toy example.
The core idea in one sentence
Instead of asking the model "what's our refund policy?" and hoping it somehow knows, RAG retrieves the actual relevant passage from your real documentation first, and asks the model to answer using that specific passage — turning an open-book question the model might guess at into a closed-book one it answers directly from provided material.
Step 1: embeddings, and what they actually represent
response = client.embeddings.create(
model="text-embedding-3-small",
input="Our refund policy allows returns within 30 days of purchase.",
)
vector = response.data[0].embedding
print(len(vector))1536
An embedding is a list of numbers (a vector) representing a piece of text's meaning in a way that supports mathematical comparison. Two passages discussing similar ideas produce vectors that are numerically close together; two passages about unrelated topics produce vectors that are numerically far apart — even if they don't share a single word in common. This numerical closeness is the entire mechanism RAG's retrieval step relies on.
Step 2: indexing your actual documents
documents = [
"Our refund policy allows returns within 30 days of purchase.",
"Corporate training sessions can be rescheduled up to 48 hours in advance.",
"Cloud architecture courses require basic networking knowledge as a prerequisite.",
]
doc_embeddings = [
client.embeddings.create(model="text-embedding-3-small", input=doc).data[0].embedding
for doc in documents
]For a handful of documents, comparing embeddings directly in Python is genuinely fine. For anything beyond that — thousands or millions of document chunks — a vector database (Pinecone, Weaviate, or pgvector as a Postgres extension are common choices) stores embeddings and performs this similarity search efficiently at real scale, which plain in-memory comparison stops handling well fairly quickly.
# pgvector, as a concrete example — Postgres with a vector similarity index
import psycopg2
conn = psycopg2.connect("dbname=novuspark")
cur = conn.cursor()
cur.execute(
"INSERT INTO document_chunks (content, embedding) VALUES (%s, %s)",
(doc, embedding),
)
cur.execute(
"""
SELECT content FROM document_chunks
ORDER BY embedding <=> %s
LIMIT 3
""",
(query_embedding,),
)<=> here is pgvector's cosine-distance operator, backed by an actual index (commonly HNSW) that makes this similarity search fast even against millions of rows — the same reasoning as any other database index, applied to vector similarity instead of exact-match lookups.
Step 3: retrieval — finding what's actually relevant
import numpy as np
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
def retrieve(query, documents, doc_embeddings, top_k=2):
query_embedding = client.embeddings.create(
model="text-embedding-3-small", input=query
).data[0].embedding
scores = [cosine_similarity(query_embedding, doc_emb) for doc_emb in doc_embeddings]
ranked = sorted(zip(scores, documents), reverse=True)
return [doc for _, doc in ranked[:top_k]]
relevant_docs = retrieve("Can I get my money back?", documents, doc_embeddings)
print(relevant_docs)["Our refund policy allows returns within 30 days of purchase.", ...]
Notice the query — "Can I get my money back?" — shares almost no words with the retrieved document, "Our refund policy allows returns within 30 days of purchase." This is the actual payoff of embeddings over naive keyword search: the retrieval is matching on meaning, not shared vocabulary, which is what makes RAG work for real user questions phrased however a real person happens to phrase them.
Step 4: generation — answering, grounded in what was retrieved
def answer_with_rag(query, documents, doc_embeddings):
relevant = retrieve(query, documents, doc_embeddings)
context = "\n".join(relevant)
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": f"Answer the user's question using only the following context. "
f"If the answer isn't in the context, say you don't know.\n\n{context}",
},
{"role": "user", "content": query},
],
)
return response.choices[0].message.content
print(answer_with_rag("Can I get my money back?", documents, doc_embeddings))Yes — our refund policy allows returns within 30 days of purchase.
"If the answer isn't in the context, say you don't know" in the system prompt is doing genuinely important work, not just filling space — without it, the model will often still fabricate a plausible-sounding answer even when the retrieved context doesn't actually contain one, defeating much of the point of building a retrieval step in the first place.
Chunking: a decision that matters more than it first appears to
Real documents are usually far longer than a single embedding call should ideally handle at once, which means splitting them into smaller chunks before embedding each one separately. Chunk size is a genuine trade-off, not an arbitrary implementation detail: chunks that are too large dilute a specific relevant sentence with a lot of surrounding, less-relevant text, hurting retrieval precision; chunks that are too small lose the surrounding context a passage needs to be properly understood on its own. Most real RAG systems land somewhere in the range of a few hundred tokens per chunk, often with a small overlap between consecutive chunks so a fact split across a chunk boundary isn't lost entirely in either piece.
def chunk_text(text, chunk_size=300, overlap=50):
tokens = encoding.encode(text)
chunks = []
start = 0
while start < len(tokens):
end = start + chunk_size
chunk_tokens = tokens[start:end]
chunks.append(encoding.decode(chunk_tokens))
start += chunk_size - overlap
return chunksA naive fixed-size splitter like this one is a genuinely reasonable starting point, but it's worth being aware it can split mid-sentence or even mid-paragraph without regard for the document's actual structure. A more structure-aware chunker — splitting on paragraph or section boundaries first, and only falling back to a fixed size for a genuinely long section — tends to produce chunks that read as coherent standalone passages, which usually improves both retrieval relevance and the quality of the final answer built from them.
Re-ranking: a second, more precise pass after initial retrieval
Vector similarity search is fast, but it's not always the most precise way to judge "is this chunk actually the best match for this specific query." A common production refinement retrieves a larger initial candidate set, then re-ranks it with a more precise (and more expensive) comparison before selecting the final top few:
def retrieve_with_reranking(query, documents, doc_embeddings, initial_k=20, final_k=3):
candidates = retrieve(query, documents, doc_embeddings, top_k=initial_k)
rerank_response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{
"role": "user",
"content": f"Query: {query}\n\nRank these passages by relevance, most relevant first:\n\n"
+ "\n---\n".join(candidates),
}],
)
ranked = parse_ranked_order(rerank_response.choices[0].message.content, candidates)
return ranked[:final_k]This two-stage approach — cheap, fast vector similarity to narrow a large corpus down to a manageable candidate set, then a more careful (often model-based) re-ranking pass on just those candidates — is a genuinely common pattern in production RAG systems, trading a small amount of extra latency and cost for meaningfully more precise final retrieval, especially on corpora large or ambiguous enough that pure vector similarity alone misses genuinely important distinctions between similar-looking chunks.
Evaluating a RAG system, not just building one
A RAG pipeline that "seems to work" on a handful of manual test questions can still fail silently on real user queries in ways that are hard to notice without deliberate evaluation. Two distinct things are worth measuring separately:
- Retrieval quality: given a query, did the system actually retrieve the chunk(s) that contain the real answer? This is measurable directly, given a test set of questions with known correct source passages — often reported as a hit rate at some
top_k. - Answer quality: given the retrieved context, did the model produce a correct, appropriately-grounded answer, including correctly saying "I don't know" when the context genuinely doesn't contain an answer? This usually needs either human review or a separate model-based evaluation pass (an "LLM as judge" comparing the answer against the retrieved context and a known-correct reference).
A RAG system with poor answers can be failing at either stage, and the fix is different depending on which: bad retrieval calls for better chunking, a different embedding model, or re-ranking; bad generation from good retrieval calls for prompt refinement instead. Measuring the two separately, rather than only checking end-to-end answer quality, is what actually tells you which stage to invest in improving.
Keeping the index current
A RAG system indexing documents once at launch and never again quietly drifts out of date the moment the underlying documents change — the same "rebuild on a schedule, not just at launch" principle covered for Docker base images and SageMaker model retraining elsewhere in this blog, applied here to a document index specifically. A production system typically re-embeds and re-indexes changed documents on a schedule or via a trigger tied to the document source itself (a webhook from a CMS, a scheduled job against a document store), rather than assuming an index built once stays accurate indefinitely.
What to actually remember from this post
- RAG retrieves relevant text first, then asks the model to answer using it — turning an open-book guess into a closed-book, grounded answer.
- Embeddings represent meaning, not vocabulary — retrieval works even when a query shares no words with the relevant document.
- A vector database becomes necessary at real scale; in-memory comparison is genuinely fine for a small, fixed document set.
- Chunk size is a genuine trade-off between retrieval precision and preserved context — structure-aware chunking usually beats naive fixed-size splitting.
- Re-ranking a larger candidate set improves precision beyond pure vector similarity, at a worthwhile cost in latency for corpora where that precision matters.
- Evaluate retrieval and generation quality separately — they fail for different reasons, and need different fixes.
- Explicitly instruct the model to say "I don't know" when context doesn't contain an answer, and keep the index current on a schedule — without either, fabrication and staleness both persist even with retrieval in place.
Next in the series: Fine-Tuning vs. Prompting: When Each Approach Makes Sense, the final post in this series — covering the other major lever for adapting a model's behavior to your specific use case.