Building RAG Applications with LangChain

This is the second post in our LangChain fundamentals series, building on chains, prompts, and models. It also assumes the RAG concepts from our OpenAI series.

We built a RAG pipeline from raw embeddings and manual cosine-similarity comparisons earlier in this blog — genuinely useful for understanding what's actually happening underneath, and genuinely more manual work than a real application needs to take on directly. LangChain provides purpose-built abstractions for the recurring parts of that pipeline: loading documents, splitting them into chunks, storing and searching embeddings.

Loading and splitting real documents

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
 
loader = PyPDFLoader("employee-handbook.pdf")
documents = loader.load()
 
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
)
chunks = splitter.split_documents(documents)
print(f"{len(documents)} pages split into {len(chunks)} chunks")

24 pages split into 87 chunks

RecursiveCharacterTextSplitter tries to split on natural boundaries first — paragraph breaks, then sentences, then words — only falling back to a hard character-count cut when nothing more natural is available nearby. This directly addresses the chunking trade-off flagged in our OpenAI RAG post: chunk_overlap gives each chunk a small amount of shared context with its neighbor, so a fact or sentence that happens to fall right at a chunk boundary isn't lost entirely from either piece.

Document loaders exist for dozens of real-world formats beyond PDF — worth knowing the ecosystem covers this, rather than writing a custom parser per format:

from langchain_community.document_loaders import (
    Docx2txtLoader,
    UnstructuredHTMLLoader,
    NotionDBLoader,
    CSVLoader,
)

Each loader normalizes its source format into the same Document object (page content plus metadata like source filename or page number), which is what lets everything downstream — the splitter, the embedder, the vector store — work identically regardless of which loader originally produced the documents.

Embedding and storing in a vector store

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
 
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory="./chroma_db")

One call replaces the manual embedding-and-storage loop from our raw-Python RAG example — Chroma.from_documents embeds every chunk and stores the results in a local vector database, persisted to disk so it doesn't need to be rebuilt on every application restart. Swapping Chroma for Pinecone, Weaviate, or pgvector for a production deployment at real scale is a difference of a few lines, not a pipeline rewrite — the same "swap the provider behind a consistent interface" benefit covered in the previous post, now applied to vector storage specifically.

Retrieval, as a first-class LangChain object

retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
relevant_chunks = retriever.invoke("How many vacation days do I get?")

as_retriever() turns the vector store into a Retriever — an object with a consistent .invoke() interface, exactly like the models and parsers from the previous post, meaning a retriever can be dropped directly into an LCEL chain alongside them.

Composing the full RAG chain

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
 
prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer using only this context. If the answer isn't here, say you don't know.\n\n{context}"),
    ("user", "{question}"),
])
 
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)
 
rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | model
    | StrOutputParser()
)
 
answer = rag_chain.invoke("How many vacation days do I get?")
print(answer)

Based on the employee handbook, full-time employees receive 25 days
of paid vacation per year, accrued monthly.

This is the exact same four-step RAG pattern from our OpenAI series — retrieve, format context, prompt with that context, generate — expressed as one composed LCEL chain instead of a hand-written function. RunnablePassthrough() simply passes the original question straight through unchanged, so it's available to the prompt template alongside the retrieved context, both flowing into the same final prompt step.

Returning sources alongside the answer

A production RAG application usually needs to show which document a given answer came from, not just the answer text — genuinely important for user trust, and for debugging a wrong answer back to its retrieval source:

from langchain_core.runnables import RunnableParallel
 
rag_chain_with_sources = RunnableParallel(
    context=retriever,
    question=RunnablePassthrough(),
).assign(answer=(
    {"context": lambda x: format_docs(x["context"]), "question": lambda x: x["question"]}
    | prompt
    | model
    | StrOutputParser()
))
 
result = rag_chain_with_sources.invoke("How many vacation days do I get?")
print(result["answer"])
print([doc.metadata["source"] for doc in result["context"]])

Based on the employee handbook, full-time employees receive 25 days of paid vacation per year, accrued monthly.
['employee-handbook.pdf']

RunnableParallel runs the retriever and passes the question through simultaneously, then .assign() adds the generated answer as an additional key alongside the raw retrieved documents — meaning the final result carries both the answer and the actual source documents it was grounded in, each document's metadata (populated automatically by the loader) tracing back to exactly which file and page it came from.

Where this earns its keep over the hand-rolled version

The manual version from our OpenAI series is genuinely clearer for understanding the concept, and entirely reasonable for a small, fixed set of documents. LangChain's version earns its abstraction once you need: document loaders for multiple real file formats (PDF, Word, HTML, Notion exports, and dozens more, each with its own loader), a persistent, swappable vector store rather than an in-memory Python list, traceable sources alongside every generated answer, and composability with everything else in the ecosystem — memory, agents, and tools, covered in the remaining posts in this series, all sharing the same underlying Runnable interface as the retriever and chain shown here.

Handling documents that update over time

A real document set rarely stays static — a policy document gets revised, a page gets deleted. Re-embedding an entire corpus from scratch on every change is wasteful once a knowledge base grows past a handful of documents:

def update_document(doc_id: str, new_content: str, vectorstore):
    vectorstore.delete(ids=[doc_id])
    new_chunks = splitter.split_text(new_content)
    vectorstore.add_texts(new_chunks, ids=[f"{doc_id}-{i}" for i in range(len(new_chunks))])

Deleting a document's existing chunks by a stable ID before re-adding its updated version keeps a vector store's contents current without a full re-index — the same incremental-update instinct as a Terraform apply only touching what actually changed, rather than destroying and recreating an entire infrastructure stack on every run.

Filtering retrieval by metadata

Beyond pure similarity search, LangChain retrievers support filtering on metadata attached during ingestion — genuinely useful once a document set spans multiple categories that shouldn't be conflated in retrieval:

retriever = vectorstore.as_retriever(
    search_kwargs={"k": 3, "filter": {"department": "hr"}}
)

Scoping retrieval to department: hr before similarity ranking even applies prevents an HR policy question from surfacing a similarly-worded but irrelevant engineering document as a top match — the same metadata-filtering principle covered for Bedrock Knowledge Bases in a later series, expressed here through LangChain's retriever interface directly.

Choosing a retrieval search type beyond plain similarity

as_retriever() supports search types beyond default similarity search — mmr (Maximal Marginal Relevance) specifically balances relevance against diversity, avoiding a top-k result set of near-duplicate chunks that all say roughly the same thing:

retriever = vectorstore.as_retriever(search_type="mmr", search_kwargs={"k": 3, "fetch_k": 10})

fetch_k pulls a larger candidate pool before MMR re-ranks it for diversity — worth reaching for specifically when a document set has genuine redundancy (several similarly-worded passages covering the same policy) and pure similarity search would otherwise return three near-identical chunks instead of three genuinely distinct, useful ones.

What to actually remember from this post

• RecursiveCharacterTextSplitter chunks on natural boundaries first — paragraphs and sentences, before falling back to a hard character cut.
• chunk_overlap prevents a fact from being lost entirely at a chunk boundary — the concrete fix for the chunking trade-off raised in our OpenAI RAG post.
• A retriever is a first-class, composable object in LangChain, sharing the same .invoke() interface as models and parsers.
• RunnableParallel and .assign() let a chain return sources alongside an answer — genuinely important for user trust and for tracing a wrong answer back to its retrieval source.
• The full RAG chain composes retrieval, formatting, prompting, and generation into one LCEL pipeline — the same four conceptual steps as the hand-rolled version, with far less manual glue code.

Next in the series: LangChain Memory: Giving Your Application Context, where we cover how a chain remembers previous turns in an actual conversation.