Debugging and Observability in LangChain Applications

This is the fifth and final post in our LangChain fundamentals series, building on everything from chains through RAG, memory, and agents.

A chain that produced the wrong answer, or an agent that took a genuinely strange sequence of actions, is only debuggable if you can actually see every intermediate step: what prompt was sent, what the retriever returned, which tool the agent chose and why, what each step's exact input and output actually was. Without that visibility, "the agent gave a weird answer" isn't a debuggable problem statement — it's a shrug.

The simplest tool: verbose mode

executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

We already saw this in the previous post — verbose=True prints each step directly to the console as it happens. Genuinely useful for local development and quick debugging, and not something you'd want scattered through production logs at real volume, nor does it give you a way to go back and inspect a specific past run after the fact.

LangSmith: tracing built for exactly this problem

import os
 
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "your-langsmith-api-key"
os.environ["LANGCHAIN_PROJECT"] = "novuspark-support-agent"
 
# Every chain and agent invocation is now traced automatically —
# no other code change required
result = rag_chain.invoke("How many vacation days do I get?")

With tracing enabled, every chain or agent run is captured in LangSmith (LangChain's dedicated observability platform) as a structured trace: every prompt actually sent to the model, every document a retriever returned, every tool call an agent made and the reasoning that preceded it, exact token counts and latency per step. This is the direct answer to "the agent gave a weird answer" — instead of guessing, you open the actual trace for that specific run and see, step by step, exactly what happened and where it diverged from what you expected.

What to actually look for in a trace

• Retrieval quality: did the retriever actually return the relevant chunk, or did it retrieve something plausible-sounding but genuinely unrelated to the question? A wrong final answer traced back to bad retrieval needs a fix in chunking or embedding strategy — covered in our RAG post — not a fix to the prompt.
• Prompt construction: is the actual, fully-assembled prompt — after every template variable was filled in — what you intended? A subtle bug in how context gets formatted before insertion is often invisible until you look at the literal, final prompt text actually sent, not just the template.
• Tool selection: for an agent, did it choose a reasonable tool given the question, or does the tool's description need to be clearer? We flagged in the previous post that a tool's docstring is what the model uses to decide when to reach for it — a trace showing an agent skipping the right tool is often a signal the description needs to be more specific, not that the model is behaving unpredictably for no reason.

Comparing runs to catch regressions visually

Beyond inspecting a single trace, LangSmith supports comparing two runs of the same chain side by side — genuinely useful for confirming whether a prompt change or a chunking-strategy change actually improved things, rather than just "feeling" different:

from langsmith import Client
 
client = Client()
runs = client.list_runs(
    project_name="novuspark-support-agent",
    filter='eq(name, "rag_chain")',
    limit=2,
)

Comparing a run from before a prompt change against one from after, on the same input question, makes a regression or improvement directly visible — a genuinely different, more reliable signal than reading a diff of the prompt itself and guessing at its likely effect on output quality.

Evaluation: testing quality, not just checking for a crash

A chain or agent can run successfully — no exception, no error — and still produce a genuinely low-quality or subtly wrong answer. Standard software tests catch crashes; they don't catch "technically ran without error, but the answer is misleading." LangChain integrates with evaluation frameworks specifically built for this gap:

from langsmith.evaluation import evaluate
 
def correctness_evaluator(run, example):
    # Compare the chain's actual output against a known-good reference answer
    predicted = run.outputs["output"]
    expected = example.outputs["expected_answer"]
    # ... scoring logic, often using another model call to judge similarity
    return {"score": 1 if predicted_matches(predicted, expected) else 0}
 
evaluate(
    rag_chain.invoke,
    data="vacation-policy-qa-dataset",
    evaluators=[correctness_evaluator],
)

Running an evaluation suite against a fixed dataset of real questions with known-good expected answers, on every meaningful change to a prompt or chain structure, is what catches a regression before it reaches production — the direct equivalent of a test suite in ordinary software, applied to LLM output quality specifically, rather than assuming "it still runs" is a sufficient bar for something this probabilistic.

Building the evaluation dataset itself

An evaluation suite is only as good as the dataset behind it. A dataset built entirely from questions the development team thought to ask tends to miss the genuinely surprising ways real users phrase things:

from langsmith import Client
 
client = Client()
dataset = client.create_dataset("vacation-policy-qa-dataset")
 
client.create_examples(
    inputs=[
        {"question": "How many vacation days do I get?"},
        {"question": "what about sick days"},
        {"question": "can i carry over unused pto to next year"},
    ],
    outputs=[
        {"expected_answer": "25 days per year, accrued monthly."},
        {"expected_answer": "10 paid sick days per year, does not require prior approval."},
        {"expected_answer": "Up to 5 unused vacation days can be carried over."},
    ],
    dataset_id=dataset.id,
)

A genuinely useful practice is sourcing real questions from actual production usage (via the structured logging covered below) into this dataset over time, alongside the initial hand-written examples — so the evaluation set grows to reflect what users actually ask, not just what the development team anticipated when the application launched.

Logging real production usage, not just traces from testing

import logging
 
logger = logging.getLogger("langchain_app")
 
def logged_invoke(chain, inputs, session_id):
    logger.info(f"session={session_id} input={inputs}")
    result = chain.invoke(inputs)
    logger.info(f"session={session_id} output={result}")
    return result

Tracing during development answers "why did this specific test case behave this way." Structured logging in production answers a different, equally necessary question: what are real users actually asking, how often do certain failure patterns show up in the wild, and which prompts or chain structures need the next round of improvement — the same shift from local debugging to production observability that's true of any real software system, applied here to a probabilistic one where "did it work" is a fuzzier question than a stack trace can fully answer on its own.

Adding custom metadata to a trace

A trace is meaningfully more useful when it's tagged with context beyond the raw prompt and response — which user, which feature, which experiment variant produced this specific run:

result = rag_chain.invoke(
    "How many vacation days do I get?",
    config={
        "tags": ["production", "vacation-policy-feature"],
        "metadata": {"user_id": "user-123", "experiment_variant": "b"},
    },
)

tags and metadata on the config passed to .invoke() attach directly to the resulting trace in LangSmith, making it possible to filter and group traces by exactly this kind of context later — genuinely useful once a real production system runs many concurrent features, users, or A/B test variants through the same underlying chains, and a raw list of undifferentiated traces stops being enough to answer "how is this specific experiment variant actually performing."

Setting alerts on production quality, not just errors

Standard application monitoring alerts on exceptions and error rates — genuinely necessary, and insufficient on its own for a probabilistic system where "no error" doesn't mean "good answer." Pairing production logging (covered above) with a periodic automated quality check against a sample of recent real interactions closes that gap:

def run_periodic_quality_check():
    recent_interactions = fetch_recent_production_logs(hours=24)
    sample = random.sample(recent_interactions, min(50, len(recent_interactions)))
    scores = [score_with_llm_judge(interaction) for interaction in sample]
    average_score = sum(scores) / len(scores)
    if average_score < QUALITY_THRESHOLD:
        alert_team(f"Quality score dropped to {average_score}")

Running this on a schedule — the same "check deliberately and regularly" instinct covered for auditing SageMaker endpoints and rescanning Docker images elsewhere in this blog — catches a genuine quality regression (a prompt change that quietly made answers worse, a data source that went stale) that a purely error-rate-based alert would never surface at all, since the system is still "working" in the narrow sense of not throwing exceptions.

What to actually remember from this series

• Verbose mode is for quick, local debugging — LangSmith or an equivalent tracing tool is what makes past production runs actually inspectable after the fact.
• A wrong final answer usually traces back to either bad retrieval or a subtly malformed prompt — inspect the actual, fully-assembled request, not just the template.
• Comparing two traces side by side confirms whether a change actually improved things, rather than relying on a subjective read of a prompt diff.
• "It ran without an error" is not the same bar as "it produced a good answer" — evaluation against a real dataset with known-good answers is what actually tests output quality, and that dataset should grow from real production usage over time, not stay fixed to its initial hand-written examples.
• Production logging and development tracing answer different questions — one tells you what real users are actually asking; the other tells you why one specific test case behaved the way it did.

That closes out our LangChain fundamentals series — from chains and prompts through RAG, memory, agents, and now observability. For genuinely complex, multi-step agent behavior — including branching and revisiting earlier steps — LangGraph, covered next in this blog, extends these same concepts further. If your team is building real production AI applications, this is exactly the kind of hands-on work we build our AI & Generative AI training around.