What Is AI Sovereignty? The New Buzzword Explained?
AI TOOLS DEVELOPER GUIDE LANGCHAIN 2026
A
Complete, Working Walkthrough — From a Blank File to a Retrieval Pipeline That
Actually Holds Up
LangChain's own official RAG tutorial gets a
working app running in about 40 lines of Python — a genuinely good way to learn
the shape of the problem, and a genuinely bad production design. This guide
builds the same core pipeline, then walks through exactly what changes once
real documents, real users, and real follow-up questions show up.
Retrieval-Augmented Generation, or RAG, solves a
specific, common problem: a large language model only knows what it was trained
on, and re-training a model every time your documents change isn't remotely
practical. RAG instead fetches relevant chunks of your own data at the moment
someone asks a question, and hands those chunks to the model as context — so
the model answers using your actual documents, not just what it happened to
learn during training.
LangChain has become the most widely used
framework for wiring this together, with well over 100,000 GitHub stars and
millions of monthly downloads across its Python and JavaScript ecosystems. It
handles the unglamorous plumbing — loading documents, splitting them into
chunks, calling an embedding model, talking to a vector database, and composing
all of it with a language model — through a consistent set of interfaces that
work whether you're using Claude, GPT, Gemini, or a fully local open-source
model.
This guide builds a working RAG application step
by step using LangChain's current v1.x interface with LangChain Expression
Language (LCEL), then covers the specific changes that separate a working
prototype from something you'd actually trust in production — chunking
strategy, conversational memory, hybrid search, and evaluation. Every code
block is complete and ordered so you can build the whole thing by following
along top to bottom.
One framing note before diving in: this guide
uses Claude via langchain-anthropic for generation, since that's the model
family this site covers in depth elsewhere, but every pattern shown here is
provider-agnostic by design. LangChain's standard chat model interface means
swapping ChatAnthropic for ChatOpenAI or a Gemini equivalent is a one-line
change — the retrieval logic, prompt structure, and chain composition around it
stay identical regardless of which model is actually generating the final
answer.
ALSO READ THIS: AI + Cybersecurity: How Hackers Are Using AI Now
|
In Two
Sentences A RAG app built with LangChain loads your
documents, splits them into chunks, embeds those chunks into a vector store,
and — at query time — retrieves the most relevant chunks and hands them to a
language model alongside the user's question, so answers stay grounded in
your actual data instead of the model's training knowledge alone. The basic
pipeline is genuinely simple to build; the real engineering work is in
chunking strategy, retrieval quality, and evaluation — the parts a 40-line
tutorial skips entirely. |
|
100K+ GitHub stars on LangChain,
making it the most widely adopted LLM application framework |
~$0.01 Typical cost of a single RAG
query with 4 retrieved chunks and a 500-token response |
4 Core pipeline stages every RAG
app needs: load, split, embed and store, retrieve and generate |
This tutorial assumes basic Python familiarity
and a terminal you're comfortable working in. You'll need Python 3.9 or later,
an API key for whichever LLM provider you choose (this guide uses Claude via
langchain-anthropic, though the pattern is identical for OpenAI or Gemini), and
— for the embedding step — either an OpenAI API key or a local embedding model
if you'd rather avoid a second provider.
Start by creating an isolated project directory
and virtual environment, which keeps this project's dependencies separate from
anything else on your system.
Terminal
|
mkdir langchain-rag-app
&& cd langchain-rag-app python3 -m venv venv source
venv/bin/activate # Windows:
venv\Scripts\activate pip install langchain
langchain-anthropic langchain-openai \ langchain-community langchain-chroma \ chromadb pypdf |
A quick note on package names, since this trips
up a lot of people coming back to LangChain after a break: as of v0.3 and the
current v1.x line, LangChain split its provider integrations into separate
packages — langchain-anthropic for Claude, langchain-openai for OpenAI,
langchain-community for open-source models via Ollama and Hugging Face — rather
than bundling every provider into the core library. Installing only what you
need keeps your dependency tree smaller and your install times faster.
The first stage of any RAG pipeline is getting
your source documents into memory and breaking them into chunks small enough to
embed and retrieve individually. LangChain's document loaders handle dozens of
formats — PDF, Markdown, HTML, plain text, even entire directories — through a
consistent interface.
ingest.py
|
from
langchain_community.document_loaders import DirectoryLoader, PyPDFLoader from
langchain_text_splitters import RecursiveCharacterTextSplitter # Load every PDF in a
folder loader =
DirectoryLoader( "./documents", glob="**/*.pdf", loader_cls=PyPDFLoader, ) raw_documents =
loader.load() # Split into
overlapping chunks that preserve context splitter =
RecursiveCharacterTextSplitter( chunk_size=1000, chunk_overlap=200, separators=["\n\n",
"\n", ". ", " ", ""], ) chunks =
splitter.split_documents(raw_documents) print(f"Loaded
{len(raw_documents)} documents") print(f"Split into
{len(chunks)} chunks") |
The chunk_size and chunk_overlap values here —
1,000 characters with a 200-character overlap — are a reasonable starting
default, not a universal answer. The overlap exists specifically so a sentence
or idea that gets cut off at a chunk boundary still appears in full inside at least
one chunk, rather than being split across two chunks in a way that damages its
meaning. Section 6 covers how and why to tune these numbers for your actual
documents.
It's worth understanding why
RecursiveCharacterTextSplitter specifically is the right default over a simpler
fixed-length splitter. Rather than cutting text at exactly character 1,000
regardless of what's there, it works through its list of separators in order of
preference — trying to split on paragraph breaks first, then line breaks, then
sentence endings, and only falling back to splitting mid-word as a last resort.
The practical effect is that chunks tend to end at natural language boundaries
far more often than a naive fixed-length approach would produce, which
measurably helps both embedding quality and how coherent a retrieved chunk
reads once it's handed to the language model as context.
Each chunk needs to be converted into an
embedding — a vector of numbers that captures its meaning — so the system can
later find chunks that are semantically similar to a user's question, not just
chunks that share exact keywords. LangChain's Chroma integration handles both
the embedding call and the storage in a single step for a local, file-based
vector store, which is the right choice for a prototype or a small-to-medium
production workload.
ingest.py (continued)
|
from langchain_openai
import OpenAIEmbeddings from langchain_chroma
import Chroma embeddings =
OpenAIEmbeddings(model="text-embedding-3-small") vector_store =
Chroma.from_documents( documents=chunks, embedding=embeddings,
persist_directory="./chroma_db", ) print("Vector
store built and saved to ./chroma_db") |
text-embedding-3-small is a reasonable default
for most projects in 2026 — it offers a strong balance of retrieval quality,
speed, and cost, at roughly two cents per million tokens embedded. If you're
working with dense technical, legal, or medical documents where retrieval
precision matters more than cost, it's worth benchmarking a higher-end
embedding model, such as one from Voyage AI or Cohere, against your own
document set before committing — the right choice genuinely depends on what
you're embedding, and the cost difference at typical document volumes is
usually small relative to what a meaningful accuracy improvement is worth.
ALSO READ THIS: What Is Generative Engine Optimization (GEO)?
With your documents embedded and stored, the
query-time pipeline is where LangChain Expression Language (LCEL) does its real
work — composing the retriever, prompt template, and language model into a
single callable chain using Python's pipe operator, which makes the data flow
through each stage easy to read top to bottom.
query.py
|
from
langchain_anthropic import ChatAnthropic from langchain_openai
import OpenAIEmbeddings from langchain_chroma
import Chroma from
langchain_core.prompts import ChatPromptTemplate from
langchain_core.output_parsers import StrOutputParser from
langchain_core.runnables import RunnablePassthrough # Reconnect to the
vector store built during ingestion embeddings =
OpenAIEmbeddings(model="text-embedding-3-small") vector_store = Chroma(
persist_directory="./chroma_db", embedding_function=embeddings, ) retriever =
vector_store.as_retriever(search_kwargs={"k": 4}) # Prompt template —
instructs the model to answer only from context prompt =
ChatPromptTemplate.from_template(""" Answer the question
using only the context below. If the context doesn't
contain the answer, say you don't know. Context: {context} Question: {question} """) llm =
ChatAnthropic(model="claude-sonnet-4-6", temperature=0) def format_docs(docs): return
"\n\n".join(doc.page_content for doc in docs) rag_chain = ( {"context": retriever |
format_docs, "question": RunnablePassthrough()} | prompt | llm | StrOutputParser() ) answer =
rag_chain.invoke("What does the onboarding policy say about remote
work?") print(answer) |
Reading this chain from the inside out:
RunnablePassthrough() lets the user's raw question flow through unchanged into
the question slot of the prompt, while the retriever pulls the four most
relevant chunks (k=4) and format_docs joins them into a single block of text
for the context slot. The pipe operator chains each stage's output directly
into the next stage's input — retriever results into the prompt, the filled
prompt into the model, and the model's response through an output parser that
returns a clean string instead of a raw message object.
The instruction to answer only from the provided
context, and to say so plainly when the context doesn't contain an answer, is
doing real work here — it's the single most effective prompt-level lever for
keeping a RAG app from confidently making something up when the retrieved
chunks simply don't cover what was asked.
The chain above works well for a single,
standalone question, but it breaks down the moment a user asks a natural
follow-up. If someone first asks "What's the deployment process?" and
then follows up with "How long does it take?", the retriever has no
way to know that "it" refers to the deployment process — it will
search for documents about the word "it," which is meaningless. The
fix is to reformulate follow-up questions using conversation history before
retrieval ever happens.
query.py (conversational version)
|
from langchain.chains
import create_history_aware_retriever, create_retrieval_chain from
langchain.chains.combine_documents import create_stuff_documents_chain from
langchain_core.prompts import MessagesPlaceholder # Prompt that rewrites
a follow-up question into a standalone one condense_prompt =
ChatPromptTemplate.from_messages([
MessagesPlaceholder("chat_history"), ("user", "{input}"), ("user", "Given the
conversation above, rewrite the last " "question as a standalone
question that includes " "any needed
context."), ]) history_aware_retriever
= create_history_aware_retriever( llm, retriever, condense_prompt ) answer_prompt =
ChatPromptTemplate.from_messages([ ("system", "Answer using
only this context:\n\n{context}"),
MessagesPlaceholder("chat_history"), ("user", "{input}"), ]) document_chain =
create_stuff_documents_chain(llm, answer_prompt) conversational_rag_chain
= create_retrieval_chain( history_aware_retriever, document_chain ) # Usage — chat_history
is a list of prior HumanMessage/AIMessage turns result =
conversational_rag_chain.invoke({ "input": "How long does it
take?", "chat_history":
chat_history, # built up turn by turn }) print(result["answer"]) |
This two-stage structure — rewrite, then
retrieve and answer — is the standard LangChain pattern for conversational RAG
as of the current v1.x release line, and it's a meaningfully different shape
than simply appending chat history onto a single prompt, which tends to confuse
the retriever the same way the raw follow-up question would.
ALSO READ THIS: How to Detect AI-Generated Text — Best Tools
Everything above will run and produce reasonable
answers on a modest document set. It will also, per widely shared 2026
practitioner writing on exactly this gap, start breaking down in specific,
predictable ways once real usage begins — and it's worth naming those gaps
directly rather than discovering them after launch.
• Chunk size is rarely
one-size-fits-all: a fixed
1,000-character chunk works reasonably for prose but fragments tables, code
blocks, and structured data badly — production pipelines increasingly use
format-aware splitting that respects a document's actual structure rather than
cutting at a fixed character count.
• Pure vector similarity search
misses exact matches: semantic
search is excellent at finding conceptually related content but can miss a
chunk containing an exact product code, error message, or proper noun that a
simple keyword search would catch instantly — production systems increasingly
combine vector search with keyword-based (BM25) search in a hybrid retriever.
• k=4 is a guess, not a tuned
parameter: retrieving too
few chunks starves the model of context it needs; retrieving too many dilutes
the prompt with irrelevant material and increases cost — the right value
depends on your chunk size and document density, and is worth testing rather
than assuming.
• There's no evaluation loop: the code above has no way to tell you whether
retrieval quality is actually good, or whether answers are grounded in the
retrieved context versus quietly drifting — LangSmith, LangChain's companion
observability platform, is built specifically to trace and evaluate exactly
this.
It's worth being specific about how these gaps
tend to surface in practice, since they rarely announce themselves clearly. A
chunking problem usually shows up as answers that are technically correct but
oddly incomplete — the model genuinely didn't have the missing information in
its context window, not because retrieval failed outright, but because the
relevant fact was split across a chunk boundary into two pieces, neither of
which scored highly enough to make the top-k cut. A retrieval-quality problem
tends to look different: confident, fluent answers that are simply wrong,
because the retriever returned chunks that were topically related but didn't
actually contain the specific fact being asked about, and the language model —
instructed to answer from context — did its best with imperfect material rather
than recognizing the gap. Distinguishing between these two failure modes is
exactly the kind of diagnosis an evaluation loop and tracing tool make
tractable, and exactly what's impossible to do reliably by just reading a
handful of outputs and guessing.
|
Why This
Matters Practically None of these gaps mean the basic pipeline is
wrong to start with — it's the fastest way to validate that RAG is even the
right approach for your use case before investing in the harder problems. The
mistake is treating the 40-line version as a finished product rather than a
first draft. Every serious production RAG deployment covered in current
LangChain guidance layers hybrid search, tuned chunking, and evaluation on
top of exactly the pipeline built in Sections 2 through 5 — it's an extension
of the same foundation, not a different architecture. |
Chroma, used throughout this guide, is a strong
default for getting started: it runs locally with no separate infrastructure to
manage, persists to disk, and integrates directly with LangChain's retriever
interface. It's genuinely fine for production too, up to a meaningful scale —
but it's worth knowing what you'd reach for next as data volume or query load
grows.
• FAISS: a fast, in-memory library from Meta, well-suited
to prototypes and research work where persistence and multi-user access aren't
primary concerns.
• Pinecone or Weaviate: fully managed, cloud-hosted vector databases
built for production scale, handling replication, high query volume, and
metadata filtering without you managing the underlying infrastructure yourself.
• SingleStore or Postgres with
pgvector: a strong choice when you want
vector search living alongside your existing relational data rather than as a
separate specialized system, simplifying your overall data architecture at the
cost of some specialized vector-search performance.
LangChain's provider-neutral design means
switching between any of these later is a matter of changing the integration
class and its configuration, not rewriting your pipeline logic — the retriever
interface built in Section 4 stays identical regardless of which vector store
sits behind it.
|
📄 An
Internal Policy Chatbot The exact pipeline from Sections 2–5, pointed at HR documents A company loads its employee handbook,
benefits documentation, and IT policies through the ingestion pipeline in
Section 2, then deploys the conversational chain from Section 5 behind a
simple internal chat interface, letting employees ask natural-language
questions about vacation policy or expense reporting instead of searching a
shared drive. Result: New hires get accurate, sourced answers to
routine policy questions in seconds, and the documents stay the single source
of truth rather than being paraphrased incorrectly from memory. |
|
⚖️ A Legal
Team Searching Case Files Hybrid search added specifically because exact terms matter A legal team building a RAG tool over case
files finds that pure vector search occasionally misses documents containing
an exact statute number or case citation a lawyer searched for verbatim.
Adding a keyword-based retriever alongside the vector retriever, combined
through LangChain's ensemble retriever pattern, catches both semantic
questions and exact-term lookups in the same system. Result: A retrieval system that handles both 'find
cases about breach of contract' and 'find the document citing Section 12.4'
equally well. |
|
🔧 Debugging
Why Retrieval Quality Dropped LangSmith tracing pinpointing a chunking problem after the
fact A support-documentation RAG app that worked
well for months starts giving vague answers after a large batch of new,
table-heavy documents was added. Tracing recent queries through LangSmith
reveals the new documents' tables were being fragmented mid-row by the
fixed-size text splitter, scattering related data across multiple
disconnected chunks. Result: A concrete diagnosis — switch to a
format-aware splitter for tabular content — reached through tracing rather
than guesswork. |
The pipeline built in this guide — load, split,
embed, retrieve, generate, with conversational memory layered on top — covers
the large majority of real RAG use cases without needing anything more exotic.
A handful of natural next steps are worth knowing about once that foundation is
solid.
• LangGraph for more complex
control flow: when a RAG app
needs to conditionally decide whether to retrieve at all, retry a failed
retrieval with a reformulated query, or route between multiple different
retrievers, LangGraph — LangChain's graph-based orchestration layer — handles
that branching logic more cleanly than a linear LCEL chain can.
• Re-ranking retrieved chunks: adding a dedicated re-ranking model as a second
pass after initial retrieval, before the top results go to the language model,
is one of the most reliable ways to improve answer quality without changing
your chunking or embedding strategy at all.
• Structured evaluation
datasets: building a set of
representative questions with known-good answers, and running that set through
LangSmith's evaluation tooling after every pipeline change, turns "does
this feel better" into an actual measurable comparison.
The core lesson underlying every section of this
guide is the same one experienced LangChain developers converge on repeatedly:
the framework makes the easy 80% of a RAG app genuinely easy, and it's honest
about the fact that the remaining 20% — chunking strategy tuned to your actual
documents, retrieval quality tuned to your actual queries, and evaluation tuned
to your actual users — is where the real engineering work lives, and where no
framework can fully do the thinking for you.
ALSO READ THIS: Neural Networks Explained Visually
Building a working RAG app with LangChain takes
a genuinely small amount of code — document loading, chunking, embedding, and
an LCEL chain tying retrieval to generation, all of it covered in the sections
above. That's real, and it's why LangChain has become the default starting
point for so many RAG projects: the path from zero to a working prototype is
short and well-documented.
What separates that prototype from something
worth deploying is exactly what this guide spent its second half on: chunking
that respects your documents' actual structure, retrieval that combines
semantic and keyword search, conversational memory that handles real follow-up
questions, and an evaluation loop that tells you when something's quietly
gotten worse. None of that requires a different framework — it's the same LCEL
foundation, extended in the specific directions your actual documents and
actual users demand.
Have you built a RAG app with LangChain? What
tripped you up first — chunking, retrieval, or something else? Share it in the
comments below.
Comments
Post a Comment