How to Use LangChain with LanceDB (Serverless Embeddings)

I was running Chroma locally for a RAG project when I noticed the database directory had grown to 4.7 GB for a corpus that was only about 800 MB of raw text. The SQLite backend Chroma uses is convenient, but it's not exactly space-efficient at scale. A colleague pointed me toward LanceDB, and after one afternoon of testing, I moved the whole project over.

LanceDB stores vectors in the Lance columnar format — the same format used by the LanceDB team for multimodal datasets. It's dramatically more compact than SQLite-backed vector stores, it supports approximate nearest neighbor (ANN) search with IVF_PQ indexing, and it runs entirely locally without any server process. The LangChain integration is clean enough that migration from Chroma or FAISS takes under an hour.

This guide covers everything from first install to a production-ready RAG pipeline: local setup, cloud mode, IVF_PQ index creation, ANN search optimization, and a complete deployable example. I'll also include a comparison table against Chroma and SQLite-VSS so you can see exactly where LanceDB wins (and where it doesn't).

For context on other vector storage options, the vector database guide covers the broader landscape including Pinecone, Weaviate, and Qdrant.

Why LanceDB Is Worth Your Attention

Most vector databases fall into one of two categories: hosted services (Pinecone, Weaviate Cloud) or local libraries (Chroma, FAISS). LanceDB sits in an interesting middle position — it's a local library that can also sync to cloud storage (S3, GCS, Azure Blob), giving you a serverless architecture without paying for a managed service.

A few things stand out:

No server process. LanceDB runs in-process, just like FAISS or Chroma. No Docker, no daemon, no port management.

Lance columnar format. Lance uses a columnar storage format optimized for random access on large embedding arrays. Reads are 3–10× faster than row-oriented formats for vector workloads.

Native versioning. Every write to a LanceDB table creates a new version. You can roll back to any previous state without extra tooling — genuinely useful for production data pipelines.

Multimodal support. Text, images, audio — LanceDB handles any fixed-dimension vector. You can store multiple vector columns in the same table.

According to benchmarks published by the LanceDB team (lancedb.com, 2024), LanceDB's IVF_PQ index achieves sub-10ms latency at the 95th percentile for 1M vectors on a standard laptop, with recall@10 above 95%.

Comparison: LanceDB vs Chroma vs SQLite-VSS

Feature	LanceDB	Chroma	SQLite-VSS
Disk usage (1M vectors, 1536-dim)	~6 GB	~14 GB	~18 GB
ANN query speed (p95)	8ms	45ms	120ms
Python ergonomics	Excellent	Excellent	Moderate
Cloud sync	S3/GCS/Azure	No	No
Free / open source	Yes (Apache 2.0)	Yes (Apache 2.0)	Yes (MIT)
Versioning / rollback	Built-in	No	No
Multimodal columns	Yes	No	No
Filtering on metadata	SQL-like	Dict filter	SQL

LanceDB's disk efficiency and query speed are the standout advantages. Chroma is simpler to get started with, but LanceDB scales much better. SQLite-VSS is worth knowing about for projects that are already deeply embedded in the SQLite ecosystem, but it's the slowest of the three.

Installation

pip install lancedb langchain-community langchain-openai
# Optional: cloud sync support
pip install lancedb[cloud]

LanceDB requires Python 3.8+ and uses pyarrow under the hood for the columnar format. That gets installed automatically with lancedb.

Local Setup: First Steps

The simplest possible LanceDB setup stores everything in a local directory:

import lancedb
import os

# Create (or connect to) a local database
db = lancedb.connect("./my_lance_db")

# List existing tables
print("Existing tables:", db.table_names())

That's it. No configuration, no initialization script. The ./my_lance_db directory gets created if it doesn't exist. Each call to db.create_table() or db.open_table() manages a subdirectory within that folder.

LangChain Integration: Storing Embeddings

LangChain's LanceDB vector store class wraps the native client. Here's the standard setup pattern:

from langchain_community.vectorstores import LanceDB
from langchain_openai import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import TextLoader
import lancedb

# Load and chunk documents
loader = TextLoader("./knowledge_base.txt")
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = splitter.split_documents(documents)

# Connect to LanceDB
db = lancedb.connect("./rag_lance_db")

# Initialize embeddings
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Create vector store from documents
vectorstore = LanceDB.from_documents(
    documents=chunks,
    embedding=embeddings,
    connection=db,
    table_name="knowledge_base"
)

print(f"Stored {len(chunks)} chunks in LanceDB")

If the table already exists and you want to add more documents:

# Open existing table
vectorstore = LanceDB(
    connection=db,
    embedding=embeddings,
    table_name="knowledge_base"
)

# Add new documents
new_docs = [...]  # Your new Document objects
vectorstore.add_documents(new_docs)

One thing I appreciate: unlike some vector stores, LanceDB doesn't silently overwrite existing data. If you try to create a table that already exists, it raises an error unless you explicitly pass mode="overwrite".

For more on chunking strategies that pair well with LanceDB, the LangChain tutorial 2025 covers splitting approaches in depth.

Similarity Search

Basic similarity search works the same as any LangChain vector store:

# Simple similarity search
results = vectorstore.similarity_search(
    "How does the authentication system work?",
    k=5
)

for doc in results:
    print(f"Score source: {doc.metadata.get('source', 'unknown')}")
    print(doc.page_content[:200])
    print("---")

# Search with scores
results_with_scores = vectorstore.similarity_search_with_score(
    "token rate limits",
    k=5
)

for doc, score in results_with_scores:
    print(f"Distance: {score:.4f}")
    print(doc.page_content[:150])

Lower distance scores mean higher relevance in LanceDB's default cosine distance metric.

Metadata Filtering

LanceDB supports SQL-like filter expressions, which is more expressive than Chroma's dictionary-based filtering:

from langchain_community.vectorstores import LanceDB

# Search with metadata filter
results = vectorstore.similarity_search(
    "authentication flow",
    k=5,
    filter="source = 'auth_docs.txt' AND created_year >= 2024"
)

# Filter by category
results = vectorstore.similarity_search(
    "rate limiting",
    k=5,
    filter="category IN ('api', 'networking')"
)

This SQL-like syntax covers most production filtering needs without the verbosity of some other APIs. Filters are applied at the storage layer before the ANN search, so they're fast.

Building the IVF_PQ Index

By default, LanceDB does brute-force exact search. This is fine for small datasets (under 50,000 vectors), but for larger corpora you want to build an IVF_PQ (Inverted File with Product Quantization) index.

IVF_PQ works in two stages:

IVF (Inverted File): Clusters vectors into num_partitions groups. At query time, only the nearest clusters are searched.
PQ (Product Quantization): Compresses vectors by splitting them into sub-vectors and approximating each with a codebook. This reduces memory usage dramatically.

import lancedb
import numpy as np

# Connect and open table
db = lancedb.connect("./rag_lance_db")
table = db.open_table("knowledge_base")

# Build IVF_PQ index
# num_partitions: number of IVF clusters (rule of thumb: sqrt(num_rows))
# num_sub_vectors: PQ compression (higher = less compression but better recall)
table.create_index(
    metric="cosine",
    num_partitions=256,       # Clusters; increase for larger datasets
    num_sub_vectors=96,       # Sub-vectors for PQ; must divide embedding dim
    vector_column_name="vector"
)

print("IVF_PQ index created successfully")

For the text-embedding-3-small model (1536 dimensions), num_sub_vectors=96 works well since 1536 / 96 = 16 bytes per sub-vector. For text-embedding-ada-002 (also 1536-dim), the same config applies.

Some rules of thumb for num_partitions:

Under 100k vectors: 64–128
100k–1M vectors: 256–1024
Over 1M vectors: 1024–4096

# Verify index was created
print(table.schema)

# Check index info
indices = table.list_indices()
for idx in indices:
    print(f"Index: {idx}")

ANN Search with the Index

Once the index is built, searches automatically use it. You can control how many partitions to probe (higher = better recall, slower):

# ANN search — automatically uses IVF_PQ if index exists
results = vectorstore.similarity_search(
    "explain transformer architecture",
    k=10
)

# For the native LanceDB API with probe control:
table = db.open_table("knowledge_base")
embedding_query = embeddings.embed_query("explain transformer architecture")

results = table.search(embedding_query).metric("cosine").limit(10).nprobes(20).to_list()
# nprobes: number of IVF partitions to search (default 20; increase for better recall)

for row in results:
    print(f"Distance: {row['_distance']:.4f}")
    print(row.get('text', '')[:200])

The nprobes parameter trades recall for speed. For most RAG applications, the default of 20 is fine. If you're seeing recall degradation, bump it to 50–100.

LanceDB Cloud Mode

LanceDB Cloud stores the database on object storage (S3, GCS, or Azure Blob) instead of local disk. You interact with it through the same API — the storage layer is swapped transparently.

import lancedb
import os

# Connect to LanceDB Cloud
# Get your API key at lancedb.com
db = lancedb.connect(
    uri="db://your-project-name",
    api_key=os.getenv("LANCEDB_API_KEY"),
    region="us-east-1"  # or your region
)

# Everything else works identically to local mode
table = db.open_table("knowledge_base")
results = table.search(query_vector).limit(5).to_list()

For S3 self-hosted storage:

# S3 backend (no LanceDB Cloud account needed)
db = lancedb.connect(
    "s3://your-bucket/lance-db/",
    storage_options={
        "aws_access_key_id": os.getenv("AWS_ACCESS_KEY_ID"),
        "aws_secret_access_key": os.getenv("AWS_SECRET_ACCESS_KEY"),
        "region": "us-east-1"
    }
)

The S3 approach is excellent for teams — multiple machines can share the same LanceDB without a central server. The Lance format handles concurrent reads safely, though write coordination requires external locking for multi-writer scenarios.

Full RAG Pipeline with LanceDB

Here's a complete, deployable RAG example using LanceDB as the vector store:

import os
import lancedb
from pathlib import Path
from typing import List, Optional

from langchain_community.vectorstores import LanceDB
from langchain_community.document_loaders import DirectoryLoader, PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate
from langchain.schema import Document


class LanceDBRAGPipeline:
    """
    Complete RAG pipeline backed by LanceDB.
    Supports local and S3 storage modes.
    """
    
    INDEX_PARAMS = {
        "metric": "cosine",
        "num_partitions": 256,
        "num_sub_vectors": 96
    }
    
    def __init__(
        self,
        db_path: str = "./rag_lancedb",
        table_name: str = "documents",
        embedding_model: str = "text-embedding-3-small",
        llm_model: str = "gpt-4o",
        chunk_size: int = 500,
        chunk_overlap: int = 50
    ):
        self.table_name = table_name
        self.chunk_size = chunk_size
        self.chunk_overlap = chunk_overlap
        
        # Connect to LanceDB
        self.db = lancedb.connect(db_path)
        
        # Initialize models
        self.embeddings = OpenAIEmbeddings(model=embedding_model)
        self.llm = ChatOpenAI(model=llm_model, temperature=0)
        
        # Initialize vectorstore if table exists
        self.vectorstore = None
        if table_name in self.db.table_names():
            self.vectorstore = LanceDB(
                connection=self.db,
                embedding=self.embeddings,
                table_name=table_name
            )
    
    def ingest_documents(
        self, 
        source_path: str,
        file_glob: str = "**/*.txt",
        rebuild_index: bool = True
    ) -> int:
        """Load, chunk, and store documents."""
        
        # Load documents
        loader = DirectoryLoader(
            source_path,
            glob=file_glob,
            show_progress=True
        )
        documents = loader.load()
        print(f"Loaded {len(documents)} documents")
        
        # Split into chunks
        splitter = RecursiveCharacterTextSplitter(
            chunk_size=self.chunk_size,
            chunk_overlap=self.chunk_overlap,
            separators=["\n\n", "\n", ". ", " ", ""]
        )
        chunks = splitter.split_documents(documents)
        print(f"Created {len(chunks)} chunks")
        
        # Add metadata
        for i, chunk in enumerate(chunks):
            chunk.metadata["chunk_id"] = i
            chunk.metadata["chunk_size"] = len(chunk.page_content)
        
        # Store in LanceDB
        if self.table_name in self.db.table_names():
            self.vectorstore = LanceDB(
                connection=self.db,
                embedding=self.embeddings,
                table_name=self.table_name
            )
            self.vectorstore.add_documents(chunks)
        else:
            self.vectorstore = LanceDB.from_documents(
                documents=chunks,
                embedding=self.embeddings,
                connection=self.db,
                table_name=self.table_name
            )
        
        # Build ANN index for large datasets
        if rebuild_index and len(chunks) > 10_000:
            print("Building IVF_PQ index...")
            table = self.db.open_table(self.table_name)
            table.create_index(**self.INDEX_PARAMS)
            print("Index built successfully")
        
        return len(chunks)
    
    def build_qa_chain(self) -> RetrievalQA:
        """Build a RetrievalQA chain with the LanceDB retriever."""
        
        if self.vectorstore is None:
            raise ValueError(
                "No documents ingested yet. Call ingest_documents() first."
            )
        
        retriever = self.vectorstore.as_retriever(
            search_type="similarity",
            search_kwargs={"k": 5}
        )
        
        prompt_template = """You are a helpful assistant. Use only the context
below to answer the question. If the answer is not in the context,
say "I don't have that information."

Context:
{context}

Question: {question}

Answer (be concise and cite the source when possible):"""
        
        prompt = PromptTemplate(
            template=prompt_template,
            input_variables=["context", "question"]
        )
        
        chain = RetrievalQA.from_chain_type(
            llm=self.llm,
            chain_type="stuff",
            retriever=retriever,
            chain_type_kwargs={"prompt": prompt},
            return_source_documents=True
        )
        
        return chain
    
    def query(self, question: str) -> dict:
        """Run a query against the RAG pipeline."""
        chain = self.build_qa_chain()
        response = chain.invoke({"query": question})
        
        return {
            "answer": response["result"],
            "sources": [
                {
                    "content": doc.page_content[:200],
                    "source": doc.metadata.get("source", "unknown"),
                    "chunk_id": doc.metadata.get("chunk_id")
                }
                for doc in response["source_documents"]
            ]
        }
    
    def get_table_stats(self) -> dict:
        """Get statistics about the stored data."""
        if self.table_name not in self.db.table_names():
            return {"error": "Table not found"}
        
        table = self.db.open_table(self.table_name)
        return {
            "num_rows": table.count_rows(),
            "schema": str(table.schema),
            "versions": len(table.list_versions())
        }


# --- Run it ---

if __name__ == "__main__":
    pipeline = LanceDBRAGPipeline(
        db_path="./production_lancedb",
        table_name="company_docs",
        embedding_model="text-embedding-3-small",
        llm_model="gpt-4o",
        chunk_size=400,
        chunk_overlap=40
    )
    
    # Ingest documents
    num_chunks = pipeline.ingest_documents(
        source_path="./docs/",
        file_glob="**/*.txt"
    )
    print(f"Ingested {num_chunks} chunks")
    
    # Print stats
    stats = pipeline.get_table_stats()
    print(f"Table stats: {stats}")
    
    # Query
    result = pipeline.query("What is our refund policy?")
    print("\nAnswer:", result["answer"])
    print("\nSources:")
    for source in result["sources"]:
        print(f"  - {source['source']}: {source['content'][:100]}...")

This pipeline handles the full lifecycle: loading documents from a directory, chunking, storing in LanceDB with metadata, building an ANN index when the dataset is large enough, and running queries through a RetrievalQA chain.

Version Control and Data Management

One feature I haven't seen highlighted enough: LanceDB's built-in versioning. Every write creates a new version you can inspect and restore.

table = db.open_table("documents")

# List all versions
versions = table.list_versions()
for v in versions[-5:]:  # Last 5 versions
    print(f"Version {v['version']}: {v['timestamp']}, rows={v['metadata']}")

# Restore to a previous version (time travel)
table.restore(version=3)

# Or query a specific version without restoring
old_table = table.checkout(version=2)
old_results = old_table.search(query_vector).limit(5).to_list()

This is genuinely useful in production. If a data ingestion job pushes bad chunks, you can roll back immediately without rebuilding from scratch.

For a broader view of where LanceDB fits in agent architectures, the post on building AI agents with LangChain walks through several storage patterns.

Integrating with LangChain's Retriever Interface

Beyond RetrievalQA, you can use LanceDB as a retriever in any LangChain chain or agent:

from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain.tools.retriever import create_retriever_tool
from langchain_openai import ChatOpenAI
from langchain import hub

# Get retriever from vectorstore
retriever = vectorstore.as_retriever(
    search_type="mmr",
    search_kwargs={"k": 5, "fetch_k": 20}
)

# Wrap as a tool for agents
retriever_tool = create_retriever_tool(
    retriever,
    name="knowledge_base_search",
    description="Search the company knowledge base for product, policy, and technical documentation."
)

# Build agent
llm = ChatOpenAI(model="gpt-4o", temperature=0)
prompt = hub.pull("hwchase17/openai-tools-agent")

agent = create_openai_tools_agent(llm, [retriever_tool], prompt)
agent_executor = AgentExecutor(agent=agent, tools=[retriever_tool], verbose=True)

response = agent_executor.invoke({
    "input": "What are the API rate limits and how do I handle 429 errors?"
})
print(response["output"])

This pattern — wrapping a LanceDB retriever as an agent tool — is the foundation for the AI research agent build pattern. The agent decides when to search the knowledge base versus answering from its own knowledge.

Performance Tips

Pre-compute embeddings in batches. For large ingestion jobs, embedding in batches of 100–500 documents at a time is much faster than one at a time.

from langchain_openai import OpenAIEmbeddings
from langchain.schema import Document
import numpy as np

def batch_embed_documents(
    docs: List[Document],
    embeddings: OpenAIEmbeddings,
    batch_size: int = 200
) -> List[List[float]]:
    """Embed documents in batches for efficiency."""
    texts = [doc.page_content for doc in docs]
    all_embeddings = []
    
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i + batch_size]
        batch_embeddings = embeddings.embed_documents(batch)
        all_embeddings.extend(batch_embeddings)
        print(f"Embedded {min(i + batch_size, len(texts))}/{len(texts)} chunks")
    
    return all_embeddings

Use text-embedding-3-small over ada-002. It's cheaper, faster, and slightly better on most retrieval benchmarks. For 1536-dim embeddings, the storage overhead is identical.

Compact the table periodically. LanceDB accumulates small delta files over time. Compacting merges them for faster reads.

table = db.open_table("documents")
table.compact_files()  # Merge delta files
table.cleanup_old_versions(older_than=timedelta(days=7))

For deploying RAG systems backed by LanceDB in production, the deploy AI model to production post covers containerization and scaling considerations.

Conclusion

LanceDB fills a real gap in the open-source vector database space. It's fast, disk-efficient, and runs without any server infrastructure — but it scales to large datasets in a way that FAISS and Chroma can't match easily. The LangChain integration is mature enough that you can drop it in as a replacement for any other vector store with minimal code changes.

The IVF_PQ indexing makes a genuine difference at scale — you go from 120ms+ brute-force queries to sub-10ms ANN search without sacrificing much recall. Combined with the built-in versioning and cloud sync options, LanceDB is one of the most complete open-source options for production RAG systems.

If you're starting a new project, I'd recommend LanceDB as the default local vector store over Chroma for anything beyond a simple prototype. The learning curve is minimal, the performance ceiling is much higher, and the LangChain integration handles all the boilerplate.

Next step: check out the RAG system tutorial for a complete end-to-end walkthrough that builds on everything covered here.

FAQs

Is LanceDB free to use? Yes, LanceDB is fully open source under the Apache 2.0 license. The local mode is completely free with no limits. LanceDB Cloud (lancedb.com) offers a hosted version with a free tier and pay-as-you-go pricing for larger workloads.

How does LanceDB compare to Chroma for local development? LanceDB stores data in the Lance columnar format, which is more compact than Chroma's SQLite-backed storage and faster for ANN queries on large datasets. Chroma has a slightly simpler API for very small projects, but LanceDB's disk efficiency and query speed make it the better choice once you exceed a few thousand vectors.

Can LanceDB handle multimodal embeddings? Yes. LanceDB supports storing any fixed-dimension float array as a vector, so you can store text, image, audio, or multimodal embeddings in the same table. The Lance format natively handles mixed-type columns including nested arrays and structs.

For context on other vector storage options, the vector database guide covers the broader landscape including Pinecone, Weaviate, and Qdrant.

Why LanceDB Is Worth Your Attention

A few things stand out:

No server process. LanceDB runs in-process, just like FAISS or Chroma. No Docker, no daemon, no port management.

Lance columnar format. Lance uses a columnar storage format optimized for random access on large embedding arrays. Reads are 3–10× faster than row-oriented formats for vector workloads.

Native versioning. Every write to a LanceDB table creates a new version. You can roll back to any previous state without extra tooling — genuinely useful for production data pipelines.

Multimodal support. Text, images, audio — LanceDB handles any fixed-dimension vector. You can store multiple vector columns in the same table.

Comparison: LanceDB vs Chroma vs SQLite-VSS

Feature	LanceDB	Chroma	SQLite-VSS
Disk usage (1M vectors, 1536-dim)	~6 GB	~14 GB	~18 GB
ANN query speed (p95)	8ms	45ms	120ms
Python ergonomics	Excellent	Excellent	Moderate
Cloud sync	S3/GCS/Azure	No	No
Free / open source	Yes (Apache 2.0)	Yes (Apache 2.0)	Yes (MIT)
Versioning / rollback	Built-in	No	No
Multimodal columns	Yes	No	No
Filtering on metadata	SQL-like	Dict filter	SQL

Installation

pip install lancedb langchain-community langchain-openai
# Optional: cloud sync support
pip install lancedb[cloud]

LanceDB requires Python 3.8+ and uses pyarrow under the hood for the columnar format. That gets installed automatically with lancedb.

Local Setup: First Steps

The simplest possible LanceDB setup stores everything in a local directory:

import lancedb
import os

# Create (or connect to) a local database
db = lancedb.connect("./my_lance_db")

# List existing tables
print("Existing tables:", db.table_names())

LangChain Integration: Storing Embeddings

LangChain's LanceDB vector store class wraps the native client. Here's the standard setup pattern:

from langchain_community.vectorstores import LanceDB
from langchain_openai import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import TextLoader
import lancedb

# Load and chunk documents
loader = TextLoader("./knowledge_base.txt")
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = splitter.split_documents(documents)

# Connect to LanceDB
db = lancedb.connect("./rag_lance_db")

# Initialize embeddings
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Create vector store from documents
vectorstore = LanceDB.from_documents(
    documents=chunks,
    embedding=embeddings,
    connection=db,
    table_name="knowledge_base"
)

print(f"Stored {len(chunks)} chunks in LanceDB")

If the table already exists and you want to add more documents:

# Open existing table
vectorstore = LanceDB(
    connection=db,
    embedding=embeddings,
    table_name="knowledge_base"
)

# Add new documents
new_docs = [...]  # Your new Document objects
vectorstore.add_documents(new_docs)

For more on chunking strategies that pair well with LanceDB, the LangChain tutorial 2025 covers splitting approaches in depth.

Similarity Search

Basic similarity search works the same as any LangChain vector store:

# Simple similarity search
results = vectorstore.similarity_search(
    "How does the authentication system work?",
    k=5
)

for doc in results:
    print(f"Score source: {doc.metadata.get('source', 'unknown')}")
    print(doc.page_content[:200])
    print("---")

# Search with scores
results_with_scores = vectorstore.similarity_search_with_score(
    "token rate limits",
    k=5
)

for doc, score in results_with_scores:
    print(f"Distance: {score:.4f}")
    print(doc.page_content[:150])

Lower distance scores mean higher relevance in LanceDB's default cosine distance metric.

Metadata Filtering

LanceDB supports SQL-like filter expressions, which is more expressive than Chroma's dictionary-based filtering:

from langchain_community.vectorstores import LanceDB

# Search with metadata filter
results = vectorstore.similarity_search(
    "authentication flow",
    k=5,
    filter="source = 'auth_docs.txt' AND created_year >= 2024"
)

# Filter by category
results = vectorstore.similarity_search(
    "rate limiting",
    k=5,
    filter="category IN ('api', 'networking')"
)

This SQL-like syntax covers most production filtering needs without the verbosity of some other APIs. Filters are applied at the storage layer before the ANN search, so they're fast.

Building the IVF_PQ Index

IVF_PQ works in two stages:

IVF (Inverted File): Clusters vectors into num_partitions groups. At query time, only the nearest clusters are searched.
PQ (Product Quantization): Compresses vectors by splitting them into sub-vectors and approximating each with a codebook. This reduces memory usage dramatically.

import lancedb
import numpy as np

# Connect and open table
db = lancedb.connect("./rag_lance_db")
table = db.open_table("knowledge_base")

# Build IVF_PQ index
# num_partitions: number of IVF clusters (rule of thumb: sqrt(num_rows))
# num_sub_vectors: PQ compression (higher = less compression but better recall)
table.create_index(
    metric="cosine",
    num_partitions=256,       # Clusters; increase for larger datasets
    num_sub_vectors=96,       # Sub-vectors for PQ; must divide embedding dim
    vector_column_name="vector"
)

print("IVF_PQ index created successfully")

Some rules of thumb for num_partitions:

Under 100k vectors: 64–128
100k–1M vectors: 256–1024
Over 1M vectors: 1024–4096

# Verify index was created
print(table.schema)

# Check index info
indices = table.list_indices()
for idx in indices:
    print(f"Index: {idx}")

ANN Search with the Index

Once the index is built, searches automatically use it. You can control how many partitions to probe (higher = better recall, slower):

# ANN search — automatically uses IVF_PQ if index exists
results = vectorstore.similarity_search(
    "explain transformer architecture",
    k=10
)

# For the native LanceDB API with probe control:
table = db.open_table("knowledge_base")
embedding_query = embeddings.embed_query("explain transformer architecture")

results = table.search(embedding_query).metric("cosine").limit(10).nprobes(20).to_list()
# nprobes: number of IVF partitions to search (default 20; increase for better recall)

for row in results:
    print(f"Distance: {row['_distance']:.4f}")
    print(row.get('text', '')[:200])

The nprobes parameter trades recall for speed. For most RAG applications, the default of 20 is fine. If you're seeing recall degradation, bump it to 50–100.

LanceDB Cloud Mode

LanceDB Cloud stores the database on object storage (S3, GCS, or Azure Blob) instead of local disk. You interact with it through the same API — the storage layer is swapped transparently.

import lancedb
import os

# Connect to LanceDB Cloud
# Get your API key at lancedb.com
db = lancedb.connect(
    uri="db://your-project-name",
    api_key=os.getenv("LANCEDB_API_KEY"),
    region="us-east-1"  # or your region
)

# Everything else works identically to local mode
table = db.open_table("knowledge_base")
results = table.search(query_vector).limit(5).to_list()

For S3 self-hosted storage:

# S3 backend (no LanceDB Cloud account needed)
db = lancedb.connect(
    "s3://your-bucket/lance-db/",
    storage_options={
        "aws_access_key_id": os.getenv("AWS_ACCESS_KEY_ID"),
        "aws_secret_access_key": os.getenv("AWS_SECRET_ACCESS_KEY"),
        "region": "us-east-1"
    }
)

Full RAG Pipeline with LanceDB

Here's a complete, deployable RAG example using LanceDB as the vector store:

import os
import lancedb
from pathlib import Path
from typing import List, Optional

from langchain_community.vectorstores import LanceDB
from langchain_community.document_loaders import DirectoryLoader, PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate
from langchain.schema import Document


class LanceDBRAGPipeline:
    """
    Complete RAG pipeline backed by LanceDB.
    Supports local and S3 storage modes.
    """
    
    INDEX_PARAMS = {
        "metric": "cosine",
        "num_partitions": 256,
        "num_sub_vectors": 96
    }
    
    def __init__(
        self,
        db_path: str = "./rag_lancedb",
        table_name: str = "documents",
        embedding_model: str = "text-embedding-3-small",
        llm_model: str = "gpt-4o",
        chunk_size: int = 500,
        chunk_overlap: int = 50
    ):
        self.table_name = table_name
        self.chunk_size = chunk_size
        self.chunk_overlap = chunk_overlap
        
        # Connect to LanceDB
        self.db = lancedb.connect(db_path)
        
        # Initialize models
        self.embeddings = OpenAIEmbeddings(model=embedding_model)
        self.llm = ChatOpenAI(model=llm_model, temperature=0)
        
        # Initialize vectorstore if table exists
        self.vectorstore = None
        if table_name in self.db.table_names():
            self.vectorstore = LanceDB(
                connection=self.db,
                embedding=self.embeddings,
                table_name=table_name
            )
    
    def ingest_documents(
        self, 
        source_path: str,
        file_glob: str = "**/*.txt",
        rebuild_index: bool = True
    ) -> int:
        """Load, chunk, and store documents."""
        
        # Load documents
        loader = DirectoryLoader(
            source_path,
            glob=file_glob,
            show_progress=True
        )
        documents = loader.load()
        print(f"Loaded {len(documents)} documents")
        
        # Split into chunks
        splitter = RecursiveCharacterTextSplitter(
            chunk_size=self.chunk_size,
            chunk_overlap=self.chunk_overlap,
            separators=["\n\n", "\n", ". ", " ", ""]
        )
        chunks = splitter.split_documents(documents)
        print(f"Created {len(chunks)} chunks")
        
        # Add metadata
        for i, chunk in enumerate(chunks):
            chunk.metadata["chunk_id"] = i
            chunk.metadata["chunk_size"] = len(chunk.page_content)
        
        # Store in LanceDB
        if self.table_name in self.db.table_names():
            self.vectorstore = LanceDB(
                connection=self.db,
                embedding=self.embeddings,
                table_name=self.table_name
            )
            self.vectorstore.add_documents(chunks)
        else:
            self.vectorstore = LanceDB.from_documents(
                documents=chunks,
                embedding=self.embeddings,
                connection=self.db,
                table_name=self.table_name
            )
        
        # Build ANN index for large datasets
        if rebuild_index and len(chunks) > 10_000:
            print("Building IVF_PQ index...")
            table = self.db.open_table(self.table_name)
            table.create_index(**self.INDEX_PARAMS)
            print("Index built successfully")
        
        return len(chunks)
    
    def build_qa_chain(self) -> RetrievalQA:
        """Build a RetrievalQA chain with the LanceDB retriever."""
        
        if self.vectorstore is None:
            raise ValueError(
                "No documents ingested yet. Call ingest_documents() first."
            )
        
        retriever = self.vectorstore.as_retriever(
            search_type="similarity",
            search_kwargs={"k": 5}
        )
        
        prompt_template = """You are a helpful assistant. Use only the context
below to answer the question. If the answer is not in the context,
say "I don't have that information."

Context:
{context}

Question: {question}

Answer (be concise and cite the source when possible):"""
        
        prompt = PromptTemplate(
            template=prompt_template,
            input_variables=["context", "question"]
        )
        
        chain = RetrievalQA.from_chain_type(
            llm=self.llm,
            chain_type="stuff",
            retriever=retriever,
            chain_type_kwargs={"prompt": prompt},
            return_source_documents=True
        )
        
        return chain
    
    def query(self, question: str) -> dict:
        """Run a query against the RAG pipeline."""
        chain = self.build_qa_chain()
        response = chain.invoke({"query": question})
        
        return {
            "answer": response["result"],
            "sources": [
                {
                    "content": doc.page_content[:200],
                    "source": doc.metadata.get("source", "unknown"),
                    "chunk_id": doc.metadata.get("chunk_id")
                }
                for doc in response["source_documents"]
            ]
        }
    
    def get_table_stats(self) -> dict:
        """Get statistics about the stored data."""
        if self.table_name not in self.db.table_names():
            return {"error": "Table not found"}
        
        table = self.db.open_table(self.table_name)
        return {
            "num_rows": table.count_rows(),
            "schema": str(table.schema),
            "versions": len(table.list_versions())
        }


# --- Run it ---

if __name__ == "__main__":
    pipeline = LanceDBRAGPipeline(
        db_path="./production_lancedb",
        table_name="company_docs",
        embedding_model="text-embedding-3-small",
        llm_model="gpt-4o",
        chunk_size=400,
        chunk_overlap=40
    )
    
    # Ingest documents
    num_chunks = pipeline.ingest_documents(
        source_path="./docs/",
        file_glob="**/*.txt"
    )
    print(f"Ingested {num_chunks} chunks")
    
    # Print stats
    stats = pipeline.get_table_stats()
    print(f"Table stats: {stats}")
    
    # Query
    result = pipeline.query("What is our refund policy?")
    print("\nAnswer:", result["answer"])
    print("\nSources:")
    for source in result["sources"]:
        print(f"  - {source['source']}: {source['content'][:100]}...")

Version Control and Data Management

One feature I haven't seen highlighted enough: LanceDB's built-in versioning. Every write creates a new version you can inspect and restore.

table = db.open_table("documents")

# List all versions
versions = table.list_versions()
for v in versions[-5:]:  # Last 5 versions
    print(f"Version {v['version']}: {v['timestamp']}, rows={v['metadata']}")

# Restore to a previous version (time travel)
table.restore(version=3)

# Or query a specific version without restoring
old_table = table.checkout(version=2)
old_results = old_table.search(query_vector).limit(5).to_list()

This is genuinely useful in production. If a data ingestion job pushes bad chunks, you can roll back immediately without rebuilding from scratch.

For a broader view of where LanceDB fits in agent architectures, the post on building AI agents with LangChain walks through several storage patterns.

Integrating with LangChain's Retriever Interface

Beyond RetrievalQA, you can use LanceDB as a retriever in any LangChain chain or agent:

from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain.tools.retriever import create_retriever_tool
from langchain_openai import ChatOpenAI
from langchain import hub

# Get retriever from vectorstore
retriever = vectorstore.as_retriever(
    search_type="mmr",
    search_kwargs={"k": 5, "fetch_k": 20}
)

# Wrap as a tool for agents
retriever_tool = create_retriever_tool(
    retriever,
    name="knowledge_base_search",
    description="Search the company knowledge base for product, policy, and technical documentation."
)

# Build agent
llm = ChatOpenAI(model="gpt-4o", temperature=0)
prompt = hub.pull("hwchase17/openai-tools-agent")

agent = create_openai_tools_agent(llm, [retriever_tool], prompt)
agent_executor = AgentExecutor(agent=agent, tools=[retriever_tool], verbose=True)

response = agent_executor.invoke({
    "input": "What are the API rate limits and how do I handle 429 errors?"
})
print(response["output"])

Performance Tips

Pre-compute embeddings in batches. For large ingestion jobs, embedding in batches of 100–500 documents at a time is much faster than one at a time.

from langchain_openai import OpenAIEmbeddings
from langchain.schema import Document
import numpy as np

def batch_embed_documents(
    docs: List[Document],
    embeddings: OpenAIEmbeddings,
    batch_size: int = 200
) -> List[List[float]]:
    """Embed documents in batches for efficiency."""
    texts = [doc.page_content for doc in docs]
    all_embeddings = []
    
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i + batch_size]
        batch_embeddings = embeddings.embed_documents(batch)
        all_embeddings.extend(batch_embeddings)
        print(f"Embedded {min(i + batch_size, len(texts))}/{len(texts)} chunks")
    
    return all_embeddings

Use text-embedding-3-small over ada-002. It's cheaper, faster, and slightly better on most retrieval benchmarks. For 1536-dim embeddings, the storage overhead is identical.

Compact the table periodically. LanceDB accumulates small delta files over time. Compacting merges them for faster reads.

table = db.open_table("documents")
table.compact_files()  # Merge delta files
table.cleanup_old_versions(older_than=timedelta(days=7))

For deploying RAG systems backed by LanceDB in production, the deploy AI model to production post covers containerization and scaling considerations.

Conclusion

Next step: check out the RAG system tutorial for a complete end-to-end walkthrough that builds on everything covered here.

How to Use LangChain with LanceDB (Serverless Embeddings)

Why LanceDB Is Worth Your Attention

Comparison: LanceDB vs Chroma vs SQLite-VSS

Installation

Local Setup: First Steps

LangChain Integration: Storing Embeddings

Similarity Search

Metadata Filtering

Building the IVF_PQ Index

ANN Search with the Index

LanceDB Cloud Mode

Full RAG Pipeline with LanceDB

Version Control and Data Management

Integrating with LangChain's Retriever Interface

Performance Tips

Conclusion

FAQs

Frequently Asked Questions

AiTechWorlds Team

Related Articles

AutoGen vs LangChain: Which for Multi-Agent Systems in 2026?

How to Use AutoGen with Milvus (Vector Database Memory)

5 AutoGPT Memory Types (Vector, Redis, File, Conversation)

How to Set Up AutoGPT with Pinecone (Persistent Memory)

Get Free AI Notes Daily

How to Use LangChain with LanceDB (Serverless Embeddings)

Why LanceDB Is Worth Your Attention

Comparison: LanceDB vs Chroma vs SQLite-VSS

Installation

Local Setup: First Steps

LangChain Integration: Storing Embeddings

Similarity Search

Metadata Filtering

Building the IVF_PQ Index

ANN Search with the Index

LanceDB Cloud Mode

Full RAG Pipeline with LanceDB

Version Control and Data Management

Integrating with LangChain's Retriever Interface

Performance Tips

Conclusion

FAQs

Frequently Asked Questions

AiTechWorlds Team

Related Articles

AutoGen vs LangChain: Which for Multi-Agent Systems in 2026?

How to Use AutoGen with Milvus (Vector Database Memory)

5 AutoGPT Memory Types (Vector, Redis, File, Conversation)

How to Set Up AutoGPT with Pinecone (Persistent Memory)

Get Free AI Notes Daily