Naveen Singh

AI

Building a Local Retrieval-Augmented Generation (RAG) System on MacBook with Ollama and LangChain

Local RAG with Ollama & LangChain

Navdeep SinghOctober 10, 20268 min read
#ai#rag#ollama#langchain#chromadb#python#macbook
Text size:M

Building a Local Retrieval-Augmented Generation (RAG) System on MacBook with Ollama and LangChain

Running a Retrieval-Augmented Generation (RAG) pipeline locally ensures complete data privacy, zero API costs, and full control over your AI stack. When working with sensitive enterprise documents, internal notes, or proprietary knowledge bases, sending raw data to third-party cloud LLM APIs poses compliance and privacy risks.

By running everything on your local machine—leveraging Apple Silicon's unified memory architecture or modern Intel Mac hardware—you can query private documents with low latency, offline availability, and zero recurring token bills.

This guide walks through setting up a lightweight, end-to-end local RAG system on a MacBook using Python, LangChain, Chroma, Hugging Face embeddings, and Ollama.

1. Architectural Overview

A local RAG system bridges private documents with an offline Large Language Model (LLM) through semantic search:

┌──────────────────┐
│  Raw Documents   │ (sample.txt)
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│ Text Splitter    │ RecursiveCharacterTextSplitter (chunks + overlap)
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│ Embedding Model  │ Hugging Face (all-MiniLM-L6-v2)
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│ Vector Database  │ Chroma Vector Store (Local In-Memory / File-based)
└────────┬─────────┘
         │
         │ (Semantic Similarity Search)
         ▼
┌────────────────────────────────────────────────────────┐
│ Query Engine & Context Injection                      │
│ - User Query: "Why are MacBook Air M-series silent?"  │
│ - Retrieved Context: Relevant chunks from Chroma       │
└──────────────────────────┬─────────────────────────────┘
                           │
                           ▼
                 ┌──────────────────┐
                 │ Local LLM        │ Ollama (gemma4:e4b / llama3)
                 └─────────┬────────┘
                           │
                           ▼
                 ┌──────────────────┐
                 │ Grounded Answer  │ High accuracy, no hallucination
                 └──────────────────┘

The workflow breaks down into four fundamental stages:

  1. Document Loading & Chunking: Raw documents are ingested and sliced into manageable pieces (chunks) with overlapping boundaries to preserve context across splits.
  2. Local Embeddings: Text chunks are converted into numerical vector representations using a local Hugging Face transformer model (all-MiniLM-L6-v2). No external calls are made.
  3. Vector Database: Vectors and metadata are indexed and stored locally in Chroma, a lightweight, file-based embedded vector store.
  4. Retrieval & Generation: When a query is submitted, the system performs a semantic similarity search to fetch the most relevant chunk, injects it into a prompt template, and passes it to Ollama running an offline LLM (such as gemma4:e4b or llama3) to generate a grounded, hallucination-free response.

2. Environment Setup & Step-by-Step Installation

Step 1: Install & Start Ollama

Ollama allows you to run open-source LLMs locally with optimized hardware acceleration (including Apple Silicon Metal support).

Install Ollama via Homebrew:

brew install ollama

Start the Ollama server:

ollama serve

Note: Keep this terminal window running, or launch Ollama as a macOS background application from your Applications folder.

In a separate terminal, pull your preferred local model (e.g., gemma4:e4b or llama3):

ollama pull gemma4:e4b

You can confirm that the model was downloaded successfully by listing all available models:

ollama list

Step 2: Set Up Python Virtual Environment

Create an isolated directory and virtual environment for your project:

mkdir local-rag-mac && cd local-rag-mac
python3 -m venv venv
source venv/bin/activate

Always ensure your virtual environment is active before installing packages or running scripts.

Step 3: Install Required Dependencies

Install the modern core libraries for LangChain, vector storage, and local embeddings:

pip install langchain-classic langchain-core langchain-text-splitters langchain-ollama langchain-huggingface langchain-chroma chromadb sentence-transformers

Here is a breakdown of what each package does:

  • langchain-classic & langchain-core: Foundation abstractions, chains, and prompt templates.
  • langchain-text-splitters: Chunking utilities to break raw text into semantically coherent segments.
  • langchain-ollama: Official LangChain integration to interface directly with the local Ollama daemon.
  • langchain-huggingface & sentence-transformers: Local embedding generation without cloud API dependencies.
  • langchain-chroma & chromadb: Embedded vector database for local storage and nearest-neighbor search.

Step 4: Create a Sample Document

Create a local text file named sample.txt in your project folder for your RAG system to learn from:

echo "MacBook Air with M-series chips is a line of laptops introduced by Apple. They are known for high energy efficiency and a fanless design that allows for completely silent operation." > sample.txt

3. Implementation Code (app.py)

Create a script named app.py in your project folder (local-rag-mac/app.py):

from langchain_classic.chains import create_retrieval_chain
from langchain_classic.chains.combine_documents import create_stuff_documents_chain
from langchain_community.document_loaders import TextLoader
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_ollama import OllamaLLM
from langchain_chroma import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_text_splitters import RecursiveCharacterTextSplitter
 
# 1. Load your local document
print("Loading document...")
loader = TextLoader("sample.txt")
docs = loader.load()
 
# 2. Split the document into small chunks
print("Chunking document...")
text_splitter = RecursiveCharacterTextSplitter(chunk_size=250, chunk_overlap=30)
chunks = text_splitter.split_documents(docs)
 
# 3. Create local embeddings and store in Chroma vector database
print("Generating embeddings & building vector database...")
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = Chroma.from_documents(chunks, embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 1})
 
# 4. Connect to your local Ollama LLM using the modern class
print("Connecting to local Ollama (gemma4:e4b)...")
llm = OllamaLLM(model="gemma4:e4b")
 
# 5. Set up the RAG prompt template and chain
system_prompt = (
    "You are a helpful study assistant. Use the following context "
    "to answer the question. If you don't know the answer, say "
    "that you don't know.\n\n"
    "{context}"
)
prompt = ChatPromptTemplate.from_messages([
    ("system", system_prompt),
    ("human", "{input}"),
])
 
question_answer_chain = create_stuff_documents_chain(llm, prompt)
rag_chain = create_retrieval_chain(retriever, question_answer_chain)
 
# 6. Ask a question!
query = "Why are MacBook Air M-series laptops silent?"
print(f"\nAsking Question: '{query}'\n")
 
response = rag_chain.invoke({"input": query})
 
print("--- Answer from Local RAG ---")
print(response["answer"])

Running the Script

Execute the script with your active virtual environment:

python3 app.py

Expected Output

Loading document...
Chunking document...
Generating embeddings & building vector database...
Connecting to local Ollama (gemma4:e4b)...
 
Asking Question: 'Why are MacBook Air M-series laptops silent?'
 
--- Answer from Local RAG ---
MacBook Air M-series laptops are silent because of their fanless design, made possible by the high energy efficiency of Apple's M-series chips.

4. Common Problems and Troubleshooting Guide

Issue 1: OllamaEndpointNotFoundError (Status Code 404)

  • Symptom: The script crashes during execution with a 404 or connection error when attempting to communicate with Ollama.
  • Root Cause: The Ollama background service is not running, or the model name specified in app.py does not match any downloaded model on your system.
  • Solution:
    1. Verify Ollama is running: 
      ollama serve
    2. Check installed models via: 
      ollama list
    3. Ensure the model name in your script exactly matches the installed tag (e.g., gemma4:e4b, llama3:latest, or mistral).

Issue 2: ModuleNotFoundError: No module named 'langchain.chains'

  • Symptom: Python throws an import error pointing to missing core chains when executing from langchain.chains import ....
  • Root Cause: In modern LangChain v1.0+ releases, modular components like retrieval chains have been isolated or relocated into langchain-classic and langchain-core.
  • Solution: Install langchain-classic and import the retrieval chains from langchain_classic.chains:
pip install langchain-classic
from langchain_classic.chains import create_retrieval_chain
from langchain_classic.chains.combine_documents import create_stuff_documents_chain

Issue 3: Package Dependency Conflicts with pip

  • Symptom: Incompatible version constraints between langchain-core, text splitters, and community packages.
  • Root Cause: Mixing legacy monolithic langchain packages with modular integration packages.
  • Solution: Cleanly reinstall the modern stack by clearing conflicts and setting up the recommended package grouping:
pip uninstall -y langchain langchain-community langchain-classic
pip install langchain-classic langchain-core langchain-text-splitters langchain-ollama langchain-huggingface langchain-chroma chromadb sentence-transformers

Issue 4: File Path Error (can't open file .../venv/app.py)

  • Symptom: Python complains it cannot find app.py because it references the venv/ directory path.
  • Root Cause: Saving the Python script directly inside the virtual environment directory instead of the root project workspace.
  • Solution: Ensure your terminal working directory is the base project folder (cd ~/local-rag-mac), and keep app.py outside of the venv directory:
local-rag-mac/
├── app.py
├── sample.txt
└── venv/

5. Summary & Best Practices for Apple Silicon

  • Metal Acceleration: Ollama automatically offloads layers to the GPU using Apple Silicon's unified memory (MPS). Ensure your Mac has sufficient unified RAM for larger models (e.g., 8 GB RAM handles 3B–7B models comfortably; 16 GB+ is recommended for 8B–14B models).
  • Persistent Storage: In this guide, Chroma is initialized in-memory for simplicity. For production, specify a persist_directory="./chroma_db" to keep embeddings cached on disk across runs.
  • Chunk Size Tuning: For technical documents and code, tune chunk_size (e.g., 500–1000 characters) and chunk_overlap (e.g., 50–100 characters) to avoid splitting essential contextual paragraphs mid-thought.