Local AI Engineering with Ollama
Run, understand, customize, fine-tune, and build agentic apps on your own hardware
Available as: Online course, Kindle, Paperback
What You'll Learn
- Tell whether a model fits your RAM or VRAM before you pull it
- Choose between Q4_K_M, Q5_K_M, Q8_0 and the rest with the tradeoffs in view
- Drive Ollama over HTTP and compare models on tokens per second, not vibes
- Size the context window deliberately and see exactly what is sent each turn
- Package a custom model with a Modelfile a teammate can pull and run
- Fine-tune with QLoRA and Unsloth, then export to GGUF to run it in Ollama
- Grow a chat loop into a tool-using agent with the Python SDK
About This Book
A course about owning the model that runs instead of renting it. It covers enough of what a model actually does — tokens, weights, embeddings, attention, the KV cache — to make decisions, with every concept tied to a setting you later change.
The rest is hands-on: install Ollama, judge whether a model fits your RAM or VRAM before you download it, read GGUF quantizations, drive the HTTP API, take control of the context window, and tune temperature, top_p, penalties, keep-alive and concurrency. You then package a custom model with a Modelfile, fine-tune Granite for text-to-SQL using QLoRA with Unsloth and export the result to GGUF, and build a chat application in nine passes until it becomes a tool-using agent. Every command was run on a real machine and its actual output recorded.
Table of Contents
-
Local AI Engineering with Ollama
- Why This Book Exists
- What You Will Learn
- Who Is This Book For?
- About the Author
-
How to Get the Most Out of This Book
- What This Book Asks of You
- Conventions
-
What's the Point of Local AI?
- Should You Run AI Locally, or Just Use an API?
- Why People Run Local Even When the API Is Cheaper
- A Sensible Default
-
What Is Ollama?
- What It Actually Solves
- Why Local, and Why Now
- What Ollama Is, and What It Is Not
-
Core Concepts: From Tokens and Embeddings to Quantization and KV Cache
- What Is a Token?
- Embeddings
- What Is a Neural Network?
- What Is a Weight?
- What Are Inference Parameters?
- What Is GGUF and Why Does It Exist?
- Quantization: Trade Precision You Don't Need for Memory You Do
- Transformer Models
- The KV Cache
-
Requirements and Setup
- Installing Ollama
-
Picking and Pulling Models
- Understanding What You Can Run on Your System
- Pulling Models
- Understanding How Models Are Stored
- Where to Find Models
- Reading Ollama's Model Library
-
Running Models and Understanding How They Work inside Ollama
- Running a Model
- One-Shot Mode
- Running Models with the Ollama API
- Ollama Conversation Flow
-
The Context Window
- How a Model "Remembers" with Ollama
- What Actually Goes to the Model
- The Context Window: num_ctx
- Silent Truncation: The Trap
-
Controlling and Tuning Model Behavior at Runtime
- The /set Command
- Using the API to Control the Model
- The /save and /load Commands
- Erasing History with `/clear`
-
Working with the Model Library
- Understanding What's Loaded
- Inspecting Models
- Listing Saved and Loaded Models
- Stopping a Loaded Model
- Removing a Model
- Copying a Model
-
Keep-Alive and Memory Control
- Why Keep Models Loaded at All
- Setting Keep-Alive Globally
- Setting Keep-Alive per Request
- Forcing an Unload Right Now
- Multiple Models in Memory at Once
- Picking a Keep-Alive That Makes Sense
-
Concurrency: Parallel Requests and the Queue
- How Many Requests One Model Handles at Once
- Choosing the Right Number of Parallel Slots
- How Many Waiting Requests Are Tolerated
- How They Interact
-
Building, Running, and Sharing Custom Models for Ollama (Modelfile)
- Step 1: Put the Model under Your Own Name
- Step 2: Give It One Job with SYSTEM
- Step 3: Control the Output with PARAMETER
- Step 4: Stop the Chatter with a Stop Sequence
- Step 5: Teach the Format by Example with MESSAGE
- Step 6: See and Pin the Prompt with TEMPLATE
- Step 7: Package It and Read It Back
- Sharing Your Model
-
Creating a Fine-Tuned Model (English to SQL)
- Why Fine-Tune
- How Fine-Tuning Works
- Base Models vs Instruct Models
- Fine-Tuning the Model
- Understanding What Happened
- Fine-Tuning in the Browser with Unsloth Studio
-
Running Your Fine-Tuned Model in Ollama
- Step 1: Export to GGUF
- Step 2: Create the Model in Ollama
- Step 3: Run and Test the Fine-Tuned Model
-
Building a Management CLI for Ollama Using the SDK
- Setup and Requirements
- Building a Management CLI for Ollama
-
Building Advanced Agents: Introduction
- Pass 1: A Bare-Minimum Chat Loop against a Local Ollama Model
-
Building Advanced Agents: Conversation History
- Pass 2: Keeping a Conversation History
-
Building Advanced Agents: Streaming and Multi-Line Input
- Pass 3: Stream the Reply Token-by-Token and Accept Multi-Line Input
-
Building Advanced Agents: Long Conversations
- Pass 4: Trim the History so It Never Outgrows the Model's Context Window
-
Building Advanced Agents: Summarization with LangChain
- Pass 5: Swap Hard Trimming for an Automatic Summary of Older Messages
- Cost Notes
-
Building Advanced Agents: Caching
- Pass 6: Cache Model Replies in Redis so Repeated Questions Come Back Instantly
-
Building Advanced Agents: Long-Term Memory with mem0
- Pass 7: Give the Chat a Long-Term Memory That Survives Restarts
-
Building Advanced Agents: Function-Calling
- Pass 8: Let the Model Call Python Functions ("Tools") to Fetch Live Data
- The Shape of a Tool Call, End to End
-
Building Advanced Agents: Integrating MCP Servers
- LangChain and MCP
- Pass 9: Get Tools from an External MCP Server Instead of Writing Them In-Process
-
User-Friendly Interfaces for Ollama
- Local Chat UIs for Ollama
- Installing and Using Open WebUI
-
Afterword: Where to Go from Here
- What's Next?
- Keep Going
- Your Feedback Matters
Who This Is For
Developers who can read code and run commands, with no machine learning degree and no wish for one.
About the Author
Aymen El Amri is a software engineer, author and founder. He runs eralabs, built FAUN.dev, and writes about cloud native systems, AI engineering and developer tooling. More about Aymen.
