# Running Open-Source AI Locally: A Beginner’s Guide to Ollama and Docker

If you’ve been experimenting with AI tools recently, you’ve likely noticed a common trade-off: reliance on cloud APIs, subscription costs, latency, and privacy concerns regarding where your prompt data goes.

Fortunately, open-weight models (like Llama 3, DeepSeek, and Gemma) have matured rapidly. Today, you can run state-of-the-art language models directly on your own computer—completely offline, private, and free.

In this guide, we’ll set up a clean, isolated local AI workflow using Ollama inside a Docker container.

# Why Run AI Models Locally?

Before jumping into terminal commands, here is why running local models is worth setting up:

1. Complete Data Privacy: Your prompts, code snippets, and data never leave your local machine.

2. Zero API Costs: No token limits, subscription tiers, or unexpected usage bills.

3. Offline Reliability: Build and test applications even without an active internet connection.

4. Clean Isolation: Running Ollama via Docker keeps your host operating system clutter-free without scattering dependencies everywhere.

# Prerequisites

To follow along, you only need two things:

• A working installation of Docker (or Podman if you are on Linux).

• Basic familiarity with running commands in the terminal.

# Step 1: Run Ollama via Docker

Instead of installing binaries directly on your OS, we’ll run Ollama inside a lightweight Docker container.

Open your terminal and run:

docker run -d \
  --name ollama \
  -v ollama_storage:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama

# What is happening here?

• -d: Runs the container in detached (background) mode.

• -v ollama_storage:/root/.ollama: Creates a persistent volume so downloaded AI models aren't lost when the container stops.

• -p 11434:11434: Exposes Ollama's local REST API on port 11434.

# Step 2: Download and Run Your First Model

With the container running, let's pull a lightweight model. We'll use Llama 3.2 (or gemma2:2b if you want an ultra-lightweight option).

Execute the following command to pull and start an interactive chat session:

docker exec -it ollama ollama run llama3.2

Ollama will download the model weights (this may take a minute depending on your internet speed). Once complete, you’ll see a prompt where you can start typing directly:

>>> Why is the sky blue?
The sky appears blue to the human eye because of a phenomenon called Rayleigh scattering...

To exit the interactive chat session at any time, type /exit.

# Step 3: Querying Your Local AI via API

Since Ollama exposes a REST API on port 11434, you can send requests to it directly from your terminal, custom scripts, or backend applications (Laravel, Python, Node.js, etc.).

Test it in your terminal using curl:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain Docker in two concise sentences.",
  "stream": false
}'

You'll receive a JSON response containing the generated answer right inside your terminal:

{
  "model": "llama3.2",
  "response": "Docker is a containerization platform that allows developers to package applications and their dependencies into lightweight, portable containers. These containers run consistently across any system that supports Docker, eliminating environment setup issues."
}

# Finding Models: Where to Discover open-weight AI

Curious about what other models you can run locally? There are two primary hubs to explore:

• Ollama Library: The easiest place to start. It features pre-configured, optimized models (like Llama, DeepSeek, Gemma, and Mistral) that you can pull with a single command.
• Hugging Face Hub: The central repository for the entire AI community. You can find thousands of fine-tuned GGUF models (quantized formats specifically designed for local CPU/GPU inference) that can be imported directly into Ollama.

# Popular Starting Models

Depending on your system specs (RAM and VRAM), here are a few great open-source models to test:

• Llama 3.2 (3B): Fast, general-purpose, and great for low-resource machines. Command: ollama run llama3.2

• DeepSeek-R1 (7B / 8B): Excellent for reasoning, math, and code generation. Command: ollama run deepseek-r1

• Gemma 2 (2B / 9B): Google's lightweight open model, well-balanced for quick tasks. Command: ollama run gemma2

# What’s Next?

Now that you have a private, containerized AI engine running on your local machine, the possibilities open up significantly.

In the next post, we’ll build directly on top of this setup: we'll learn how to connect your local Ollama container directly into VS Code and other local code editors (using extensions like Continue.dev) for free, completely private inline code completion and chat!

After that, we'll dive into building a local RAG system to chat with your GitHub repositories.

Stay tuned, and let me know in the comments what models or use cases you are planning to test locally!