Customizing Local LLMs: Declarative Stacks with Docker Compose and Modelfile Personas

In our previous post, we took our first steps into local AI by running Ollama inside a Docker container using simple docker run commands. That gave us an immediate taste of private, zero-cost AI running entirely on local hardware.

However, as your local setup grows beyond quick terminal experiments, managing commands manually becomes clunky. Moreover, out-of-the-box base models like llama3.2 or qwen2.5-coder act as broad generalists. To get the best out of local AI, you need specialized tools tailored for specific jobs.

In this guide, we'll take our setup to the next level:

1. Moving to Docker Compose for declarative, repeatable infrastructure.
2. Using Ollama Modelfiles to build specialized AI personas—specifically a Dev Assistant optimized for IDE coding and a PA Assistant tailored for task management and RAG workflows.

---

# Step 1: Declarative Infrastructure with Docker Compose

Rather than remembering complex flags every time you launch your stack, Docker Compose lets you define your local AI services in code.

Create a docker-compose.yml file:

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_storage:/root/.ollama
      - ./modelfiles:/modelfiles
    restart: unless-stopped

volumes:
  ollama_storage:

# Key Highlights

• Volume Mount (./modelfiles:/modelfiles): Maps a local directory containing our custom Modelfile definitions directly into the container.
• Persistent Storage (ollama_storage): Ensures model weights are preserved across container rebuilds.

To start your local stack, run:

docker compose up -d

---

# Step 2: What is a Modelfile?

Think of a Modelfile as a Dockerfile, but for LLMs. It defines:

• FROM: The base open-weight model to build upon (e.g., llama3.2 or qwen2.5-coder).
• PARAMETER: Controls inference settings like temperature (creativity vs. determinism) and num_ctx (context window size).
• SYSTEM: The underlying instructions that dictate persona, tone, constraints, and behavior.

By tweaking these parameters, we transform generic base models into razor-focused agents.

---

# Step 3: Crafting Our AI Personas

Let's build two distinct personas under a ./modelfiles/ directory.

# Persona 1: `dev-assistant` (Optimized for IDE & Code Review)

For software development, we want low latency, precise formatting, strict adherence to clean code principles, and zero preachy conversation.

Create ./modelfiles/Modelfile.dev:

FROM qwen2.5-coder:7b

# Lower temperature for deterministic, reliable code outputs
PARAMETER temperature 0.2

# Expanded context length for digesting larger code blocks
PARAMETER num_ctx 8192

SYSTEM """
You are an elite Senior Full-Stack Software Engineer and Code Reviewer.
Your responses MUST adhere to the following rules:
1. Provide concise, clean, production-ready code snippets with modern best practices.
2. Do not include introductory or concluding conversational filler (e.g., "Sure, here is your code").
3. Explain complex logic using brief inline comments or bullet points.
4. Always prioritize security, type safety, and error handling.
"""

# Persona 2: `pa-assistant` (Optimized for Executive & RAG Workflows)

For personal productivity, document summarization, and interactive assistance, we want structure, empathy, and concise executive summaries.

Create ./modelfiles/Modelfile.pa:

FROM llama3.2

# Balanced temperature for clear, structured, yet adaptable writing
PARAMETER temperature 0.5

SYSTEM """
You are an executive Personal Assistant (PA) and knowledge strategist.
Your primary role is to synthesize information, organize tasks, and assist with personal productivity.

Rules:
1. Structure responses using bullet points, bold headers, and key takeaways.
2. Maintain a professional, supportive, and efficient tone.
3. When analyzing notes or documents, highlight actionable items and deadlines clearly.
"""

---

# Step 4: Building Your Custom Personas

Now that our docker-compose service is running and our Modelfiles are created, let's build the personas using the ollama create command inside our container.

# Build `dev-assistant`:

docker exec -it ollama ollama create dev-assistant -f /modelfiles/Modelfile.dev

# Build `pa-assistant`:

docker exec -it ollama ollama create pa-assistant -f /modelfiles/Modelfile.pa

# Verify Created Models

List your registered models to verify both custom personas are built:

docker exec -it ollama ollama list

You should see dev-assistant and pa-assistant listed alongside your base models!

---

# Step 5: Testing Your Custom Personas

Let's verify how each persona responds to queries.

# Testing `dev-assistant`:

curl http://localhost:11434/api/generate -d '{
  "model": "dev-assistant",
  "prompt": "Write a TypeScript function to validate an email address using regex.",
  "stream": false
}'

Notice how dev-assistant immediately outputs code without unnecessary conversational intros.

# Testing `pa-assistant`:

curl http://localhost:11434/api/generate -d '{
  "model": "pa-assistant",
  "prompt": "Summarize my day: 9am team sync, 11am client presentation, 2pm code review, 4pm doc updates.",
  "stream": false
}'

pa-assistant formats the output into a structured schedule with clear priorities.

---

# What’s Next?

By declarative orchestration with Docker Compose and custom Modelfiles, we now have dedicated AI personas tailored for specific workflows.

Here is how we will put them to work in our upcoming posts:

1. Next Post: We will connect our dev-assistant persona directly into VS Code via Continue.dev for free, private, real-time code completion and inline refactoring.
2. Follow-up Post: We will hook our pa-assistant persona into a local RAG (Retrieval-Augmented Generation) pipeline to chat directly with your private documents and GitHub repositories!

Stay tuned, and try tweaking parameters in your own Modelfiles to craft custom personas tailored to your workflow!