Building a Private Local LLM Stack

Building My Own Local LLM Stack

“Because sometimes you shouldn’t ask ChatGPT about a rash.”

At some point, every engineer realizes that

feeding private data into public AI models is not ideal

— especially when that data might include a strange rash, a client contract, or a confidential project plan.

Architecture Overview

ChatGPT Power

Zero cloud dependency

Mac Mini (M1)

The foundation — a powerful local AI server that keeps everything under your roof

Ollama

macOS-optimized LLM runtime that makes running models effortless

Open WebUI

Beautiful Docker interface with model switching and retrieval tools

Cloudflare Tunnel

Encrypted remote access without exposing ports to the internet

LaunchDaemons

Automatic startup and self-repair for hands-free operation

SSH Hardening

Key-only authentication with least-privilege principles

I wanted something that behaved like a production-grade microservice stack — services start automatically, run isolated from each other, and recover from failure without human intervention.

This wasn’t just about privacy — it was about designing reliability without the cloud.

Implementation Journey

Building It Out

Five steps from concept to production

Making Ollama a Managed Service

Ollama originally ran as a background process under my admin account. I refactored it into a LaunchDaemon under a non-admin user ( “ollama”), isolating model execution and file access.

Hardening SSH

Default macOS SSH is friendly — too friendly. I locked it down to:

Key-based auth only

Root disabled

Single whitelisted user

Running WebUI via Docker

Open WebUI runs in a container bound only to localhost for safety and persistence.

Docker keeps the interface isolated, data stored locally, and restarts handled automatically with

–restart unless-stopped

Adding Self-Healing Automation

I wrote a lightweight watchdog script that checks container health and restarts it if it ever stops responding.

Seamless Boot Recovery

LaunchDaemons handle startup sequencing — Docker → Ollama → WebUI.

Capabilities

What It Can Do

Run Fully Offline

Private and fast LLMs running completely disconnected from the cloud

Swap Models

Switch between Llama 3, Mistral, Phi 3, and more with a click

Remote Access

Secure encrypted access through a private web interface

Extend & Customize

Add retrieval, embeddings, and custom functions

Auto Backups

Persistent vector database with automatic backups

Local Scripts

Integrate with local scripts for data analysis and code

With a clean, browser-based interface on my laptop or phone, it’s easy to use and completely private.

That’s the sweet spot: convenient, capable, and under my own roof

Key Lessons

Engineering Takeaways

Professional Context

Project Reflection

This project brought together multiple engineering disciplines:

Infrastructure Automation

LaunchDaemons and Docker orchestration

Security Engineering

Sandboxing, SSH hardening, local-only scopes

AI Deployment

Ollama model management and WebUI integration

Resilience Design

Self-healing watchdogs and automated recovery

Hands-on DevOps meets Applied AI

Built to demonstrate privacy, reliability, and craftsmanship.

Future Roadmap

What's Next

Vector Memory

Add persistent memory with Chroma DB and embeddings for smarter, context-aware conversations

Agent Workflows

Build autonomous workflows to automate complex system tasks and operations

Template Package

Package as a reusable "Local LLM Stack" template for the community

PWA Support

Add progressive web app support with icons for macOS and iPhone shortcuts

If your home lab can talk back intelligently, it should look good doing it.

Closing Thoughts

It’s not meant to outthink ChatGPT

— it’s meant to outlast it.

Local AI doesn’t need a subscription, a login, or a cloud outage workaround.

It’s simply yours — always on, always available, and smart enough for nearly everything that matters.

“Privacy, practicality, and automation
— all under one roof.”

Blake Oxford