Ollama is pretty easy to run. It’s where I’m going to start for running a local model. I know there are some better options, I might get to those later.

I put a link to this docker compose file in the last blog https://github.com/joshbressers/ai-skeptic/blob/main/docker/docker-compose.yaml

If you try to use this, look at the paths. I’m lazy and haven’t turned those into something better (maybe I have by the time you read this).

It’s pretty simple, just run docker compose up -d

Then run docker compose logs and look for a line that looks like this

ollama | time=2025-12-17T23:28:17.982Z level=INFO source=types.go:42 msg="inference compute" id=0 filter_id=0 library=ROCm compute=gfx1151 name=ROCm0 description="AMD Radeon Graphics" libdirs=ollama,rocm driver=60342.13 pci_id=0000:c2:00.0 type=iGPU total="96.0 GiB" available="95.0 GiB"

The important part is this type=iGPU total="96.0 GiB". That means the GPU is seen. In this case I have a GPU with 96 GiB of memory to work with (this is why the framework desktop is so great for this).