I haven’t updated this blog in a while. Mostly because nothing super interesting has been happening. Things have been mostly just working (tm).

But a lot has changed in the last few weeks. There are a bunch of new models to run. Qwen 3.8, a new Nemotron, a new DeepSeek, and a new model called Muse Glimmer. And there have been a bunch of the things I was using to run the models that are no longer maintained.

The way I’ve running things won’t really work anymore because it looks like many of the previous things I’ve mentioned have been put on hold by their developers or are taking a different direction. Rather than trying to use Ollama or one of the other similar toolkits, it’s going to be easier to just run Llama.cpp directly.

Toolbox#

I’m using a podman toolbox to do this. It’s pretty easy and I struggled with getting Llama.cpp working on the base system because you need a bunch of specific libraries.

I first run

toolbox create vllm --image docker.io/kyuz0/vllm-therock-gfx1151:latest -- --device /dev/dri --device /dev/kfd --group-add video --group-add render --security-opt seccomp=unconfined --network host

This does all the setup things. Then I run this command to use the toolbox

toolbox enter vllm

I’m now using all the things from that toolbox container. I run all this inside a tmux session because I keep llama.cpp running in the foreground. There’s probably a way to do this in the background, but I like looking at the various metrics it prints.

Here’s how I run the latest Qwen 3.8

llama-server --hf-repo unsloth/Qwen3.8-27B-GGUF --alias model --host 0.0.0.0 --port 8000 -ngl 999 -fa 1 --no-mmap --jinja -c 262144 --reasoning-preserve --spec-draft-n-max 5 -np 1 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0

llama-server will download that model from huggingface automatically which is great. Everything ends up in ~/.cache/huggingface if you need to remove something.

The URL to access everything will be at http://hostname:8000/v1

The flags to watch for are -np 2 which lets me connect two clients (it’s very slow, but keeps some of my agents more happy). The other one is --ctx-size the context window size. Change this to whatever your hardware supports.

The most important flag is probably the --alias model. Because I have a bunch of things that need to know which model to use, and I’m obviously controlling what model is running (I can only run one at a time), there is minimal benefit to having to reconfigure everything anytime I change the model. So I just alias them all to being called model. It’s a little bit of a hack, but whatever.

I won’t pretend I understand what all those options are for. I rely on the Unsloth instructions for how to run all this, whatever they suggest is what I use. For example Run Qwen3.8 in llama.cpp

Open WebUI#

I also had to change how I run Open WebUI. I have a docker compose file in GitHub that will run just Open WebUI. Most of my previous examples also ran the LLM while also running Open WebUI.

What’s next#

I don’t have any grand plans for the future really. I’m sure things will keep moving along at a wild pace. No doubt everything above will be useless in a few months. Things are working pretty well. I’ll add periodic updates here as this is where I go to remember how things work :)