Ollama is pretty easy to run. It’s where I’m going to start for running a local model. I know there are some better options, I might get to those later.

I put a link to this docker compose file in the last blog https://github.com/joshbressers/ai-skeptic/blob/main/docker/docker-compose.yaml

If you try to use this, look at the paths. I’m lazy and haven’t turned those into something better (maybe I have by the time you read this).

I ordered a framework desktop to start my research into using LLMs for various tasks.

Why a framework? I want to run the LLM locally. The biggest reason for this is I will learn more than if I let someone else do it. After looking around and talking to some people I trust, the Framework desktop was a no brainer compared to the other options.

The 128G of RAM will let me run some fairly large models locally. It’s not the fastest thing ever, but for what I want to do, it’ll work.