My local ML and LLMs setup - February 2025

Update: Added Obsidian Smart Connections.

The topic of LLMs and agents/<somethingsomething>-assistants is reaching a point in which commodity hardware can easily run even multiple models locally. I did some experimentation, and this is the setup I ended up with. This is on my personal computer, not for work, so your mileage may vary.

As this is a very fast-moving topic, I've also clearly timestamped this post.

General models

Ollama: Basically, "Docker for LLMs". Trivially easy to use, and new models are added rapidly. It is command-line based, so you need some UI if you want fancy interactions outside the terminal.

Open-WebUI: As the name implies, it is a web-based interface to interact with Ollama models. The Docker deploy setup is fantastic and effortless to maintain.

LM Studio: I was using Open-WebUI most of the time, but at times just launching a native app is faster. Thanks to the Ollama LM Studio Bridge, I reuse my main Ollama models instead of re-downloading anything.

ChatGPT, Claude, Gemini, Perplexity: I use these via the web interface. The preference of which one to try depends a lot on the kind of task I have in mind, and many times I try at least two just for the sake of comparing results.

Coding Assistants

GitHub Copilot: My pick for code autocompletion. It is evolving nicely and getting smarter as of late. And even if it provided nothing else (which it does), it's still autocompletion on steroids.

GitHub Copilot Chat: My default "coding buddy" for questions, suggestions, explanations, and the like. Usefulness depends a lot on the development language, but for common ones, it is excellent. Again, at minimum, it saves you a lot of time doing simple web searches, and seems to no longer hallucinate things it does not know.

Continue: To hook local coding models into VS Code. So far, I've only used the chat features, but there it works quite nicely. It also provides text autocompletion. I've noticed that it does not separate the "thinking" block with reasoning models (at least with DeepSeek R1), but I'd bet it will be fixed very soon.

Knowledge Base Assistants

Smart Connections: I am still in "trial mode" with the Obsidian plugin version of this tool. The "smart notes" feature shows related content, and you can query your notes, although the query feels too brief to me (even when I ask for details or lists). Recommended also to use a small LLM for quick results (at the time of this writing, I'm using Llama 3.1).

Text2Img and Img2Img

ComfyUI: My main tool to generate images. It is quite complex and I am no expert in the topic, so I mostly just follow tutorials to set workflows for each image model.

DifussionBee: Narrowed to a subset of models only, with an elementary but complete UI. For quick things, I try this tool first and, if not satisfied with the results, then jump to ComfyUI.

Freescaler: For image upscaling. I've found that a specialized tool such as this one works out of the box way better than generic upscaler models.

Stable Diffusion web UI: The ultimate image generation workspace. You can do anything, but a) the learning curve is similar to ComfyUI and b) installation is not easy [1]. I rarely use it any more.

[1] Despite the macOS-specific instructions, you may face issues unless you dockerize it, because of Python runtime version mismatches.