Do you host your own ML / AI / LLM? What do you use, and what do you use it for?
You must log in or # to comment.
An aside for anyone reading this:
https://sleepingrobots.com/dreams/stop-using-ollama/
And that barely scratches the surface. Please.
Use anything but Ollama. Even APIs.
Thanks for this link. Because of this article, I had claude stand up a llama.cpp container next to my already running ollama container. It ran side by side tests with the same model and parameters, and the results blew ollama out of the water. I’m in the process of moving hermes and openwebgui over to the llama.cpp instance to see how it goes day to day.
If you’re using docker anyway, and “fast” pure GPU models, you might try a vllm container while you’re at it.
It should be much faster than even llama.cpp, albeit at the cost of context length, and it supports some exotic 4-bit quantization like SPQA.
Same with TabbyAPI. It’s quantization is SOTA, though it does not support CPU offloading, and it’s speed is somewhere between vllm and llama.cpp.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters Git Popular version control system, primarily for code LTS Long Term Support software version SSH Secure Shell for remote terminal access
3 acronyms in this thread; the most compressed thread commented on today has 11 acronyms.
[Thread #27 for this comm, first seen 25th Jun 2026, 15:40] [FAQ] [Full list] [Contact] [Source code]
Yes, I got a Strix Halo machine before the RAM price hike and use it to run all my ML stuff on it.
Currently using llama-swap with llama.cpp/ComfyUI and opencode/Open WebUI as frontend.
I’m running Qwen3.6-27b, Voxtral Mini 4b, Piper and Qwen Image. Also, some embedding and reranking models.
I use them for:
- Tagging and classification of my documents in Paperless
- Home Assistant (voice assistant)
- Translations (both text and image)
- Transcriptions
- Some light coding and debugging
- Avatar/Backdrop generation for DnD sessions
I recently gave it a try with qwen3.5 and deepseek coder v2. I have a RTX3090 and these are the largest models that can run comfortably on it.
Conclusion, they are both fucking useless. Free tier claude runs circles.
Running qwen3.6 27b through llama.cpp.
It’s about as capable as sonnet 3.5.
I use it for light scripting, but real coding is done by cloud models.
I’m also using it as the brain for my Hermes agent. It sends me digests of news, subreddits, chats that I’d like to read but don’t have time for. It does a great job researching things on the web for me, too.
That’s a great model and it’s the one I use too.
I was hosting LLM with LMStudio occasionally but can’t access it anymore due to some fuckery with CORS and http vs https in browsers.
I googled it, and it seems like you can just enable cors.
Yes you can enable cors in LMStudio. But since few months it’s blocked by all major web browsers if you aren’t using HTTPS.
Which I don’t. I had LMStudio server open to local network so I can use it on my phone or laptop via third party website.
Why would it be blocked? I can use http sites just fine, and even then you could setup a self-signed certificate.
Seems this has been rectified. At least in Brave - it asks for permission to access local network at the first try, so this is now usable again
I host my own AI, mostly for testing and because I wanted something that was mine and mine alone. I use Ollama and run models like Llama, Mistral, and Qwen. I honestly don’t use it much, but I wanted to have my own setup just in case online services go down or become less available. It’s part of my whole “own everything I use” mantra that I’ve been on lately.