Does anyone run their own AI service?
If so,
Which one?
What hardware?
Do you have a way to access it from outside or from a phone?
And, if you’re open to sharing, what do you do with it?
I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don’t absolutely need specialized hardware to run and perform well enough for many purposes.
I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I’m not comfortable using them to help me with web searches and the such, because there’s no way that all this information won’t be going into a profile. This makes me want to have my own “helper” service.
You must log in or register to comment.
Llama.cpp running qwen 3.8 on a mini PC with 32GB ddr5 ram and 8GB vram. Not the most speedy but it works. I only really access it while at home but I’m also set up with tailscale.
So you’re using CPU offloading? Or the model’s Bonsai2 variation?
I wouldn’t mind running a model at home, mostly to help document my setup. But I just don’t trust it. So no.
It depends on your hardware. But right now the “cheapest” hardware you can buy for a usable setup, is the strix halo (128GB), and run gufo with qwen 3.8 flash next (Q4XL). It is sonnet 5 level. Very good speed (both prefill 1000+ and decode 50+), with up to 256k context.
I have it set up with omp + searxng. Sometimes with hermes. Really good! It changes the game for local setup.
strix halo (128GB)
googles
Jesus christ
Just get the cheapest OEM, Bosgame M5. I know “cheapest” is in quotes. But again, it is the cheapest 128GB unified vram device you can get right now. I got mine at pre-orders, so yes, more than a year and a half ago, prices were a steal. Also, great desktop (mini) PC overall and very efficient, even if you don’t want to do AI.
I have like 150GB of DDR3 spread over multiple 2/4/8/16GB sticks, I wonder if I can get that working for something like this 🤔
Why wouldn’t you trust a local model? I mean sure they are less intelligent but if you dont expose it publicly you should be good to go
I don’t trust it to be accurate; I’d have to reference the sources it pulled from. Extra work for little payoff.
Got it, makes sense. You’d probably have to do some benchmarking first. Depending on how large the source catalog is.
A quick glance shows that I’m in the minority. I grabbed a Bosgames M5 with 128GB UMA RAM when I discovered openclaw (at the time claudebot). I’m running Lemonade, which is a wrapper for llama.cpp, sd-cpp, flux, and others. I’m happy to have paid a “mere” $2400 compared to what similar systems cost now, and now I’m toying with fine tuning my own models. Though, I primarily run Hermes Agent paired with Qwen3.6 35b A3B (using a heretic ablitieration for 0 refusals).
Hermes Agent does a lot of random stuff for me right now. It manages my calendar and to-do list in nextcloud via MCP server. It created and maintains an llm-wiki based on my Obsidian Vault. Then it creates a report every morning about both and the weather.
The server has tailscale installed, so yes I can access it anywhere.
Try gufo with qwen 3.8 flash next. Much much better than 3.6-35B.
How well does it handle concurrency? I could see this as a vast improvement on llamacpp, but I was already speccing out vLLM.
Not only do I run and share it. I’ve built an entire service to allow everyone else to do so safely. https://aihorde.net/
I just bought a 32GB M6 Mac Mini that I’ll also use for local ai. Will probably use Qwen3.8-27B. Accessing it from outside my lan would happen via Tailscale.
If you are playing around at home I suggest seeking a MoE model. You get less penalty from having less vram and offloading to normal ram. I run Qwen 3.6 quant 6 on 250k context on a 16gb vram 64gb ram ddr4 PC. I get about 31tps which is enough for my use. After a little tweaking I have put results of it’s output into Claude Pro for vetting and it hasn’t found any flaws in the simple scripts and things I’ve asked it to make. I can only run Qwen 3.8 which is a dense model and not MoE. At low quant with not as much context. It spits out accurate short context answers if that’s all you need. So I prefer 3.6. Tiel is faster but Claude picked up more errors in what I was doing. Orinith kept getting into thinking loops for me so I stopped using it. Anyway I’m just a little guy.
I use Qwen 3.8 27b Q4 with 100k context (llama.cpp and the Mistral AI harness) on a 24 GB AMD Radeon 7900 XTX and it’s been great. It can handle pretty much anything I throw at it and I can let it work indepedently overnight / over days on long tasks.
I have nearly the same setup, except OpenCode for the harness. I run 200k context for anything that doesn’t need vision, or 150k otherwise. Typically get about 30 t/s decode With MTP, starts at close to 800 t/s for prefill and drops to the 200s for a long context. I’d imagine higher decode speeds are possible with some tinkering, but I haven’t gotten around to it.
OpenCode recently got hacked and advises changing API keys fyi.
I get about 45 t/s after some tweaking, getting the thermals under control helped a lot; mainly by reducing the power consumption to 261 W (and downclocking a little bit).
What vision model do you use?
How much electricity does it burn? And how much does it cost to let it run overnight? I feel like this is the same problem that the companies have, its way too expensive to use it on a lots of things and it is way cheaper to use it on someone else’s server…
It uses 261 W, probably more expensive than getting a 2nd Mistral subscription with the cost of power where I live (Belgium), but it will be free heat once I have solar panels and a battery.
Thank you, that is not as bad as I expected. 👍
Yes! I’ve got quite a few projects going right now. Probably too many of them to keep track
I am the only user, so it’s relatively easy to run multiple models for what ever task. Ollama is pretty good at balancing which models are working at what time to not bog down they system
I’m running gpt-oss 120 for a general purpose Hermes agent. Out of the box, it was ok. Not very snappy, but it was one of the best ones for orchestrating other agents.
Glm flash (4.something?) does pretty good as well.
I’ve found a hybrid of cloud and local is the best solution. Use local for smaller or more defined tasks. Use cloud for orchestrating, planning, and…fixing the local agents mistakes lol
I’ve got an LLM maintained Obsidian wiki that is written and edited by a few Ollama agents based on chats via my own self hosted matrix instance. I’ve taken it to a point where I’m pretty happy with the results. But I think if I had enough time and energy, I could really dial it in.
One of my favorite use cases for this is when I’m researching topics like UFOs/high strangeness/conspiracy theories. I have my very own LLM maintained Obsidian vault filled with curated notes and links to sources.
Literally, no one cares about this and they think it’s stupid and I’m stupid and I should feel bad for spending time on this or having the audacity of sharing it… they just don’t appreciate the amount of work and knowledge it took to get what I have set up.
I think it’s cool as fuck. I geek out every time that it works.
Oh, forgot to mention I am running it on a Mac studio with 96 gig of unified memory. I think it’s an M3 ultra chip.
Do what makes you happy mate! Tell the naysayers to fk off.
I self host both at home and at work.
At work (for completion and generation) we have a now quite old A6000 running mainly Qwen 3.6 (though I’m trialling 3.8 from Monday) with OpenClaw as the harness. The use case is mainly triage and pre-work on bugs. We have an agent which can be given an exception report with a bunch of information like stack trace and local variable values and it will go off and gather the required code, order files, and data to analyse the issue and come up with a fair idea of the cause and possible solution. This saves devs and support team time. Also looking at adding a PA agent to this which will be able to read emails and calendars and be able to set up (with approval) meetings and move things around and generally make someone’s life a bit less admin-y without having to hire a full PA role.
Also technically in the AI space we have a fleet of Pis with Hailo AI hats running object and anomaly detection in visible and LWIR spectrums for QC
Are you planning on using a specific package for your PA agent? Or is this something you plan on developing in house?
Using openclaw for the harness and then attaching office 365 mcp with an Entra ID App token to give it access to email, calendar, and teams. The MCP format means that we can explicitly control what it reads and writes
Even ignoring all of the functional issues with LLMs/“AI” related to their design to never admit to not knowing and preferring to make stuff up instead (hallucinate), I have an ethical qualms about the way they are trained. Two primary issues are that they aggressively scrape data from sources that explicitly do not consent to sharing their work and the data centers which have made hardware impossibly expensive for everyone but them and are so impactful to our underfunded electrical grids that they cause brown outs that keep people from heating their homes among other things, cause major spikes that cause house fires and destroy home appliances, and create an excuse to generate power using the worst polluting generators that we had finally gotten shut down to slow climate change (personally I’m tired of the wildfires and extreme storms from climate change).
So, I personally consider it extremely unethical to use the technology for that and several other reasons which are hard to touch on exhaustively in a post like this.
Yes, I installed an LLM at home. I use it for many useful things:
- reading novels by translating them from English to my language on the fly via machine
- for communicating on lemmy by translating posts and comments
- For HomeAssistant
- And very rarely for development and searching the internet
I have a RX9070XT and llama-swap to switch models on the fly. I’m thinking about installing a second RX9070XT so that I don’t take away the translation capability from my lemmy instance while I stress the other neural network with other tasks.
I tried, but didn’t want to invest into it, so I used my 12-GB-vram gpu but no matter what I did it was worse than Lumo from proton,which is basically only one LLM from 3rd party that I am okay to use on occasions and since it has really good free tier I gave up on self hosted LLM.
Yes, I have a desktop with a radeon 9700 ai pro with 32GB of VRAM. Its more than doubled since I bought it but its basically a down clocked 9070xt with double the VRAM. I also have a little arc b50 in my server.
Mostly yeah use it for privacy reasons. I run llama.CPP directly and access via open webui. I host a searx instance for it to use as web search. Mostly run qwen3.6/3.8 or gemma4 on the radeon and almost exclusively gptoss 20b on the B50 along with a second small model for tags and titles in open webui. Yeah I just use it mostly as a basic chatbot or as an agent in obsidian using the engram plugin. Helps search and summarize years of notes and works pretty well.
I’ve tried to use it as a coding assistant in zed without much luck so usually just post my code into open webui for feedback and it works alot better for me.
Yeah I can access it with other machines on my network including phone with conduit app. I can get to it on the go by connecting to home network with wireguard and sending a magic packet to wakeup desktop.
I havent done any image gen nor have much use for it outside of maybe using it to make app icons in the style of papirus icons for programs they don’t and won’t support.
It definitely feels more comprable to first ai wave than current frontier models but the privacy and price trade off is worth it to me. I’d say using it in obsidian has been the best use of it so far and one I certainly wouldn’t trust to an external AI.
Hardware:
2 x ASUS GX10 128GB(each)
2 x Mac Mini M4 Pro 24GB(each)
The GX10s are clustered and run DeepSeek-V4-Flash-0731-Abliterated. The M4 Pros run various utility LLMs such as Qwen 3.8
Software:
OpenWebUI
Hermes Agent
LiteLLM
Llama.cpp
vLLM
StoryBinder - custom web app that I use for story/script writing when in the mood
LASM - Local AI Stack Manger - custom web interface to manage pretty much all aspects of my local stack. Agents run on the 4 nodes as well as my NAS(Download GGUF/HF snapshots once and copy out to the nodes)
Have yet to connect my Home Assistant into the stack.
For mobile access I use Conduit https://github.com/cogwheel0/conduit
I run Gemma4 12B on my RX9070 XT. it’s not permanently on, I just turn it on when I want to use it. Access it through a browser. Does what I want out of it. Nothing “agentic”.
