• 0 Posts
  • 499 Comments
Joined 2 years ago
cake
Cake day: February 10th, 2025

help-circle

  • It depends on how complex you want to make it.

    The simplest way would be to use a GUI tool like LM Studio, it has the benefit of selecting the model quants that are likely to work on your hardware and having sane defaults for the various parameters that you can work from.

    Typically you’ll want to increase the context window, the default is good for short one-off questions but if you’re going to do RAG or coding tasks you will need a much larger context window. You will typically find this through trial and error, increasing until you hit memory limits or notice a large slowdown on tokens/second.

    You can also have the model respond to an OpenAI-like API interface on a designated port/interface if you want to run non-chat tools through the same model.

    Then, depending on the task that you’re trying to solve, you may want to find an MCP server built for that task which can provide the ability for the AI to interact with tools in a way that’s more controlled (giving an AI raw terminal access on a user’s machine is typically a bad idea).

    If you need something more flexible that you can incorporate into scripts and automation, llama.cpp has been popular for a while. Like Linux distros, everyone is going to have an opinion on the best and whatever you decide on will be the wrong choice. There’s also Ollama and vLLM if you want to run a datacenter.

    Model-wise it completely depends on the task. Do you need a multi-modal model that supports vision, is it going to be coding, is it trained to use tools/MCP servers, does it use reasoning, etc. The Qwen models are open weight and available in many different quants. The version 3.5 models run well on consumer hardware and can use vision, reasoning and tools.

    I’m running it through Llama.cpp but it will likely be available via LM Studio as well.


  • Anti-circumvention laws already exist and carry penalties including prison, it only takes one judge to create case law that this kind of reverse engineering is circumvention and then it’ll disappear from the commercial Internet.

    The safest way to go is to create the software from scratch, using reversed engineered proprietary software to make a different, but very similar, product is a legal minefield and largely unnecessary given that the same AI time that goes into reverse engineering the project could be directed towards making a clean room replacement which could completely avoid those legal issues while still replacing the product with a FOSS product.








  • AI is good at RAG tasks. When you’re asking it questions about text that is in the context window, the hallucination rate is significantly lower.

    It can fill that gap between documentation and implementation that isn’t always obvious to someone who is new and answer question that they may not be able to form into a search string. The kind of simple questions that, in some Linux communities, would invite assholes (like ‘how do you set permissions’ or ‘I want to add my user to a group’) to tell them to RTFM. Now TFM can answer their questions directly.









  • I was looking into upgrading storage and ended up opting to buy 20TB in HDDs for my NAS.

    I have to shuffle Steam games around but at 10Gb, it goes pretty fast (and you can run most games directly off the NAS) and the array can survive if a drive fails (ZFS).

    You need the extra hardware to run the NAS but HDDs are a significantly cheaper option and you don’t need SSD speeds for most things.