Do you use it to help with schoolwork / work? Maybe to help you code projects, or to help teach you how to do something?
What are your preferred models and why?
You must log in or register to comment.
I’ve been experimenting with it for different use cases:
- Standard chat style interface with open-webui. I use it to ask things that people would normally ask ChatGPT. Researching things, vacation plans, etc. I take it all with a grain of salt and also still use search engines
- Parts of different software projects I have using ollama-python. For example, I tried using it to auto summarize transaction data
- Home Assistant voice assistants for my own voice activated smart home
- Trying out code completion using TabbyML
I only have a GeForce 1080 Ti in it, so some projects are a bit slow and I don’t have the biggest models, but what really matters is the self-satisfaction I get by not using somebody else’s model, or that’s what I try to tell myself while I’m waiting for responses.
As a voice assistant server for my home assistant setup.
Edit. I leaned it by watching a Networkchuck video on YouTube. Basically a Piper integration for “voice to text” then a Whisper intergration for “text to voice”, and openWakeWord integration for wake word.
That sounds interesting! Can you describe what software you used for that? And how powerful does the hardware has to be?
Mostly to help quickly pull together and summarize / organize information from web searches done via Open WebUI
Also to edit copy or brainstorm ideas for messaging and scripts etc
Sometimes to have discussions around complex topics to ensure I understand them.
Favorite model to run locally now is easily Qwen3-30B-A3B. It can do reasoning or more quick response stuff and runs very well on my 24 GB of VRAM RTX 3090. Plus, because it has a MoE architecture with only 3B parameters active when doing inference its lightning fast.
have you tried LM Studio as an interface? I havent tried open webui yet, just lm studio and text-generation-webui, so I am not sure if I’m limiting myself by using LM studio so much (I am very much a novice to the tech and do not work in computer science, so I’m trying to balance ease of use with customization)
It sounds like we’re on similar levels of technical proficiency. I have learned a lot by reading and going down wormholes on how LLMs work and how to troubleshoot and even optimize them to an extent but I’m not a computer engineer or programmer for sure.
I started with LM studio before Ollama/Open WebUI and it does have some good features and an overall decent UI. I switched because OWUI seems to have more extensibility with tools and functions etc and I wanted something I could run as a server and use on my phone and laptop elsewhere etc. OWUI has been great for that although setting up remote access for the server on the web did take a lot of trial and error. The OWUI team also develops and updates the software very quickly so that’s great.
I’m not familiar with text-generation-WebUI but at this point I’m not really wanting for much more out of a setup than my docker stack with Ollama and OWUI
Thanks for the excellent response! I’m going to give openwebui a try and do some of that trial and error as well - best way to learn!
I’m in the process of trying them out again
Phi4 has been okay for me, and I use deepseek R1 32B quantized for some coding tasks. Both are a lot for my aging m1 MacBook Pro to handle.
Lately Ive been trying deepseek 8b for document summaries and it’s pretty fast but janky.
What I’m working towards is setting up an RSS app and feeding that into a local model (freshRSS I think lets you subscribe to a combined feed) to build a newspaper of my news subscriptions, but that’s not viable until I get a computer to run as a server.
I’m currently using it to generate initial contact emails, and generate contextual responses to received replies, for a phishing project at work.
in order to prevent phishing, right? (cue anakin/padme meme)
at work, right?
It is at work i promise lol.
lol. Uhhhhhhh not so much lol.
I work on an internal red team, so covert in prod operations instead of limited scope one off pen tests. We actively phish employees, but any victim user isn’t named in the report and we provide follow up training with them that’s not shame based and it’s with the operators directly, not some mandatory online class annual training bullshit.
It sucks, but this is a huge vector of initial compromise for APTs, and I work in an industry and for a company that are both extremely frequently targeted by APTs, so we have to do what we do. It lets us identify gaps in security and signature known ttps so our defensive teams can id those alerts when they pop.
i like the addition of ‘not shame based’ haha
Bro it’s a huge problem, companies will blame the employee that opened the malicious document instead of blaming their fucking abysmal internal security controls, detection, and response.
I employ this technique to embellish my email communications, thereby enhancing their perceived authenticity and relatability. Admittedly, I am not particularly adept at articulating my thoughts in comprehensive, well-structured sentences. I tend to favor a more primal, straightforward cognitive style—what one might colloquially refer to as a “meat-and-potatoes” or “caveman” approach to thinking. Ha.
I currently don’t. But I am ollama-curious. I would like to feed it a bunch of technical manuals and then be able to ask it to recite specs or procedures (with optional links to it’s source info for sanity checking). Is this where I need to be looking/learning?
you might want to look into RAG and ‘long-term memory’ concepts. I’ve been playing around with creating a self-hosted LLM that has long-term memory (using pre-trained models), which is essentially the same thing as you’re describing. Also - GPU matters. I’m using an RTX 4070 and it’s noticeably slower than something like in-browser chatgpt, but I know 4070 is kinda pricey so many home users might have earlier/slower gpu’s.
deleted by creator
How have you been making those models? I have a 4070 and doing it locally has been a dependency hellscape, I’ve been tempted to rent cloud GPU time just to save the hassle.
I’m downloading pre-trained models. I had a bunch of dependency issues getting text-generation-webui to work and honestly I probably installed some useless crap in the process, but I did get it to work. LM Studio is much simpler, but less customization(or I just don’t know how to use it all in lm studio). But yea, I’m just downloading pre-trained models and running them in these UI’s (right now I just loaded up ‘deepseek-r1-distill-qwen-7b’ in LM Studio). I also have the nvidia app installed and I make sure my gpu drivers are always up to date.
