home.social

#oobabooga — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #oobabooga, aggregated by home.social.

fetched live
  1. jordan @[email protected] ·

    Have updated my local #LLM server from #Mistral 8B to #Qwen3 30B A3B. Still very fast. "Thinking" model so it tends to follow instructions and handle prompts with copious amounts of data better (#HomeAssistant reports).

    This is all done on CPU, too. 20 cores, but still. If you have #HomeLab equipment needing a use, I highly recommend setting up #oobabooga and slapping one of these bad bois in there so you can call it via API in your other projects.

    #llm #mistral #qwen3 #homeassistant #homelab #oobabooga
  2. jordan @[email protected] ·

    @Mawoka @kellogh I run #Mistral #7B Instruct via CPU on a #Debian #Linux server via #API through #oobabooga. I didn't need realtime responses, and I had spare CPU power, so it worked out perfectly.

    #mistral #7b #debian #linux #api #oobabooga
  3. Migrated to @[email protected] 🏳️‍🌈🎃🇧🇷Luana🇧🇷🎃🏳️‍🌈 @[email protected] ·
    CW: AI self-hosting

    Are there any ‘multimodal’ #AI frontends for #SelfHosting out there? Currently I use #oobabooga, which is really cool but it’s text-only.

    I mean like something similar to the new GPT-4o, which lets you upload pdfs, photos, access websites and etc on the chat.

    Also, would the models need to be prepared for this or are the same models usable?

    #ai #selfhosting #oobabooga
  4. jordan @[email protected] ·

    Hey fellow #AI / #LLM #nerds, I've discovered a weird issue with some LLMs I've been foolin' around with. I'm using #oobabooga, and Some models seem to just quit when formatting block text.

    ```markdown
    ```

    For example, the above is what *some* models do. Others don't seem to care. Something about the ``` token makes it give up. Seen anything like this?

    #ai #llm #nerds #oobabooga
  5. Migrated to @[email protected] 🏳️‍🌈🎃🇧🇷Luana🇧🇷🎃🏳️‍🌈 @[email protected] ·
    CW: AI, LLMs, Question

    I found a really good model!
    It’s really impressively good, specially for something I can run at my own machine!

    Here’s a link: huggingface.co/lmsys/vicuna-13

    I don’t have enough VRAM for it in the default settings, but it works just great in 8-bit mode.

    They have smaller versions here too: github.com/lm-sys/FastChat/blo

    I tried the 13b and 13b16k models, the 13b16k seems to understand my requests slightly better so that’s what I’ll use for now!

    (also I just noticed that it’s only 24GB in storage? Wtf?)

    It’s WAY better than anything else I’ve ever tried running locally, so I’m really impressed by it! It’s actually usable and pretty great!

    (If you know of any even better models that can run on a 4090, do let me know!)

    #AI #LLM #GPT #GPT4All #oobabooga #huggingface

    #ai #llm #gpt #gpt4all #oobabooga #huggingface
  6. Migrated to @[email protected] 🏳️‍🌈🎃🇧🇷Luana🇧🇷🎃🏳️‍🌈 @[email protected] ·
    CW: AI, LLMs, Question

    What should one consider when choosing a model in #Huggingface?

    Just the number of parameters and the filesize (the bigger these two are the better the model is)?

    Which would be the best model in there? I tried running some models locally but they’re so much worse (“dumber”) than ChatGPT, I wanted to run something closer to it. Maybe the models I chose just weren’t big enough.

    What is the “smarter”/biggest model in there?

    I have an empty ssd I could use if the model is really big, and I have a good gpu so that’s not a problem either as long as the model doesn’t need like 10 gpus to run

    Edit: okay apparently some models need like 300GB of VRAM to run, so lemme ask this differently:

    What is the biggest, best model I can run on a RTX 4090? (24GB VRAM)

    (64GB RAM)

    Edit 2: apparently there’s this bitsandbytes thing that could help?

    (Probably irrelevant but I’ll use the #oobabooga webUI)

    #AI #LLM #GPT #GPT4all

    #huggingface #oobabooga #ai #llm #gpt #gpt4all