Linux Saloon 69: Self-Hosting Our Own Robot Overlords

This Open Mic Night acted as a carryover from last week’s Application Appetizer just a bit as we had some extensive conversation about VIM. Colin starts out the night with his experience using openSUSE with KDE Plasma which, of course, brought a huge smile to my face.

If you have an suggestions for topics, be they news, distributions, applications or anything that is Linux, tech or open source related, comment below or send an email to suggestions@linuxsaloon.com

Thanks so much for your continued support in watching, sharing and subscribing to Linux Saloon.

Discuss here on the Tux Digital Linux Forum:

00:00:00 Introductions
00:01:51 50 Years of Ethernet
00:02:44 Jinda talks about MX Linux
00:11:05 StrawPoll – Double Click vs Single Click
00:37:34 Self-Hosting AI
01:52:07 Next Week – On Location at South East Linux Fest
01:54:17 Last Call
01:57:49 Bloopers

Live demos (select “most likes”): https://huggingface.co/spaces
https://mashable.com/article/samsung-chatgpt-leak-details
Hardware requirements

https://iq.opengenus.org/basics-of-quantization-in-ml/
State of AMD compute: https://www.youtube.com/watch?v=Zsh6lPqvAcw
Self-hosted text to art with stable-diffusion

https://forum.tuxdigital.com/t/guide-self-hosting-stable-diffusion-on-ubuntu-using-an-nvidia-gpu/5815
https://github.com/AUTOMATIC1111/stable-diffusion-webui
https://github.com/lllyasviel/ControlNet
https://github.com/fkunn1326/openpose-editor
https://github.com/jexom/sd-webui-depth-lib
https://github.com/Uminosachi/sd-webui-inpaint-anything
Rent from a service: https://rundiffusion.com/
Live demo: https://huggingface.co/spaces?search=stable-diffusion-webui
Self-hosted image “dragging”

https://www.youtube.com/watch?v=dUpCy00PkNc
https://github.com/Zeqiang-Lai/DragGAN
https://vcai.mpi-inf.mpg.de/projects/DragGAN/
Live demo: https://huggingface.co/spaces/fffiloni/DragGAN
Self-hosted text chat

https://github.com/lm-sys/FastChat
https://huggingface.co/tiiuae/falcon-40b (top of huggingface leaderboard)
https://www.tii.ae/news/uaes-falcon-40b-dominates-leaderboard-ranks-1-globally-latest-hugging-face-independent
https://www.tii.ae/news/uaes-falcon-40b-now-royalty-free
https://www.mosaicml.com/blog/mpt-7b (32k context size)
Live demo: https://huggingface.co/spaces/mosaicml/mpt-7b-chat
Live demo: https://huggingface.co/spaces/mosaicml/mpt-7b-storywriter
Live demo: https://huggingface.co/spaces/mosaicml/mpt-7b-instruct
Self-hosted chat with documents

https://github.com/PromtEngineer/localGPT
Live demo: https://www.chatpdf.com/
Self-hosted speech-to-text

https://github.com/openai/whisper
https://github.com/mallorbc/whisper_mic
Live demo: https://huggingface.co/spaces/openai/whisper
Self-hosted text-to-speech

https://github.com/suno-ai/bark
https://github.com/neonbjb/tortoise-tts
https://github.com/snakers4/silero-models
Recommended channels

https://www.youtube.com/@engineerprompt
https://www.youtube.com/@NerdyRodent

Quick history of recent developments
MosaicMl’s MPT-7B
$300,000 estimated into weights (TO DO: provide link to source)
Based off WizardLM
Limitions of LLM context windows
Token comparison (top 3)
OpenAI ChatGPT – 4,096 tokens
OpenAI GPT-4 – 8,000 tokens (and a 32,000 token version that isn’t publicly available)
MosaicMl’s MPT-7B – 65,000 tokens
What you can do with 65,000 tokens that you can’t with 8,000
Samsung employees accidently leaking trade secrets to OpenAI
https://mashable.com/article/samsung-chatgpt-leak-details
Hardware & OS requirements for self-hosting
Model quantization for running on GPUs and CPUs
AMD GPUs aren’t built for AI
cuda cores, drivers, software
Open source AI lacks good UIs
Self-hosting Stable Diffusion demo
Ask for help and you’ll get it, TuxDigital forum or Linux Saloon Telegram
Question(s)
Currently GPU cores best perform the necessary tasks for these locally run AI programs, but as we see in almost every smartphone now, hardware dedicated to AI processing (ML/NE Cores) is a long-term necessity. Any thoughts on how Linux might address hardware ML cores in the future as they come to x86 chips?

Dr. Ian Malcolm: Yeah, but your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.

Other Resources:
Linux Saloon Community on Telegram – http://t.me/linuxsaloon/
Discord Server – http://discord.linuxsaloon.com/

Robot graphic taken from PETSCII Robots game
https://www.the8bitguy.com/product/petscii-robots/