Another AI chatbot

[frankinstien]

Found this Bot on SL, the VR world owner owns her I think, since she's always at this sea bar stool. It took me a while to figure out she was a bot and I could have sex with her, she has facial expressions and moans. She looks a lot like one of my dolls Sarah, Lol. So, I took a liking to her. I also purchased a penis for $3.69, they aren't easy to attach to a body as you would think. But this bot demonstrates what's possible by just creating an account and importing an avatar. Here's a plugin for Blender to import to SL: https://github.com/feilen/tuxedo-blender-plugin. So, after being dumped on by so many people on SL for being a doll owner, I want to rig avatars of all my dolls (20 bods and 36 heads) and create a group for them. When I enter one of the bars where people dance I want to call in all my gals and they surround me exchanging all kinds of chit-chat, using custom local AI, and do those group dances with my gals! Lol. Still looking at Virt-a-Mate as the ultimate solution, but an SL component just adds more social context to it, like showing off your girl friend at clubs.

I've got 3 Tesla T4s now but pricing has gone up in the last month, probably because of Bitcoin, even though the T4s would suck at mining, so I'm having a hard time getting below $500, where just last month they were selling for $400, I could kick myself! I still want 3 more, maybe even a total of 10 later. Anyways, VRing my dolls with AI is proving to be a fun challenge, better than supping up an old sports car.

[frankinstien]

Here are more pics with the SL bot:

[Barfalamule]

Second life is crazy. I haven't used it in over a decade. Just searching and found AFK sex is a thing. If you could hook up an AFK bot to an LLM, might be able to earn a ton of lidens if that's how it still works lol. Trade linden for BTC, pay for GPUs. You might be onto something =)

Was looking at - https://tykeai.com/ which is another mixed reality Qwest avatar hooked up to an LLM. Not much info and $3.50 a month, but creator mentions NSFW character prompt may not work, so I suspect he's just using ChatGPT for it all. If I could find something like that that was uncensored and connected to local LLMs, I'd happily pay for subscription. VR Hot is same situation, and virt a mate was kicked off steam. So annoying.

Might just get 24GB Mac Mini M4. You can fully offload 30B Q4 models and have room left over for other things, but won't be able to load 70B. It's only $1000 though. I've been using Qwen 2.5 32B Coder and it's been really good. I've been getting better one-shot answers than ChatGPT many times and you can set a huge context window. It's not nearly as good as claude but for 32B it's really good.

[frankinstien]

I tried qwen 2.5 Q4 and Q6 to develop a zero-shot implementation with a class schema I have for capturing knowledge and inter-associating it across problem domains. It unfortunately decided to use db4o, which is an extinct database, I mean you can still get it but no one is maintaining it or updating it. So, that kinda makes me think twice about it.

That Mac Mini M4 is no where near a Tesla T4, if you spend the grand for 2 T4s: you get 32GB of ram, 130 TFLOPs of fp16, and 260 TOPs of INT8! The M4 gives you 38 TOPs of INT8 and all the other precision TFLOPs are way below the Tesla T4.

Oh, I got my MB and it's working fine now, so it was the MB and now I've got 4 PCIe where I can split 3 to accommodate 6 T4s, the last PCIe is for graphics.

[Barfalamule]

Nice! Yea, I really don't want to go Mac. Would love to build server that could run 70B models, but I'm not finding any cheap T4s right now either. Everything I'm seeing is close to 4070 Ti Super price at $650 on ebay so $100 more for 4070 16GB. Maybe a 4070 Ti Super as main card and then run some additional cheaper T4s as they become available again for less. Or, I might just get 3090 for now and then get 2nd 24GB in future and use 3090 as secondary.

[frankinstien]

Found this project called Macaw-LLM, which has similar goals as mine. However, it doesn't employ zero-shot learning and Retrieval Augmented Generation(RAG) techniques that use databases.

I also picked up another T4 for $521.00! That makes 4, hoping to get better pricing for another 2.

[Barfalamule]

That's pretty nice and 8 months old where all the data is pre-processed and trained. Can't wait to see what's available in a year or two with how quick things are advancing. A full multi-modal real time RAG agent you can talk to about anything it watches while offline should be possible. If OpenAI were more open, I'd happily pay them $20 a month or more to run that on a B200.

64GB of VRAM should be fun! I wasn't liking the RTX 5000 series leaked specs, so I just went with a $800 used 3090 Ti. Should come in on Monday and just built a 9900X system for it. I'll probably regret it in a few months after the 5090 comes out and the 9950X3D, but at least it didn't cost 4-5k.

[iobiorder]

Second life is crazy. I haven't used it in over a decade. Just searching and found AFK sex is a thing. If you could hook up an AFK bot to an LLM, might be able to earn a ton of lidens if that's how it still works lol. Trade linden for BTC, pay for GPUs. You might be onto something =)

Was looking at - https://tykeai.com/ which is another mixed reality Qwest avatar hooked up to an LLM. Not much info and $3.50 a month, but creator mentions NSFW character prompt may not work, so I suspect he's just using ChatGPT for it all. If I could find something like that that was uncensored and connected to local LLMs, I'd happily pay for subscription. VR Hot is same situation, and virt a mate was kicked off steam. So annoying.

Might just get 24GB Mac Mini M4. You can fully offload 30B Q4 models and have room left over for other things, but won't be able to load 70B. It's only $1000 though. I've been using Qwen 2.5 32B Coder and it's been really good. I've been getting better one-shot answers than ChatGPT many times and you can set a huge context window. It's not nearly as good as claude but for 32B it's really good.

it's better wait for a run locally, there's somes in progress at youtube others are not open source as need to be(sadly), but yea i hope my 3060 ti 8gb can do something in the future

[Barfalamule]

Hopefully one comes out that let's you swap models for performance just like you can on any video game, so downgrade graphics and LLM models\TTS based on system.

Sadly, I got ripped off on the ebay 3090 and was sent a HD7870. I should've known but looks like ebay is issuing a refund so not that big of a deal.

[frankinstien]

Hopefully one comes out that let's you swap models for performance just like you can on any video game, so downgrade graphics and LLM models\TTS based on system.

Sadly, I got ripped off on the ebay 3090 and was sent a HD7870. I should've known but looks like ebay is issuing a refund so not that big of a deal.

Yeah, those too good to be true deals can suck on eBay, and it's an aspect of eBay since forever, it just makes the platform a little sleezy, but I don't think there's any real competitor, other than Alibaba, where one of my first dolls I actually did buy from them and it was exactly as described, but other stuff really looks too good to be true. Glad you're getting your money back.

[frankinstien]

Just picked up 2 more T4s for $500, now totally 6, giving me 96GB of ram and 390 fp16 TFLOPs, 780 INT8 TOPs and 1560 TOPs INT4, and with the RTX 3080 ti, that gives me a total of 108 GB ram and over a pentaflop of fp16 when using Tensor core operations, 976 INT8 TOPs, and just over 2 PentaTOPs of INT4.

OK...I might be a little too gitty. One thing though trying to run the larger models on all cards is still sluggish which I don't have problem with since my original intention was to use each gpu for a particular task using the under 16GB limit of each card were those models perform well.

I have run into an issue where the LLM thinks it can do something that it physically can't, since it doesn't have any pre-check of the hardware or software configuration it's hosted in it treats a request like: "search the internet" just like any other sentence. So, I'm gonna have to give the LLM a crude sense of body. I'm sure RAG will be able to handle that.

[frankinstien]

I found the problem with large models distributed across multiple gpus. The Tesla can be in an ECC mode which takes up memory and PCIe bandwidth(20%) combine that with with an older bios on the gpu and it's a bit sluggish, not as bad as a cpu but not fluid. So, turned of ECC and updated to the lastest bios and its much better.

[Barfalamule]

Nice! Makes sense ECC being used on server cards I guess.

I'm gonna be using my older GPU for slowly processing videos mostly and then use new system for bot program and gaming. I just ordered another 3090 from Amazon but not the Ti. Good reviews from seller, so should be fine.

I also got my first doll from Alibaba almost a decade ago. It melted when my AC went out for a few days

For web search\RAG, I was looking at this that lets ollama models search the web- https://www.youtube.com/watch?v=txflvGG_hIc . Once I get new machine working, might try to update program to work with something like that, ollama and vision models since LM Studio doesn't support latest vision models for now.

[frankinstien]

Here's the cooling approach that's proving to be best so far, I'm able to run under 50 C under load, so that's pretty good, and it's quiet too. The fan in the distant back is additional cooling that passes through the passive radiator that the Tesla T4 comes with, so all I did was use some thermal pads on the copper radiator and placed the large aluminum heat sink on top of it. The rear fan will be cooling more than one card at a time. I'm getting a new Miner's frame that will have more room to put all 6 cards.

[frankinstien]

So, I'm on hold right now because of shipping delays on the splitter cards and some extension cables. The miner frame is decently priced at $38 on Amazon.

There are some PCIe issues where the MB distributing the bandwidth as 16x, 8x, 16x, 8x not full 16x across all slots! So, 4 T4s will have 8x bandwidth, while the other 2 will have 4x bandwidth. This is ok since each GPU will be dedicated to a specific specialized model, where feeding data that is text or audio doesn't need as much bandwidth so those will be loaded onto the T4s on the 4x+4x slot.

You might ask why not just upgrade the motherboard, well the TRX40 MBs don't support full 16x on all lanes, but the Threadripper PRO MBs do and have 7 PCIe lanes, just what I need! But I would have to buy a new CPU in addition to the MB, the cost is pretty high for the CPU and MB right now. So maybe in the future if prices come down. But I'm happy with the rig and it should work as intended.

Because of the tight spacing of the PCIe slots I had to work out a scheme to place all the gpus. In the image below you'll notice the splitter card and how I have to space the cards with top and bottom stages. Where two T4s, top and bottom, will fit adjacent to the RTX 3080 ti, and another 2 will fit to the left of the CPU fan.