Can I run ollama on RTX 3060 and Inter iGPU to increase speed?

submitted by

I'm on Arch Linux btw. and I have a RTX 3060 with 12 GB VRAM which is cool so a 14b model fits into the VRAM. It works quite well but I wonder if there is any way to help with the speed even more by trying to utilize the iGPU in my Intel 14600K. It always just sits there not doing anything.

But I don't know if it even makes sense to try. From what I read in some comments on the internet, the bottleneck will be the ram speed in the iGPU, which will use my normal ram which is a magnitude slower than the VRAM.

Does anyone have any experience with that?

4
8

Log in to comment

Hot Top New Old

Deleted by author

 reply
5
OP

I see, that's a shame, thanks for explaining it.

 reply
2
edited

More gpus do improve performance:

https://medium.com/@geronimo7/llms-multi-gpu-inference-with-accelerate-5a8333e4c5db

All large AI systems are built of multiple "gpus" (AI processers like Blackwell ). Really large AI models are run on a cluster of individual servers connected by 800 GB/s network interfaces.

However igpus are so slow that it wouldn't offer significant performance improvement.

 reply
8

Deleted by author

 reply
4

No, full models are not loaded into each GPU to improve the tokens per second.

The full Gpt 3 needs around 640GB of vram to store the weights. There is no single GPU (ai processor like a100) with 640 GB of vram. The model is split across multiple gpus (AI processers).

 reply
2
Insert image