I ran Q2 on RTX 3060 12GB with 30GB RAM

#7
by RustingStudio - opened

Thanks for the quant! I got it running on an RTX 3060 12 GB with 32 GB of RAM. The file is 7 times bigger than the card and it still writes 4.3 to 5.1 tokens/s.
Long prompts hurt: 20,000 tokens took 29 minutes to read. But it's just cool that I can run such big model on my shitty hardware
Engine is RustingStrata, my open-source one in Rust: https://github.com/RustingStudio/RustingStrata
If someone has an AMD card or enough RAM for the whole file, I'd love to see your numbers.

Sign up or log in to comment