Dieser Bereich kann Inhalte enthalten, die nicht für alle Nutzer geeignet sind. Dazu können unter anderem Texte, Medien oder Diskussionen gehören, die als beleidigend, extremistisch, gewaltbezogen oder anderweitig belastend empfunden werden. Wenn du solche Inhalte nicht sehen möchtest, nutze bitte die jeweiligen Filter- und Meldeoptionen der Plattform oder meide entsprechende Threads/Communities.
How do you run qwen 3.8 flash next on 16gb vram, if you don’t mind me asking ?
At 3 tokens a second and calling it useble I would guess?
I see people claiming that shit all the time and it’s baffling.
They said 50- i looked up strata, and it uses smaller quantizations/optimizations of that model depending on your system.
I have 48g vram and 192g system ram and get ~15t/s on qwen 3.8 flash next at Q4. Off the top of my head. I think Q6 is 12.
I started looking into it finally after making that comment and came across strata come back here and everyone is saying strata. I guess im trying strata?
https://sh.itjust.works/comment/28041846
Try it out and let me know if you can get higher.
https://sh.itjust.works/comment/28041846
I didn’t believed it until I started using it.
need 64gb of system ram. qwen next architecture uses smaller experts than others, and it can keep up with ram swap. I believe it uses a file on disk (ngram) to predict what experts will be needed.
https://github.com/niko1221/strata
The model I’m using is a Q4. Unsloth UD-IQ4_XS