Ampere?

#6
by dandandelion - opened

Will this work on Ampere? Spcifically the NVFP4 one? I noticed in FreeToken, NVFP4 quants work just fine on Ampere but I assume it does some fallback executions.

Edge0 org

Yes, it works on Ampere like RTX 3090

I did some brief testing with the NVFP4 on my little laptop:

Specs

i9-12900HK
64GB DDR5-4800
RTX 3080 Ti (@175W) 16GB
4TB NVMe Gen4

The NVFP4 performs around ~20-40 t/s (~40 on code, ~20 or below on prose). Haven't tested long prefill but I assume it would also be below the figures of the box in the model card. This iss configured to run at full 16GB VRAM and 48GB of my RAM (last 16GB's enough for my OS needs). I cannot seem to make it go faster than this so I assume this is my ceiling with this particular quant. Anyways it's definitely a feat for such a big model to run on my potato. Thanks!

Edge0 org

I did some brief testing with the NVFP4 on my little laptop:

Specs

i9-12900HK
64GB DDR5-4800
RTX 3080 Ti (@175W) 16GB
4TB NVMe Gen4

The NVFP4 performs around ~20-40 t/s (~40 on code, ~20 or below on prose). Haven't tested long prefill but I assume it would also be below the figures of the box in the model card. This iss configured to run at full 16GB VRAM and 48GB of my RAM (last 16GB's enough for my OS needs). I cannot seem to make it go faster than this so I assume this is my ceiling with this particular quant. Anyways it's definitely a feat for such a big model to run on my potato. Thanks!

Hi, thank you for your interest in our work. Your tested decoding speed is close to the theoreticals on your setups. You have more than 40GiB of RAM which allows you to reach theoretical top speed. But how high the theoretical top speed can go is determined by the GPU. Laptop GPU is constrained by its power therefore usually yields lower token speed thank desktop counterparts.

wanglamao changed discussion status to closed
wanglamao changed discussion status to open

Sign up or log in to comment