why avg draft acceptance rate so low?

[hsyue]

(APIServer pid=1) INFO 05-01 14:24:50 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 2.37, Accepted throughput: 8.90 tokens/s, Drafted throughput: 97.50 tokens/s, Accepted: 89 tokens, Drafted: 975 tokens, Per-position acceptance rate: 0.600, 0.385, 0.215, 0.108, 0.062, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 9.1%

why avg draft acceptance rate so low?

[MesoX]

Having same problem here -> mostly in range of 1 - 10 % for acceptance rate. It does not seems right?

[AEON-7]

those are lower stats than I was seeing, but there will be a lower match with abliteration I think you are more likely to hit higher match if you set temperature to 0 greedy and you will find match will be very high on things like reasoning, math, and lower on creative writing due to the lack of censorship and guardrail guidance. Acceptance will be very heavily corelated to how deterministic the answer is and how natural your input prompt is. A lot of default benchmarks are going to throw a bunch of random synthetic tokens that don't line up with real world use and that is antagonistic to DFlash and MTP. You want more natural prompts that reflect more real world use in the benchmarks to know the true acceptance stats. Good to do a benchmark per category of expertise to see what hits high acceptance vs low.

[bcdiaconu]

In my case is even worse (single DGX Spark):

(APIServer pid=42) INFO 05-04 15:43:57 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 1.27, Accepted throughput: 0.90 tokens/s, Drafted throughput: 49.50 tokens/s, Accepted: 9 tokens, Drafted: 495 tokens, Per-position acceptance rate: 0.242, 0.030, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 1.8%
(APIServer pid=42) INFO 05-04 15:44:07 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 50.0%, Prefix cache hit rate: 0.0%
(APIServer pid=42) INFO 05-04 15:44:07 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 1.31, Accepted throughput: 1.00 tokens/s, Drafted throughput: 48.00 tokens/s, Accepted: 10 tokens, Drafted: 480 tokens, Per-position acceptance rate: 0.312, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 2.1%

[vio1ator]

The image is missing this PR, at longer context, acceptance rates plummet

vllm-project/vllm#40898

[AEON-7]

Compiling an updated container, standby it should resolve the low acceptance, and boost performance.

[ceejx]

I'm seeing 10-20% as of the release avaliable today, single spark for code generation. OpenWebUI -> LiteLLM -> Aeon-VLLM

aeon-ultimate-xs  | (APIServer pid=1) INFO 05-10 17:30:24 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 26.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.0%, Prefix cache hit rate: 0.0%
aeon-ultimate-xs  | (APIServer pid=1) INFO 05-10 17:30:24 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.26, Accepted throughput: 18.10 tokens/s, Drafted throughput: 120.00 tokens/s, Accepted: 181 tokens, Drafted: 1200 tokens, Per-position acceptance rate: 0.675, 0.500, 0.400, 0.212, 0.125, 0.100, 0.100, 0.050, 0.050, 0.025, 0.025, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 15.1%
aeon-ultimate-xs  | (APIServer pid=1) INFO 05-10 17:30:34 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 28.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.0%, Prefix cache hit rate: 0.0%
aeon-ultimate-xs  | (APIServer pid=1) INFO 05-10 17:30:34 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.48, Accepted throughput: 20.10 tokens/s, Drafted throughput: 121.48 tokens/s, Accepted: 201 tokens, Drafted: 1215 tokens, Per-position acceptance rate: 0.728, 0.568, 0.420, 0.222, 0.160, 0.123, 0.111, 0.062, 0.037, 0.025, 0.025, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 16.5%
aeon-ultimate-xs  | (APIServer pid=1) INFO 05-10 17:30:44 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.0%, Prefix cache hit rate: 0.0%
aeon-ultimate-xs  | (APIServer pid=1) INFO 05-10 17:30:44 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 2.60, Accepted throughput: 12.80 tokens/s, Drafted throughput: 119.99 tokens/s, Accepted: 128 tokens, Drafted: 1200 tokens, Per-position acceptance rate: 0.675, 0.463, 0.225, 0.138, 0.050, 0.037, 0.013, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 10.7%
aeon-ultimate-xs  | (APIServer pid=1) INFO 05-10 17:30:54 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 30.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 0.0%
aeon-ultimate-xs  | (APIServer pid=1) INFO 05-10 17:30:54 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.83, Accepted throughput: 22.60 tokens/s, Drafted throughput: 119.99 tokens/s, Accepted: 226 tokens, Drafted: 1200 tokens, Per-position acceptance rate: 0.775, 0.588, 0.463, 0.275, 0.225, 0.200, 0.163, 0.025, 0.025, 0.025, 0.013, 0.013, 0.013, 0.013, 0.013, Avg Draft acceptance rate: 18.8%