BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Los_Angeles
X-LIC-LOCATION:America/Los_Angeles
BEGIN:DAYLIGHT
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
TZNAME:PDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
TZNAME:PST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20260402T024534Z
LOCATION:3001\, Level 3
DTSTART;TZID=America/Los_Angeles:20250623T141500
DTEND;TZID=America/Los_Angeles:20250623T143000
UID:dac_DAC 2025_sess108_RESEARCH274@linklings.com
SUMMARY:KVO-LLM: Boosting Long-Context Generation Throughput for Batched L
 LM Inference
DESCRIPTION:Zhenyu Li, Dongxu Lyu, Gang Wang, Yuzhou Chen, Liyan Chen, Wen
 jie Li, Jianfei Jiang, Yanan Sun, and Guanghui He (Shanghai Jiao Tong Univ
 ersity)\n\nWith the widespread deployment of long-context large language m
 odels (LLMs), efficient and high-quality generation is becoming increasing
 ly important. Modern LLMs employ batching and key-value (KV) cache to impr
 ove generation throughput and quality. However, as the context length and 
 batch size rise drastically, the KV cache incurs extreme external memory a
 ccess (EMA) issues. Recent LLM accelerators face substantial processing el
 ement (PE) under-utilization due to the low arithmetic intensity of attent
 ion with KV cache, while existing KV cache compression algorithms struggle
  with hardware inefficiency or significant accuracy degradation. To addres
 s these issues, an algorithm-architecture co-optimization, KVO-LLM, is pro
 posed for long-context batched LLM generation. At the algorithm level, we 
 propose a KV cache quantization-aware pruning method that first adopts sal
 ient-token-aware quantization and then prunes KV channels and tokens by at
 tention guided pruning based on salient tokens identified during quantizat
 ion. Achieving substantial savings on hardware overhead, our algorithm red
 uces the EMA of KV cache over 91% with significant accuracy advantages  co
 mpared to previous KV cache compression algorithms. At the architecture le
 vel, we propose a multi-core jointly optimized accelerator that adopts ope
 rator fusion and cross-batch interleaving strategy, maximizing PE and DRAM
  bandwidth utilization. Compared to the state-of-the-art LLM accelerators,
  KVO-LLM improves generation throughput by up to 7.32x, and attains 5.52~8
 .38x better energy efficiency.\n\nTopics: AI\n\nTracks: AI3: AI/ML Archite
 cture Design\n\nSession Chairs: Abdelrahman Hosny (Apple Inc.) and Marina 
 Neseem (Nvidia)\n\n
END:VEVENT
END:VCALENDAR