Amazon Science and Booking.com use knowledge distillation plus Medusa to speed up LLM inference by up to 20x
Amazon Science and Booking.com combined knowledge distillation with Medusa to compress a 70B-parameter Llama-3.1 teacher model into a 1.1B TinyLlama student model, increasing inference speed by 10 to 20 times and reducing cost to less than one-third while maintaining the teacher model's quality. The approach has been used in three features—AI Trip Planner, Smart Filters, and accommodation description generation—and won the Best Paper Award in the Industry Track at ACL.
Matched fieldsBody

