Speeding up language models for travel applications

Researchers combined knowledge distillation, which trains a compact student model on a large teacher's outputs, with Medusa, a framework that predicts multiple tokens in parallel, and applied the pipeline with Booking.com to three travel features. The final model matched the quality of a 70-billion-parameter teacher while running 10 to 20 times as fast, with reported gains including 2.9% higher engagement versus GPT-3.5 for AI Trip Planner and 15-times-faster Smart Filters responses for 99% of queries. The work received the Industry Track best-paper award at ACL.