Product-of-Experts Defense Limits Model Distillation at Inference Time

A minimax framework models the arms race between model providers and adaptive distillation attackers, finding that standard defenses leave a large passive-to-adaptive performance gap. The proposed Product-of-Experts defense combines a teacher model with a proxy student during generation using only forward passes, suppressing outputs most useful for imitation without extra training.