Text Entropy Rate Filtering to Prevent Fine-Tuning Collapse

A non-parametric Kontoyiannis entropy rate estimator ($h_k$) filters synthetic training data to prevent model collapse during iterative fine-tuning. In Llama-3.1-8B experiments, this method increased text uniqueness by 42% compared to logprob-based filtering without requiring model access.