Adaptive Subdocument Deduplication via Frequency and Length-Aware Retention

This framework decouples duplicate detection from retention by using natural-boundary segmentation and distributed aggregation to identify subdocument groups. It applies an adaptive budget that allocates more copies to groups based on their specific frequency and length patterns, addressing cross-shard redundancy that suffix-array methods often miss.