Random matrix theory (RMT) metrics used to assess model layers are shown to be unrelated to actual performance or training. The study demonstrates that functionally preserving reparameterisations can produce arbitrary RMT metrics across various architectures and scales.
HOW THIS AFFECTS YOU
●
researcherYou should not rely on RMT-based metrics to guide pre-training or compression, as they do not reflect model function.