Measuring LLM Belief States via Latent Variable Injection

By injecting a ring-shaped Markov chain of latent variables into natural text, researchers demonstrated that small Transformers can track Bayesian posteriors of hidden states. The study also links these belief states to the geometric arrangement of features in the model's activation space.