Visual Pretraining Captures Layout and Equation Knowledge for LLMs

Visual pretraining uses raw visual representations of documents and web pages instead of converting them to plain text. This method preserves critical information found in figures, typeset equations, and complex page layouts during foundation model training.