FOAM Provides Efficient Second-Order Optimization via SOAP Variant
November 11, 2026
FOAM is a lightweight variant of the SOAP optimizer that achieves AdamW-like efficiency by using input-only preconditioning. By dropping the output-side Kronecker factor, it reduces optimizer-state memory and eliminates one eigendecomposition per step.
HOW THIS AFFECTS YOU
●
builderYou can train larger transformers more efficiently by reducing the optimizer memory footprint.
●
researcherYou can use second-order optimization benefits with much lower memory and computational costs.