Opto-ViT-v2: Noise-Resilient On-Chip Fine-Tuning for Photonic Near-Sensor Vision Transformer Accelerators

Opto-ViT-v2 enables parameter-efficient fine-tuning on silicon-photonic Vision Transformer accelerators by using tensorized low-rank decomposition. This method reduces trainable parameters to as few as 8K for ViT-Base, minimizing activation storage and weight write-back requirements to MRR banks. A gradient-accumulated sparse classifier further optimizes performance by freezing low-importance weights via top-k selection.