Metric-based Loss Weighting improves multimodal machine translation grounding
September 28, 2026
A new training method uses Point-wise Cross-mutual Information (PCXMI) to identify tokens that benefit from visual context and increases their loss weight. This approach improves visual sensitivity and grounding in pretrained Multimodal Large Language Models during fine-tuning.
HOW THIS AFFECTS YOU
●
builderYou can use this method to reduce instances where your multimodal models ignore image inputs.
●
researcherThe PCXMI metric provides a more granular way to weight multimodal loss functions.