A Comparative Study of Evaluation Metrics for Long-Document Financial Narrative Summarization with Transformers

This study benchmarks transformer-based summarization methods on the FNS 2020 dataset of UK annual reports. The authors introduce BRUGEscore, a metric calculated as the harmonic mean of ROUGE-2 and BERTscore, to address inconsistencies in how traditional metrics reflect summarization quality in long-form financial narratives.