MTEB-PT introduces a 14-dataset benchmark to evaluate Portuguese text embeddings across STS, classification, retrieval, and reranking tasks. Testing 17 models reveals that multilingual rankings are unreliable predictors of Portuguese-specific performance and that long-context capacity is critical for retrieval tasks.
HOW THIS AFFECTS YOU
●
builderYou should not rely on multilingual benchmarks when optimizing retrieval systems for Portuguese users.
●
researcherThis provides a specialized evaluation framework for testing language-specific embedding capabilities.