about
Alvorada-Bench: Can Language Models Solve Brazilian University Entrance Exams? (arxiv.org)
1 point by henriquegodoy on Aug 25, 2025 | hide | past | pdf | discuss on HN

In plain words: A test set of 4,515 questions from five Brazilian university entrance exams was used to check how well 20 language models handle Portuguese, culture, and reasoning under three prompting styles. The best models scored above 94% overall but fell short on math and the engineering-focused exams, showing multi-step reasoning remains their weak spot.

Abstract

Language models are increasingly used in Brazil, but most evaluation remains English-centric. This paper presents Alvorada-Bench, a 4,515-question, text-only benchmark drawn from five Brazilian university entrance examinations. Evaluating twenty models under zero-shot, role-playing, and chain-of-thought prompting, producing 270,900 responses with structured self-reports of confidence, perceived difficulty, and Bloom level. The top models exceed 94% accuracy overall, but accuracy declines on Mathematics and on the engineering oriented IME and ITA exams, indicating persistent weaknesses in multi-step reasoning. Confidence is well calibrated and correlates with perceived difficulty, revealing that models can accurately assess their own certainty capabilities. A cost accuracy analysis shows that high accuracy is achievable at under $2 per 1K tokens. On ENEM 2024 the top model (O3) achieved perfect scores in Languages subject questions while even the weakest system (GPT-4.1 Nano) only underperforms humans in Mathematics. Through exams that distill decades of Brazilian educational priorities and assess millions of students yearly, Alvorada-Bench establishes whether language models can navigate the intersection of language, culture, and reasoning that defines academic readiness in Brazil.

Henrique Godoy
arXiv:2508.15835 · cs.CL, cs.AI · submitted Aug 19, 2025
abstract · pdf · html

add comment on HN