about
The Differences Between Direct Alignment Algorithms Are a Blur (arxiv.org)
8 points by t55 on Feb 9, 2025 | hide | past | pdf | discuss on HN

In plain words: They compared ways to train a chatbot to prefer good answers without a reward model, holding all else fixed so the only difference was comparing two answers versus scoring one alone. Comparing pairs, not scoring singles, drove the gains; the score formula mattered little.

Abstract · The Differences Between Direct Alignment Algorithms are a Blur

Direct Alignment Algorithms (DAAs) simplify LLM alignment by directly optimizing policies, bypassing reward modeling and RL. While DAAs differ in their use of SFT (one-stage vs. two-stage) and the scalar score they optimize (likelihood vs. odds ratios), the key performance drivers remain underexplored. We present a systematic comparison and analyze a previously overlooked axis - the ranking objective (pairwise vs. pointwise). To isolate this factor, we propose a unified training framework across DAAs by (i) converting one-stage methods (ORPO, ASFT) into a two-stage pipeline with an explicit SFT phase and (ii) introducing a $β$ parameter that places all methods in the same hyperparameter space and improves the quality of odds-ratio DAAs (ORPO, ASFT). Under this setup, the ranking objective emerges as the primary determinant of alignment quality, whereas the particular scalar score (policy-reference ratio vs. odds ratio) is secondary. We corroborate this on instruction-following tasks and further confirm it on math-reasoning benchmarks across model scales. Evidence suggests that this stems from how these objectives interact with prompt-specific biases, supported both by strictly controlled experiments and by observations on real data. Our findings underscore the need for nuanced evaluations in DAA research to avoid oversimplified claims of superiority.

Alexey Gorbatovski, Boris Shaposhnikov, Viacheslav Sinii, Alexey Malakhov, Daniil Gavrilov
arXiv:2502.01237 · cs.LG · submitted Feb 3, 2025 · updated May 8, 2026
abstract · pdf · html

add comment on HN