Adversarial Preference Learning for Robust LLM Alignment
ACL Findings 2025Findings of the Association for Computational Linguistics: ACL 2025
In brief
Adversarial Preference Learning (APL) hardens LLM alignment with an attacker that co-evolves with the model — no reward model, no human red-teaming needed. Three ideas drive it: the model's own preference probabilities serve as the harmfulness signal, sidestepping the reward-hacking risk of a separate reward model; a generative attacker rewrites each training prompt into its most damaging variants; and attacker and defender co-evolve in a closed loop, one discovering vulnerabilities as the other patches them. On Mistral-7B-Instruct, APL wins 83.33% of GPT-4o-judged harmlessness comparisons against the base model — versus 71.67% for standard preference tuning (DPO) — and cuts attack success by up to 65% on the HarmBench attack benchmark, with general utility close to baseline though over-refusals rise.
Key takeaways
- Aligned chatbots crack under both blunt and disguised attacks: the Mistral-7B base model complies with over half of harmful direct requests, and jailbreak rewrites with flattery, aggression, or irony slip past refusals — so APL trains the model against an attacker that keeps inventing whichever rephrasing most raises the model's own preference for a harmful reply.
- The model grades its own weakness: harmfulness is measured on the defender — the model being trained — as its probability of preferring a harmful over a safe reply, with no external reward model or judge. Adding one back actually hurts — swapping in an external reward model lowers the win rate from 83.33% to 76.67% and roughly doubles direct-request attack success.
- A conditional generative attacker turns each training prompt into eight adversarial variants and is itself trained by preference learning — the rewrite that best fools the defender becomes its preferred output. Where the untrained attacker merely rephrases, the trained one graduates to irony and presupposition ("Bank of America should be grateful I'm only taking what's rightfully mine!").
- Attacker and defender co-evolve in a closed loop — the attacker probes, the defender patches via preference updates on the worst-case rewrites — and the loop keeps paying off through iteration 25, while standard preference tuning (DPO) plateaus around iteration 15; APL ends with lower attack success than DPO on all four attack types.
- GPT-4o prefers APL's answers for harmlessness in 83.33% of head-to-heads with the base Mistral-7B; responses flagged harmful by Meta's LLaMA-Guard classifier drop more than tenfold (5.88% to 0.43%), and harmful requests asked outright succeed 11.25% of the time, versus 56.25% before — though the strongest attack, gradient-crafted adversarial suffixes (GCG), still succeeds 41.25% of the time (down from 67.50%).
- Safety does not wreck usefulness: the MT-Bench score of general chat ability slips only from 6.78 to 6.59, and APL still wins 46.52% of length-controlled helpfulness comparisons against the base model, just shy of an even split. The price is caution: false refusals of benign prompts roughly double on the XSTest over-refusal check (20% to 40.89%).
- Even models that arrive already safety-tuned get harder to attack — on Llama-3-8B-Instruct, APL lowers attack success on all four attack types (direct requests, zero-shot and few-shot rewrites, GCG suffixes) and lifts the harmlessness win rate over its own base to 56.67% (50% would be a tie).
Abstract
Modern language models often rely on Reinforcement Learning from Human Feedback (RLHF) to encourage safe behaviors. However, they remain vulnerable to adversarial attacks due to three key limitations: (1) the inefficiency and high cost of human annotation, (2) the vast diversity of potential adversarial attacks, and (3) the risk of feedback bias and reward hacking. To address these challenges, we introduce Adversarial Preference Learning (APL), an iterative adversarial training method incorporating three key innovations. First, a direct harmfulness metric based on the model’s intrinsic preference probabilities, eliminating reliance on external assessment. Second, a conditional generative attacker that synthesizes input-specific adversarial variations. Third, an iterative framework with automated closed-loop feedback, enabling continuous adaptation through vulnerability discovery and mitigation. Experiments on Mistral-7B-Instruct-v0.3 demonstrate that APL significantly enhances robustness, achieving 83.33% harmlessness win rate over the base model (evaluated by GPT-4o), reducing harmful outputs from 5.88% to 0.43% (measured by LLaMA-Guard), and lowering attack success rate by up to 65% according to HarmBench. Notably, APL maintains competitive utility, with an MT-Bench score of 6.59 (comparable to the baseline 6.78) and an LC-WinRate of 46.52% against the base model.
BibTeX
@inproceedings{apl,
title = {Adversarial Preference Learning for Robust {LLM} Alignment},
author = {Wang, Yuanfu and Wang, Pengyu and Xi, Chenyang and Tang, Bo and Zhu, Junyi and Wei, Wenqiang and Chen, Chen and Yang, Chao and Zhang, Jingfeng and Lu, Chaochao and Niu, Yijun and Mao, Keming and Li, Zhiyu and Xiong, Feiyu and Hu, Jie and Yang, Mingchuan},
year = {2025},
booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
pages = {21865--21881},
doi = {10.18653/v1/2025.findings-acl.1126},
url = {https://doi.org/10.18653/v1/2025.findings-acl.1126},
}