Does Refusal Training in LLMs Generalize to the Past Tense?

When you think about training a large language model (LLM) to refuse harmful requests, you might assume that once it's trained, it should be able to handle any form of those requests. But what if you just change the tense of the request? This is the question Maksym Andriushchenko and Nicolas Flammarion set out to answer in their paper, "Does Refusal Training in LLMs Generalize to the Past Tense?"

Goal

The main goal of their research is straightforward: they want to see if refusal training in LLMs holds up when harmful requests are reformulated in the past tense. The fundamental concept here is that there's a significant gap in how well these models generalize. Simply put, changing a harmful request to the past tense can often trick the model into bypassing its refusal mechanisms.

Methodology

To test this, the researchers used a systematic approach. They evaluated various LLMs, including Llama-3 8B, Claude-3.5 Sonnet, GPT-3.5 Turbo, Gemma-2 9B, Phi-3-Mini, GPT-4o-mini, GPT-4o, and R2D2. They used GPT-3.5 Turbo to automatically convert harmful requests into the past tense and then tested these reformulated requests against the models' refusal mechanisms.

Evaluation

Their evaluation was thorough. They took 100 harmful requests from JailbreakBench and tried to reformulate each one 20 times. They measured the success rate of these attacks using different judges, including GPT-4, Llama-3 70B, and a rule-based heuristic.

Findings

• Vulnerability: Past tense reformulations can effectively bypass refusal mechanisms.

• Success Rate: Past tense reformulations significantly increased the success rate of attacks. For instance, the success rate on GPT-4o jumped from 1% with direct requests to 88% with past tense reformulations.

• Mitigation: Fine-tuning LLMs with explicit past tense examples can mitigate this vulnerability, though it can also increase the overrefusal rate.

Experimental Setup

• Reformulations: Generated using GPT-3.5 Turbo and tested on various LLMs.

• Results: Fine-tuning with past tense examples brought the attack success rate down to 0%, but it also increased the overrefusal rate.

Advantages

• Provides a simple yet effective method to test the robustness of LLM refusal training.

• Offers practical insights into improving refusal mechanisms through targeted fine-tuning.

Limitations

• Fine-tuning may lead to overrefusals if not carefully balanced.

• Focuses on past tense reformulations; other types of reformulations or adversarial attacks may still pose challenges.

Conclusion

Andriushchenko and Flammarion's paper highlights a critical gap in current LLM refusal training methods. Past tense reformulations can effectively bypass refusal mechanisms, revealing a significant vulnerability. While fine-tuning with past tense examples can mitigate this issue, it requires careful balancing to avoid overrefusals. These findings underscore the need for more robust and comprehensive alignment techniques to address such vulnerabilities in LLMs.

[READ FULL PAPER]