Recursive Introspection: Teaching Language Model Agents How to Self-Improve

The paper "Recursive Introspection: Teaching Language Model Agents How to Self-Improve" by Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar tackles a fascinating problem: how to make large language models (LLMs) better at learning from their mistakes. The goal is ambitious but clear—develop a method that allows LLMs to introspect and refine their responses over multiple turns, even when they initially get things wrong.

Core Idea

The core idea here is to enable LLMs to recursively detect and correct their mistakes through an iterative fine-tuning process. This concept draws inspiration from online imitation learning and reinforcement learning. Essentially, the researchers want these models to learn how to learn, which is a pretty big deal.

Methodology: RISE

They introduce a methodology called RISE (Recursive IntroSpEction). The trick with RISE is to transform a single-turn prompt into a multi-turn Markov decision process (MDP). Think of it like this: the initial state is the prompt, and the model keeps refining its responses based on previous attempts and optional feedback from the environment. It's like giving the model multiple chances to get it right, each time learning from its past mistakes.

Data Collection

To collect data for this process, they use on-policy rollouts. This means the model generates multiple sequential attempts at solving a problem. Improved responses are obtained either by querying a more capable model (distillation) or by sampling multiple responses from the learner itself (self-distillation). The model is then fine-tuned using a reward-weighted regression objective, which learns from both high- and low-quality parts of the rollouts. This iterative process aims to instill a general self-improvement capability in the LLM.

Deployment Modes

RISE can operate in two modes: with oracle (early termination upon correct response) or without oracle (majority voting on candidate outputs from different turns). This flexibility is one of the distinctive features of RISE.

Novel Techniques

RISE introduces some novel techniques. The iterative fine-tuning procedure that treats single-turn prompts as multi-turn MDPs is particularly innovative. This approach is especially effective for tasks requiring logical reasoning and complex problem-solving, where direct one-shot attempts often fail. The use of reward-weighted regression and on-policy rollouts for training also sets RISE apart from other methods that rely solely on static datasets or single-turn improvements.

Experimental Results

The researchers tested their approach on mathematical reasoning tasks using datasets like GSM8K and MATH. They fine-tuned models such as Llama2, Llama3, and Mistral using RISE. The results were impressive. For instance, Llama2-7B showed a 17.7% improvement over five turns on GSM8K, while Mistral-7B improved by 23.9%. The approach also demonstrated scalability, with larger benefits observed for more capable models.

Limitations

Of course, no method is without its drawbacks. RISE requires significant computational resources for multiple iterations of training. It currently relies on manual iterations, suggesting a need for more automated or online variants for scalability. But despite these limitations, RISE enables LLMs to self-improve iteratively, outperforming single-turn strategies with equal computational resources. It generalizes well to out-of-distribution prompts and does not degrade first-turn performance.

Conclusion

In conclusion, RISE presents a robust method for enabling LLMs to introspect and improve their responses over multiple turns. It leverages iterative fine-tuning inspired by reinforcement learning principles, resulting in significant performance gains on reasoning tasks. While effective, the approach's computational demands highlight the need for further optimization and automation in future work.

[READ FULL PAPER]