人工知能：DeepSeek-R1 AIモデルの背後にある科学

The methodology used to train a large-scale reasoning model in DeepSeek-R1, the open artificial intelligence (AI) model released earlier this year, is published in Nature this week. The reasoning ability of the large language model (LLM) is shown to be improved by pure reinforcement learning, reducing the amount of human input needed to enhance the performance of the model. The resulting model performs better at tasks such as mathematics, coding competitions, and STEM field graduate-level problems than conventionally trained LLMs.

Teaching AI models to reason in the same way as humans has been challenging. LLMs have demonstrated some reasoning abilities, but training requires substantial computational resources. Such models can be improved by human prompts to encourage models to produce intermediate reasoning steps, thereby substantially enhancing their performance on complex tasks. However, this approach can become computationally expensive and limits scaling potential.

DeepSeek-R1 includes a stage of further training under human supervision to improve the reasoning process. The model uses reinforcement learning instead of human examples to develop reasoning steps, which reduces training costs and complexity, Zhen Zhang and colleagues report. DeepSeek-R1 is shown good examples of problem solving and is then given a template to produce a reasoning process. The model is rewarded for solving problems, thus reinforcing the learning. In benchmark mathematics evaluation tests designed to evaluate AI performance, DeepSeek-R1-Zero and DeepSeek-R1 achieved scores of 77.9% and 79.8%, respectively. The model also achieves favourable performance in coding competitions and graduate-level biology, physics and chemistry problems, the authors add.

The current version of DeepSeek-R1 has some capability limitations, which the authors hope to address in future iterations. For instance, the model sometimes mixes languages and is currently only optimized for Chinese and English. It is also sensitive to prompts, requiring thoughtful prompt engineering, and there are some tasks that the model is yet to show notable improvements on, such as software engineering. Finally, future research should focus on improving the reward process to ensure that the reasoning and outcomes of tasks are reliable, the authors conclude.