Write a concise summary of at most a paragraph for the following:

"The paper compares two methods of training language models for complex reasoning tasks: outcome supervision and process supervision. The authors find that process supervision outperforms outcome supervision in training models to solve problems from the MATH dataset. They also show that active learning improves the efficacy of process supervision. The paper includes a dataset of 800,000 step-level human feedback labels used to train the best reward model. The text discusses the use of large language models to solve complex reasoning tasks, but these models can produce falsehoods or "hallucinations." To improve reasoning capabilities, it is important to detect and mitigate these hallucinations. One method involves training reward models to discriminate between desirable and undesirable outputs, which can be used in a reinforcement learning pipeline or to perform search via rejection sampling. However, the resulting system is only as reliable as the reward model itself, so it is important to study how to train reliable reward models. The text describes two distinct methods for training reward models: outcome supervision and process supervision. Outcome-supervised reward models (ORMs) are trained using only the final result of the model's chain-of-thought, while process-supervised reward models (PRMs) receive feedback for each step in the chain-of-thought. Process supervision has several advantages, including providing more precise feedback and mitigating misaligned behavior. The paper presents a detailed comparison of outcome and process supervision, using a more capable base model, significantly more human feedback, and the more challenging MATH dataset. The main contributions are that process supervision can train much more reliable reward models, a large reward model can reliably approximate human supervision for smaller reward models, active learning leads to a significant improvement in data efficiency, and the release of a full process supervision dataset for related research. The paper presents a method for training a reward model for a large-scale math problem solver using a combination of human-labeled data and reinforcement learning. The human-labeled data is collected by presenting human data-labelers with step-by-step solutions to math problems and asking them to assign each step a label of positive, negative, or neutral. The data collected is referred to as PRM800K and contains 800K step-level labels across 75K solutions to 12K problems." The word "base" should not appear in your response.
user