Process Reward Models: The Navigator for LLM Reasoning

Generating more answers does not automatically make an LLM better. Extra compute matters only if the system can reject flawed reasoning and choose a path worth trusting. This article explains how Process Reward Models, or PRMs, work for developers interested in LLM reasoning and AI-agent verification. It covers Best-of-N, the difference between outcome and process reward models, step-score aggregation, PRM800K, and active learning.

The short version is this: the bottleneck in inference scaling is often verification accuracy, not generation volume.

The short version

An Outcome Reward Model, or ORM, evaluates the finished result. A PRM evaluates the intermediate reasoning one step at a time. In Best-of-N sampling, producing a correct candidate matters, but so does having a verifier accurate enough to find it.

There is no single rule for collapsing step scores into one score. OpenAI's PRM800K work multiplied the probability that each step was correct. Minimum and mean scores are also possible, but they can rank the same candidates differently depending on solution length and calibration.

For coding and agent systems, hard signals such as tests and execution results should come first. A PRM is less like a model that produces an answer and more like a verifier that decides where to spend a limited reasoning budget.

Inference scaling must account for both generation and selection

The familiar way to improve an LLM is to train a larger one. The relationship among model size, data, and training compute is well documented in Scaling Laws for Neural Language Models and the Chinchilla paper.

Another option is to spend more compute during inference without training a new model. This is usually called test-time compute scaling or inference scaling.

Figure 1. Parallel sampling and sequential search both need a verifier that can judge the generated paths before extra compute turns into better results.

One approach is parallel sampling: run the same prompt with different temperatures or seeds to create several candidate answers.

Question
 ├─ Candidate A
 ├─ Candidate B
 ├─ Candidate C
 └─ Candidate D
      ↓
   Verify and select
      ↓
   Final answer

The simplest selection rule is a vote. Self-Consistency samples different reasoning paths and chooses the answer that appears most consistently.

More samples also increase the chance that at least one correct answer appears. In Large Language Monkeys, DeepSeek-Coder-V2-Instruct solved 15.9% of SWE-bench Lite tasks with one attempt. With 250 samples, the proportion containing at least one correct solution rose to 56%.

That 56% is not the rate at which a verifier selected the correct answer. It is coverage: a correct answer exists somewhere in the candidate set. Generating a correct answer and identifying it are separate problems. Code has a strong discriminator in automated tests. In natural-language analysis, where no equally crisp verifier exists, selection accuracy can plateau even as the candidate pool keeps growing.

The other approach is sequential search. The system keeps several incomplete paths alive and expands only the promising ones.

Problem
  ↓
Step 1
 ├─ Step 2A → high score → keep exploring
 └─ Step 2B → low score  → stop exploring

Beam search and tree search can prune low-scoring branches early. Scaling LLM Test-Time Compute Optimally argues that inference compute should be allocated according to problem difficulty. Some problems benefit from generating more candidates. Others are better served by revising and exploring intermediate paths.

Both approaches eventually meet the same question: who decides whether an unfinished line of reasoning is headed in the right direction?

ORMs detect failure; PRMs locate where it begins

An ORM evaluates a completed solution and its final result. Consider this problem:

Find the largest natural number $k$ such that $2^k$ divides $100!$.

The answer comes from counting the factors of 2 in 100!.

Suppose a model leaves out the final term, $\lfloor100/64\rfloor=1$.

Step 1: Count the factors of 2 in 100!.
Step 2: 50 + 25 + 12 + 6 + 3 = 96
Step 3: Therefore, k = 96

An ORM can say that the final answer is wrong. It does not tell us where the failure began.

Full solution → final answer 96 → incorrect

A PRM scores each step separately.

Step 1 → 0.97
Step 2 → 0.08
Step 3 → 0.11

The sharp drop at Step 2 points to the likely origin of the error.

Figure 2. An ORM marks the final answer as wrong. A PRM can identify the first faulty step, where the $100/64$ term was omitted. The scores are illustrative.

The value estimated by a PRM can be written as:

Process supervision tells us which steps remain sound before an error appears. It can tie a failed outcome to a specific action and prune a bad partial path before completion. It may also expose answers that happen to be correct despite invalid reasoning. An early comparison of process and outcome supervision appears in Solving Math Word Problems with Process- and Outcome-Based Feedback.

A PRM scores each step in a solution

Training solutions are split at line breaks or dedicated step tokens.

Step 1: Reframe the problem as a factorization problem.
Step 2: Calculate the exponent of each prime factor.
Step 3: Add the values.

Clear boundaries determine where the model should predict a score. A human or automated verifier then judges each step. In PRM800K, +1 means the step is correct and reasonable, 0 means it is ambiguous or makes no progress without being clearly wrong, and -1 means it is incorrect or unreasonable.

Given the problem and the solution generated so far, a PRM returns the probability that the current step is correct.

Problem + Step 1               → 0.97
Problem + Step 1 + Step 2      → 0.92
Problem + Step 1 + Step 2 + 3  → 0.08

An implementation may add a separate classification head or train the probabilities of tokens that represent correct and incorrect steps. In Let's Verify Step by Step, the PRM predicts each step's correctness after the final token of that step. A single forward pass over the full solution produces all of the step scores.

Turning step scores into a single chain score

To compare candidates, the system has to reduce a sequence of step scores to one value.

A PRM does not inherently use the minimum. Let's Verify Step by Step multiplied the probabilities that all steps were correct.

In practice, summing log probabilities avoids numerical underflow.

Aggregation is a system choice, not part of the definition of a PRM. The right choice depends on score calibration and any bias introduced by solution length.

Implementing Best-of-N reranking

The following code chooses the most credible candidate from step scores that a PRM is assumed to have produced. It needs no third-party library and runs on Python 3.10 or later.

from dataclasses import dataclass
from math import log, prod
from typing import Literal


@dataclass(frozen=True)
class Candidate:
    name: str
    text: str
    step_scores: list[float]


def chain_score(
    scores: list[float],
    method: Literal["product", "minimum", "mean"] = "product",
) -> float:
    if not scores:
        raise ValueError("step_scores must not be empty")

    if any(score < 0.0 or score > 1.0 for score in scores):
        raise ValueError("every step score must be between 0 and 1")

    if method == "product":
        epsilon = 1e-8
        return sum(log(max(score, epsilon)) for score in scores)

    if method == "minimum":
        return min(scores)

    return sum(scores) / len(scores)


def select_best(
    candidates: list[Candidate],
    method: Literal["product", "minimum", "mean"] = "product",
) -> Candidate:
    if not candidates:
        raise ValueError("candidates must not be empty")

    return max(
        candidates,
        key=lambda candidate: chain_score(
            candidate.step_scores,
            method,
        ),
    )


candidates = [
    Candidate(
        name="A",
        text="Candidate A",
        step_scores=[0.98, 0.91, 0.12],
    ),
    Candidate(
        name="B",
        text="Candidate B",
        step_scores=[0.94, 0.92, 0.90],
    ),
    Candidate(
        name="C",
        text="Candidate C",
        step_scores=[0.99, 0.40, 0.96],
    ),
]

for candidate in candidates:
    probability = prod(candidate.step_scores)
    print(f"{candidate.name}: product={probability:.3f}")

selected = select_best(candidates, method="product")
print(f"selected: {selected.name}")

Expected output:

A: product=0.107
B: product=0.778
C: product=0.380
selected: B

Candidate A has two strong early scores, but its final step falls to 0.12. Multiplication makes that error sharply reduce the chain score. Candidate B has no single highest score, yet every step is consistently strong, so it wins the reranking.

Figure 3. Even when a correct answer exists in the candidate set, Best-of-N cannot improve if the verifier fails to find it.

This example does not implement a PRM. It isolates the smaller task of aggregating scores that a PRM is assumed to return and selecting a candidate. A production system also needs to test whether the scores match empirical accuracy and whether they are biased by solution length. It should compare rankings under different aggregation rules and check whether the generator and PRM share the same failure modes. If increasing the candidate count does not improve selection accuracy, the verifier is more likely to be the bottleneck than the generator.

What does PRM800K look like?

The official PRM800K repository contains model-generated solutions to MATH problems with human judgments for individual steps, stored as JSONL. The cleaned training data contains roughly 75,000 solutions and around 800,000 step-level labels.

A simplified record looks like this:

{
  "question": {
    "problem": "Find the largest k such that 2^k divides 100!."
  },
  "label": {
    "steps": [
      {
        "completions": [
          {
            "text": "Count the powers of 2 in 100!.",
            "rating": 1
          }
        ]
      },
      {
        "completions": [
          {
            "text": "50 + 25 + 12 + 6 + 3 = 96.",
            "rating": -1
          }
        ]
      }
    ],
    "finish_reason": "found_error"
  }
}

This version omits fields for readability. The real records also include metadata about labelers, generation rounds, and quality control.

PRM800K labelers did not keep searching for every error in a solution. They evaluated steps only up to the first incorrect one. This records not just that the answer failed, but where the failure began.

Active learning targets the wrong answers that fool the PRM

Step-level labeling is expensive. Checking a final answer is a different job from reading an entire solution and locating its first error. The gap grows as solutions get longer or demand specialized knowledge.

Figure 4. Active learning spends the labeling budget on wrong answers that fool the current PRM, rather than on easy negatives it already recognizes.

Active learning for a PRM begins by generating many solutions. An automatic grader finds those with incorrect final answers, and the current PRM scores them. The most useful cases are wrong solutions that still receive high scores. A human marks the first faulty step, and the new labels are used to train the PRM again.

A wrong answer with a high score is a false positive: the current PRM mistakes it for valid reasoning. These examples are usually more informative than obvious errors the model already scores poorly.

In a small proxy experiment from Let's Verify Step by Step, this selection strategy was about 2.6 times as data-efficient as uniform labeling. That number did not come from a direct comparison across the entire large-scale human-labeling effort. It was measured in a constrained experiment where a larger PRM served as the labeling oracle for a smaller one.

Reducing human step-level labels

Researchers are also studying ways to avoid labeling every step by hand. Math-Shepherd generates several continuations after a given step and estimates that step's quality from whether those continuations reach the correct final answer. Free Process Rewards without Process Labels derives implicit process rewards from outcome-level labels.

Lower labeling cost does not remove verification cost. If the automatic answer checker is wrong, its errors flow into the process labels. A flawed path that reaches the correct answer by chance may be rewarded. The generator and verifier can share blind spots, and process rewards learned on one benchmark may not transfer to another domain.

It is safer to treat automatic process supervision as a way to narrow the set that humans must review, not as a way to remove people entirely.

Layer verifiers for coding and agent systems

Mathematical answers are often easy to compare automatically. Coding, data analysis, and browser-agent tasks rarely have a single canonical answer.

A coding agent might interpret requirements, locate relevant files, plan a change, edit code, run a type checker, execute tests, and inspect regressions. Asking one PRM to judge this whole sequence is less practical than layering several kinds of verification.

Figure 5. Deterministic verifiers handle executable facts. Learned verifiers and people handle subjective judgments and high-uncertainty cases.

Execution results come first. Compilation, type checking, unit and integration tests, data schemas, security and static analysis, performance and memory usage, and regression checks are stronger evidence than a natural-language judgment.

A PRM or LLM judge can handle questions that execution does not settle: whether the requirements were interpreted correctly, whether the patch is broader than necessary, or whether a design fits the existing system. LLM judges can show position bias, prefer verbose answers, and favor outputs from their own model family. Judging LLM-as-a-Judge discusses these limitations.

People should review work that has a large effect on users or data, cases where verifiers disagree, and inputs outside the PRM's training distribution. The same is true for irreversible changes and security or financial risk. A PRM is better understood as a router for verification effort than as a judge that replaces every other signal.

What to check before using a PRM

Before adoption, decide what counts as one step. You also need to know who or what determines step correctness and which validation set will expose errors in the PRM itself. Measure whether selection accuracy rises with candidate count, and decide where a person can intervene after a bad selection.

If those questions are hard to answer, structure the task and its verification criteria before training a PRM.

PRMs need verification too

A PRM trained on mathematical solutions is not guaranteed to judge coding or data-analysis processes accurately. Like a generator, a verifier must be evaluated on the input distribution and deployment environment where it will run.

Reward hacking is another risk. Instead of solving the problem better, the generator may learn phrasing or solution formats that the PRM tends to reward. Do not optimize the PRM score in isolation; measure final execution results and use an independent evaluation set as well.

Aggregation changes rankings. Products can penalize long solutions, minimums are sensitive to one badly calibrated score, and means can wash out a fatal error. Compare actual candidate-selection accuracy instead of choosing a rule by intuition.

A PRM evaluates visible solution steps or tool-use traces. It does not directly inspect the computations happening inside the model. A high-scoring explanation should not be treated as a faithful transcript of the model's internal reasoning.

Closing

More inference compute raises the chance that a correct answer enters the candidate set. Real performance can still be capped by the verifier that has to choose it. An ORM evaluates the final result, while a PRM evaluates individual reasoning steps. Active learning concentrates labeling effort on wrong answers that the PRM scores highly. In coding and agent systems, execution checks, PRMs, and human review should be layered.

A strong reasoning system does not merely produce more answers. It notices bad paths early and redirects the remaining compute toward the paths most likely to work. A PRM is one tool for making that decision.

References

Scaling Laws for Neural Language Models, Self-Consistency Improves Chain of Thought Reasoning in Language Models, Solving Math Word Problems with Process- and Outcome-Based Feedback, Let's Verify Step by Step, official PRM800K repository, Math-Shepherd, Large Language Monkeys, Scaling LLM Test-Time Compute Optimally, Free Process Rewards without Process Labels