MiniCPM RL Team
<aside> π
When training small LLMs with long chain-of-thought (CoT) reinforcement learning, we observe that performance gains are difficult to sustain: training may improve at intermediate stages but degrade later in the run. We trace this instability to two interacting causes: 1) the open-sourced data contains noisy or unverifiable labels and problems whose difficulty is poorly matched to the base model, leading to sparse or misleading reward signals; 2) over extremely long reasoning trajectories, the GRPO-style group-average baseline remains a coarse response-level reference, providing limited information for token-level credit assignment and leaving later-stage training vulnerable to performance degradation.
To address these challenges, we develop a critic-equipped GRPO recipe, named JustRL II for long-CoT RL in small LLMs. On the data side, a three-stage pipeline audits and recalibrates the open-source data to the target checkpoint. On the algorithm side, we retain the group structure of GRPO, while introducing a learned value function to provide token-level credit assignment. We make a set of modifications to make this combination work reliably over long trajectories, and develop a diagnostic framework built upon the group structure to audit the critic throughout training.
This blog uses mathematical reasoning as a detailed case study to walk through the training recipe. The same complete recipe has also been applied to MiniCPM5-2B, yielding clear gains across multiple domains and reaching state-of-the-art performance among small LLMs. We also open-source the checkpoint and reference training code.
</aside>
Figure 1. RL lifts MiniCPM5-2B past a model twice its size. Left: AIME 2025 accuracy during RL on the same data and compute. The critic-free baseline tops out at 74% around step 154 and drifts afterwards; JustRL II (ours) keeps improving and reaches 81% by step 334. Right: Dashed red is the SFT starting point, solid red is after our RL recipe. All six benchmarks improve significantly. The post-RL 2B model overtakes Qwen3.5-4B on AIME 2025, LiveCodeBench v6 and IFBench, and roughly matches it elsewhere. Axes are normalized to the maximum shown at each vertex; inner rings are compressed.
Opensourced: base model checkpoint
We begin with a standard GRPO recipe described in JustRL[1] as our baseline. It uses MiniCPM5-2B, initialized from a long-reasoning checkpoint, and an initial dataset of roughly 100k problems formed by merging the open-source DAPO-Math[2], DeepScaleR[3], and DeepMath[4] datasets. We sample eight rollouts per prompt with a 128k-token generation budget. This run provides a reference for diagnosing which parts of the recipe become fragile in the long-CoT setting.
We first recall the GRPO objective. For each prompt $q$, GRPO samples a group of $n$ responses, denoted by $G(q) = {y_1, \ldots, y_n}$, and obtains one outcome reward $r_i$ for each response. The response-level advantage is the reward centered by the group mean, broadcast unchanged to every token:
$$ \hat A_i = r_i - \bar r, \qquad \bar r = \frac{1}{n}\sum_{j=1}^{n} r_j, \qquad \hat A_{i,t} = \hat A_i. $$
The policy is then updated with the standard clipped surrogate objective.
This GRPO baseline also uses dynamic sampling. For each prompt, we sample a group of responses and retain it only when the sampled rewards have nonzero variance. We oversample candidate groups and continue sampling until enough nonzero-variance groups are collected for the update.
With this objective and the original dataset, the baseline makes clear progress in the long-CoT setting. Held-out AIME 2025 performance improves throughout the early and middle stages of training, showing that the standard GRPO recipe can learn useful behavior from this setup. However, the baseline makes clear progress early but the gains are not sustained.
Figure 2. Over the first 150 steps AIME 2025 climbs from roughly 61% to 74%; across the following 250 steps it gains nothing further.
We therefore conduct a detailed diagnosis of the training dynamics, focusing on response length, truncation and repetition, and the fraction of zero variance groups:
Figure 3. Diagnostic views of the standard GRPO baseline during long-CoT training. The panels show, in reading order, held-out AIME 2025 response length, repetition fraction, truncation ratio, and the fraction of sampled groups discarded(the zero-variance drop ratio). At later training steps, responses get longer, and truncation and repetition become more frequent. A large fraction of sampled groups are zero-variance and discarded, wasting substantial rollout compute.
Three observations stand out in the baseline run. We discuss them separately:
Evaluation improves, then plateaus. AIME 2025 accuracy rises from about 61 to 74 by step 150, but remains around it till step 400.
Response length, truncation, and repetition move together. Mean response length stays near 30k tokens during the first 100 steps, then climbs to about 50k by step 200 and remains around 45β50k afterward. As responses become longer, truncation rises from around 1% to 6β8%, meaning that more rollouts hit the generation limit; repetition also increases to 2β3%.
Many rollouts are discarded. Zero-variance groups account for roughly 55β70% of sampled groups throughout the run. Since each group contains eight complete rollouts but is discarded before the update, more than half of the group-level rollout budget produces no policy gradient.