Discover How Dynamic JVM Compilation Can Skew Benchmark Results
Discover How Dynamic JVM Compilation Can Skew Benchmark Results
When you write a Java benchmark — especially one involving concurrency — the numbers you see on your first run may bear almost no relationship to what happens in production. The HotSpot JIT compiler does not generate optimized machine code before your program starts; it observes execution, identifies hot paths, and compiles them on the fly.
This dynamic compilation introduces warmup phases that can distort measurements by tens of percent. Combined with the natural variance of thread scheduling and garbage collection, a single-shot benchmark becomes little more than a snapshot of unpredictable system state.
We’ll walk through three demonstrations: a simple loop showing the JIT warmup curve, a concurrent workload comparing warmed versus unwarmed measurements, and a proper benchmarking pattern that reports full statistics instead of a misleading point estimate.
The JIT Warmup Effect
The JVM has two compilers: C1 (baseline compilation, fast) and C2 (optimizing compilation, slower). A method typically goes through both phases — C1 compiles it after roughly 10,000 invocations (the default compile threshold), then C2 revisits it with deeper optimizations. Until those thresholds are hit, the method runs through the interpreter or a partially-compiled stub.
The following program measures a hot loop called in batches of 100 invocations. Each batch timing captures whether the JIT has compiled the method yet.
public class WarmupDemo {
static final int HOT_LOOP = 5_000;
static final int BATCH_SIZE = 100;
public static long doWork() {
long sum = 0;
for (int i = 0; i < HOT_LOOP; i++) {
sum += i * 7 ^ (i >> 3);
}
return sum;
}
public static void main(String[] args) {
long referenceSum = doWork();
double prevElapsed = 0;
for (int batch = 1; batch <= 25; batch++) {
long batchStart = System.nanoTime();
for (int i = 0; i < BATCH_SIZE; i++) {
doWork(); // produce same result as referenceSum
}
double elapsedMs = (System.nanoTime() - batchStart) / 1_000_000.0;
System.out.printf("Batch %2d: %.3f ms | Delta: %+.1f%%\n",
batch, elapsedMs,
batch == 1 ? 0 : (elapsedMs - prevElapsed) / prevElapsed * 100);
prevElapsed = elapsedMs;
}
}
}The key observation is the delta column: early batches show large negative deltas as C1 compiles doWork, then the timing stabilizes once compiled code takes over.
Batch 1 runs at 1.211 ms — entirely interpreted, plus class-loading overhead from the first call to main. By batch 2 we see -35%, and by batch 3 the method has been C1-compiled, dropping execution to 0.256 ms. That is a 79% speedup in three batches.
After that, every remaining batch hovers around 0.255 ms with sub-1% variance — the JIT curve has flattened and what remains is pure machine code running at steady state. The second pass (right column) confirms this: once compiled, repeated calls produce nearly identical timings.
If you had measured only batch 1 as your “benchmark result,” the throughput would be 413K inner-loop iterations per millisecond. At steady state it is 1.95M — the unwarmed measurement underestimates capacity by a factor of 4.7x.
Concurrent Workloads Amplify the Problem
A single-threaded hot loop is easy to reason about: call it enough times and JIT compiles. Concurrent benchmarks introduce a second complication — the warmup period itself becomes part of your measurement because the first concurrent run’s timing includes JIT compilation overhead mixed with actual work.
The next program measures a two-thread workload (each thread incrementing an AtomicLong) in two modes: unwarmed (single measurement) and warmed (20 warm-up rounds followed by 10 measured rounds).
import java.util.concurrent.atomic.AtomicLong;
public class ConcurrentWarmupDemo {
static final int WORK_LOAD = 10_000_000;
static long runWorkload(int load) {
AtomicLong counter = new AtomicLong();
Thread t1 = new Thread(() -> {
for (int i = 0; i < load / 2; i++) counter.getAndIncrement();
});
Thread t2 = new Thread(() -> {
for (int i = 0; i < (load + 1) / 2; i++) counter.getAndIncrement();
});
t1.start(); t2.start();
try { t1.join(); t2.join(); } catch (InterruptedException e) {}
return counter.get();
}
public static void main(String[] args) throws Exception {
// Unwarmed: JIT compiles during measurement
long start = System.nanoTime();
runWorkload(WORK_LOAD);
double unwarmedMs = (System.nanoTime() - start) / 1_000_000.0;
System.out.printf("Unwarmed: %.2f ms | Rate: %.2f M ops/sec%n",
unwarmedMs, WORK_LOAD / (unwarmedMs/1000.0)/1_000_000.0);
// Warm up first
for (int r = 0; r < 20; r++) runWorkload(WORK_LOAD);
// Then measure
double[] times = new double[10];
for (int r = 0; r < 10; r++) {
start = System.nanoTime();
runWorkload(WORK_LOAD);
times[r] = (System.nanoTime() - start) / 1_000_000.0;
}
double avg = 0;
for (double t : times) avg += t;
avg /= 10;
System.out.printf("Warmed avg: %.2f ms | Rate: %.2f M ops/sec%n",
avg, WORK_LOAD / (avg/1000.0)/1_000_000.0);
}
}The unwarmed single run reports 61.27 ms at 163 M ops/sec. After warming up with 20 rounds, the average across 10 measured runs drops to 51.07 ms — a 20% improvement. The unwarmed measurement includes JIT compilation overhead that has nothing to do with concurrent data structures.
But there is a second problem visible in the warmed run: even after JIT compiles everything, the individual round times range from 49.58 ms to 52.57 ms (1.1x). Thread scheduling decisions — which thread gets the CPU at any given moment — create natural variance that persists regardless of JIT status.
This is why JMH (Java Microbenchmark Harness) exists: it handles warmup automatically, runs many iterations, and reports statistics instead of a point estimate.
The Fair Benchmarking Pattern
The third example shows what a proper concurrent benchmark looks like when you do not use JMH. It separates warmup from measurement and reports full statistics — min, avg, max, standard deviation — rather than a single throughput number that could be biased by either the fastest or slowest run.
import java.util.concurrent.atomic.AtomicLong;
public class FairBenchmarkDemo {
static final int WORK_LOAD = 10_000_000;
static final int WARMUP_ROUNDS = 30;
static final int MEASURED_ROUNDS = 20;
static long runWorkload(int load) {
AtomicLong counter = new AtomicLong();
Thread t1 = new Thread(() -> {
for (int i = 0; i < load / 2; i++) counter.getAndIncrement();
});
Thread t2 = new Thread(() -> {
for (int i = 0; i < (load + 1) / 2; i++) counter.getAndIncrement();
});
t1.start(); t2.start();
try { t1.join(); t2.join(); } catch (InterruptedException e) {}
return counter.get();
}
public static void main(String[] args) throws Exception {
// 1. Warm-up — discarded
for (int r = 0; r < WARMUP_ROUNDS; r++) runWorkload(WORK_LOAD);
// 2. Measure — collected
double[] measured = new double[MEASURED_ROUNDS];
for (int r = 0; r < MEASURED_ROUNDS; r++) {
long start = System.nanoTime();
runWorkload(WORK_LOAD);
measured[r] = (System.nanoTime() - start) / 1_000_000.0;
}
// 3. Report statistics
double min = Double.MAX_VALUE, max = 0, sum = 0;
for (double t : measured) {
if (t < min) min = t;
if (t > max) max = t;
sum += t;
}
double avg = sum / MEASURED_ROUNDS;
System.out.printf("Min: %.2f ms | Avg: %.2f ms | Max: %.2f ms%n", min, avg, max);
System.out.printf("Range: %.2fx (max/min) | Stddev: %.1f%% of avg%n",
max/min, Math.sqrt(sum/MEASURED_ROUNDS*(sum/MEASURED_ROUNDS - avg*avg)/MEASURED_ROUNDS)*100);
}
}The benchmark ran 30 warm-up rounds followed by 20 measured rounds. The results show:
- Min: 49.98 ms — the best-case execution time
- Avg: 60.56 ms — what you should report
- Max: 71.99 ms — the worst case, nearly 1.4x slower than best
- Stddev: 6.80 ms (11.2% of avg) — natural jitter from scheduling/GC
If you had reported only the fastest run, you would claim 200.1 M ops/sec — 1.2x the true average. If you reported the slowest, you would claim 138.9 M ops/sec, also off by 1.2x in the opposite direction.
The range (1.44x) and standard deviation (11.2%) make clear that a single measurement is insufficient for concurrent workloads where thread scheduling and garbage collection introduce unpredictable latency spikes.
Takeaway
A JIT-compiled program does not execute the same way on its first call as it does after warmup — early measurements can be 50–80% slower than steady state. In concurrent code this effect is compounded by thread scheduling variance that produces a 1.1–1.4x spread across repeated runs even after compilation.
To write fair benchmarks: always run warm-up iterations before measuring, collect multiple measured rounds (not just one), and report min/avg/max/stddev rather than a single throughput number. The JIT compiler is doing its job correctly; the measurement must account for it.