datmt.com

SSD Benchmarking with fio: Throughput and IOPS, Test-File Only

Sat Jul 25 2026

Want real numbers on your SSD’s speed? fio (Flexible I/O Tester) is the standard tool. This post benchmarks sequential and random read/write performance on a real NVMe SSD, using only regular test files on an existing filesystem: never a raw device path like /dev/sdX or /dev/nvme0n1.

Why test files, not raw devices

Pointing fio at /dev/sdX benchmarks the whole block device directly, bypassing the filesystem entirely. Two problems:

  • It’s destructive. fio with rw=write against a raw device overwrites whatever partition table, filesystem, or data lives there. No undo.
  • It’s not what your apps see. Your database, container runtime, and app server all read/write through a filesystem. Benchmarking the bare device measures something you don’t actually use.

The safe, realistic approach: create a directory on the target filesystem, point fio’s --filename at a regular file inside it, give it a --size, and let fio create and fill that file itself. Same underlying SSD hardware, same driver stack, zero risk to existing partitions.

Machine used for this run: 16 vCPU, ext4 root filesystem on an NVMe SSD (/dev/nvme0n1p4), fio 3.42.

$ mkdir -p /home/dat/fio-test
$ df -hT /home/dat/data
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme0n1p4 ext4  938G  725G  165G  82% /
/dev/nvme0n1p4 on / type ext4 (rw,relatime)

The core fio flags

  • --filename=testfile: a plain file, not a device node
  • --size=2G: how big the test file is; fio creates/pads it as needed
  • --direct=1: bypass the page cache so you measure the disk, not RAM
  • --ioengine=libaio: async I/O engine, needed to drive real queue depth
  • --iodepth=N: how many I/Os in flight at once (1 for sequential, 32 for IOPS tests)
  • --rw=: access pattern (write, read, randwrite, randread, randrw)
  • --bs=: block size, large (1M) for throughput tests, small (4k) for IOPS tests
  • --runtime= + --time_based: run for a fixed wall-clock duration instead of until size is exhausted

Sequential write: throughput

Large blocks, low queue depth: this is the “how fast can it write a big file” number.

$ fio --name=seqwrite --filename=testfile --size=2G --rw=write \
    --bs=1M --direct=1 --ioengine=libaio --iodepth=4 --numjobs=1 --group_reporting

seqwrite: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=4
fio-3.42
Starting 1 process
seqwrite: Laying out IO file (1 file / 2048MiB)

seqwrite: (groupid=0, jobs=1): err= 0: pid=2074955: Sat Jul 25 22:36:56 2026
  write: IOPS=4240, BW=4240MiB/s (4446MB/s)(2048MiB/483msec)
    slat (usec): min=9, max=126, avg=22.09, stdev= 9.89
    clat (usec): min=173, max=51482, avg=918.71, stdev=2364.83
     lat (usec): min=188, max=51500, avg=940.81, stdev=2365.06
    clat percentiles (usec):
     |  1.00th=[  277],  5.00th=[  334], 10.00th=[  404], 20.00th=[  537],
     | 30.00th=[  644], 40.00th=[  660], 50.00th=[  734], 60.00th=[  791],
     | 70.00th=[  832], 80.00th=[  881], 90.00th=[  996], 95.00th=[ 1319],
     | 99.00th=[ 5211], 99.50th=[ 6259], 99.90th=[51119], 99.95th=[51119],
     | 99.99th=[51643]
   bw (  MiB/s): min= 4096, max= 4096, per=96.60%, avg=4096.00, stdev= 0.00, samples=1
   iops        : min= 4096, max= 4096, avg=4096.00, stdev= 0.00, samples=1
  lat (usec)   : 250=0.68%, 500=16.65%, 750=36.43%, 1000=36.67%
  lat (msec)   : 2=6.79%, 4=1.32%, 10=1.12%, 20=0.15%, 100=0.20%
  cpu          : usr=3.73%, sys=6.43%, ctx=2195, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2048,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0.00ns, window=0.00ns, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=4240MiB/s (4446MB/s), 4240MiB/s-4240MiB/s (4446MB/s-4446MB/s), io=2048MiB (2147MB), run=483-483msec

Disk stats (read/write):
  nvme0n1: ios=3/6205, sectors=24/3174888, merge=0/121, ticks=0/5195, in_queue=5246, util=70.60%

~4.4 GB/s sequential write. Note util=70.60%: the disk wasn’t even fully saturated at queue depth 4, a fast NVMe drive laughs at 1M sequential writes.

Sequential read: throughput

Same file, read back sequentially.

$ fio --name=seqread --filename=testfile --size=2G --rw=read \
    --bs=1M --direct=1 --ioengine=libaio --iodepth=4 --numjobs=1 --group_reporting

seqread: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=4
fio-3.42
Starting 1 process

seqread: (groupid=0, jobs=1): err= 0: pid=2075216: Sat Jul 25 22:37:00 2026
  read: IOPS=6360, BW=6360MiB/s (6669MB/s)(2048MiB/322msec)
    slat (usec): min=9, max=265, avg=19.41, stdev=12.07
    clat (usec): min=288, max=5089, avg=607.95, stdev=235.65
     lat (usec): min=308, max=5326, avg=627.36, stdev=242.09
    clat percentiles (usec):
     |  1.00th=[  363],  5.00th=[  433], 10.00th=[  474], 20.00th=[  519],
     | 30.00th=[  537], 40.00th=[  545], 50.00th=[  570], 60.00th=[  594],
     | 70.00th=[  635], 80.00th=[  652], 90.00th=[  750], 95.00th=[  840],
     | 99.00th=[ 1270], 99.50th=[ 1942], 99.90th=[ 4080], 99.95th=[ 4080],
     | 99.99th=[ 5080]
  lat (usec)   : 500=13.87%, 750=75.88%, 1000=9.08%
  lat (msec)   : 2=0.78%, 4=0.20%, 10=0.20%
  cpu          : usr=0.93%, sys=10.87%, ctx=2046, majf=0, minf=1034
  IO depths    : 1=0.1%, 2=0.1%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2048,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0.00ns, window=0.00ns, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=6360MiB/s (6669MB/s), 6360MiB/s-6360MiB/s (6669MB/s-6669MB/s), io=2048MiB (2147MB), run=322-322msec

Disk stats (read/write):
  nvme0n1: ios=3647/18, sectors=1867264/2920, merge=0/0, ticks=1924/12, in_queue=1935, util=53.41%

~6.7 GB/s sequential read: reads outrun writes here, typical for NVMe (no erase-before-write cost on the read path).

Random write: IOPS

Small blocks (4k), deep queue (32): this is what stresses the SSD’s controller and FTL, the number that matters for databases and VMs doing scattered small I/O.

$ fio --name=randwrite --filename=testfile --size=2G --rw=randwrite \
    --bs=4k --direct=1 --ioengine=libaio --iodepth=32 --numjobs=1 \
    --runtime=15 --time_based --group_reporting

randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.42
Starting 1 process

randwrite: (groupid=0, jobs=1): err= 0: pid=2075281: Sat Jul 25 22:37:17 2026
  write: IOPS=145k, BW=568MiB/s (595MB/s)(8517MiB/15002msec)
    slat (nsec): min=490, max=1403.8k, avg=1607.41, stdev=2612.30
    clat (usec): min=2, max=100068, avg=218.27, stdev=1414.66
     lat (usec): min=4, max=100069, avg=219.88, stdev=1414.67
    clat percentiles (usec):
     |  1.00th=[   14],  5.00th=[   47], 10.00th=[   59], 20.00th=[   70],
     | 30.00th=[   91], 40.00th=[  112], 50.00th=[  145], 60.00th=[  151],
     | 70.00th=[  155], 80.00th=[  167], 90.00th=[  223], 95.00th=[  537],
     | 99.00th=[ 1450], 99.50th=[ 1827], 99.90th=[ 6587], 99.95th=[49546],
     | 99.99th=[51643]
   bw (  KiB/s): min=78504, max=811704, per=100.00%, avg=581442.67, stdev=210306.79, samples=30
   iops        : min=19626, max=202926, avg=145360.67, stdev=52576.70, samples=30
  lat (usec)   : 4=0.01%, 10=0.88%, 20=1.01%, 50=3.82%, 100=30.78%
  lat (usec)   : 250=54.51%, 500=3.67%, 750=1.72%, 1000=1.09%
  lat (msec)   : 2=2.06%, 4=0.29%, 10=0.09%, 20=0.01%, 50=0.03%
  lat (msec)   : 100=0.04%, 250=0.01%
  cpu          : usr=5.54%, sys=23.43%, ctx=1279580, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2180410,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0.00ns, window=0.00ns, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=568MiB/s (595MB/s), 568MiB/s-568MiB/s (595MB/s-595MB/s), io=8517MiB (8931MB), run=15002-15002msec

Disk stats (read/write):
  nvme0n1: ios=2/2157112, sectors=16/17283096, merge=0/2034, ticks=60/409444, in_queue=412165, util=86.61%

~145k IOPS random write, 568 MiB/s at 4k block size. Notice the IOPS swing (min 19.6k, max 202.9k across samples): random-write performance on real SSDs isn’t flat, it wobbles with garbage collection and write amplification under the hood.

Random read: IOPS

$ fio --name=randread --filename=testfile --size=2G --rw=randread \
    --bs=4k --direct=1 --ioengine=libaio --iodepth=32 --numjobs=1 \
    --runtime=15 --time_based --group_reporting

randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.42
Starting 1 process

randread: (groupid=0, jobs=1): err= 0: pid=2076295: Sat Jul 25 22:37:38 2026
  read: IOPS=46.3k, BW=181MiB/s (190MB/s)(2712MiB/15002msec)
    slat (nsec): min=431, max=2239.3k, avg=1779.24, stdev=3344.11
    clat (usec): min=121, max=22467, avg=689.12, stdev=361.45
     lat (usec): min=124, max=22470, avg=690.90, stdev=361.39
    clat percentiles (usec):
     |  1.00th=[  247],  5.00th=[  379], 10.00th=[  408], 20.00th=[  506],
     | 30.00th=[  537], 40.00th=[  627], 50.00th=[  652], 60.00th=[  685],
     | 70.00th=[  766], 80.00th=[  807], 90.00th=[  922], 95.00th=[ 1045],
     | 99.00th=[ 1663], 99.50th=[ 2245], 99.90th=[ 4948], 99.95th=[ 5866],
     | 99.99th=[ 9896]
   bw (  KiB/s): min=106800, max=266736, per=100.00%, avg=185162.13, stdev=36913.77, samples=30
   iops        : min=26700, max=66684, avg=46290.53, stdev=9228.44, samples=30
  lat (usec)   : 250=1.10%, 500=16.60%, 750=48.73%, 1000=27.83%
  lat (msec)   : 2=5.04%, 4=0.47%, 10=0.23%, 20=0.01%, 50=0.01%
  cpu          : usr=2.51%, sys=8.73%, ctx=258909, majf=0, minf=40
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=694358,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0.00ns, window=0.00ns, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=181MiB/s (190MB/s), 181MiB/s-181MiB/s (190MB/s-190MB/s), io=2712MiB (2844MB), run=15002-15002msec

Disk stats (read/write):
  nvme0n1: ios=687780/208, sectors=5510584/5008, merge=1043/292, ticks=467187/123, in_queue=467378, util=88.38%

~46.3k IOPS random read: lower than random write here because reads on a fresh, never-written region of the test file can hit different internal paths than writes at qd=32, ratios vary by drive and firmware, which is exactly why you benchmark instead of trusting a spec sheet.

Mixed workload: 70/30 read/write

Real databases don’t do pure read or pure write: --rwmixread=70 simulates a 70% read / 30% write OLTP-style pattern.

$ fio --name=randrw --filename=testfile --size=2G --rw=randrw --rwmixread=70 \
    --bs=4k --direct=1 --ioengine=libaio --iodepth=16 --numjobs=1 \
    --runtime=10 --time_based --group_reporting

randrw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.42
Starting 1 process

randrw: (groupid=0, jobs=1): err= 0: pid=2077436: Sat Jul 25 22:37:57 2026
  read: IOPS=76.6k, BW=299MiB/s (314MB/s)(2991MiB/10001msec)
    slat (nsec): min=621, max=153239, avg=1754.62, stdev=1319.47
    clat (usec): min=10, max=6431, avg=202.75, stdev=162.30
     lat (usec): min=12, max=6438, avg=204.50, stdev=162.29
    clat percentiles (usec):
     |  1.00th=[   38],  5.00th=[   45], 10.00th=[  106], 20.00th=[  126],
     | 30.00th=[  129], 40.00th=[  133], 50.00th=[  143], 60.00th=[  165],
     | 70.00th=[  210], 80.00th=[  255], 90.00th=[  383], 95.00th=[  506],
     | 99.00th=[  766], 99.50th=[ 1012], 99.90th=[ 1598], 99.95th=[ 1762],
     | 99.99th=[ 2409]
   bw (  KiB/s): min=163816, max=360760, per=100.00%, avg=306277.60, stdev=47503.17, samples=20
   iops        : min=40954, max=90190, avg=76569.40, stdev=11875.79, samples=20
  write: IOPS=32.8k, BW=128MiB/s (134MB/s)(1283MiB/10001msec)
    slat (nsec): min=831, max=116950, avg=2092.88, stdev=1326.43
    clat (nsec): min=141, max=1249.6k, avg=7067.69, stdev=5683.62
     lat (usec): min=4, max=1251, avg= 9.16, stdev= 6.23
    clat percentiles (nsec):
     |  1.00th=[ 3632],  5.00th=[ 4256], 10.00th=[ 4384], 20.00th=[ 4640],
     | 30.00th=[ 5152], 40.00th=[ 5536], 50.00th=[ 5792], 60.00th=[ 6368],
     | 70.00th=[ 7008], 80.00th=[ 8256], 90.00th=[11072], 95.00th=[14272],
     | 99.00th=[23168], 99.50th=[28032], 99.90th=[40192], 99.95th=[46848],
     | 99.99th=[69120]
   bw (  KiB/s): min=69584, max=155096, per=100.00%, avg=131339.20, stdev=20302.93, samples=20
   iops        : min=17396, max=38774, avg=32834.80, stdev=5075.73, samples=20
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 2=0.01%, 4=0.82%, 10=25.39%, 20=3.28%, 50=4.93%
  lat (usec)   : 100=2.43%, 250=48.73%, 500=10.81%, 750=2.81%, 1000=0.42%
  lat (msec)   : 2=0.34%, 4=0.01%, 10=0.01%
  cpu          : usr=6.64%, sys=21.87%, ctx=494746, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=765694,328348,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0.00ns, window=0.00ns, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=299MiB/s (314MB/s), 299MiB/s-299MiB/s (314MB/s-314MB/s), io=2991MiB (3136MB), run=10001-10001msec
  WRITE: bw=128MiB/s (134MB/s), 128MiB/s-128MiB/s (134MB/s-134MB/s), io=1283MiB (1345MB), run=10001-10001msec

Disk stats (read/write):
  nvme0n1: ios=756215/324460, sectors=6049736/2604424, merge=2/195, ticks=150418/1523, in_queue=151969, util=82.20%

76.6k read IOPS + 32.8k write IOPS concurrently: shows how read and write throughput trade off when they share the same queue depth on the same device.

Watch it run

Terminal recording of an actual fio random-read IOPS test executing against the test file (queue depth 32, 4k blocks, 8-second run):

Results summary

Test Pattern Block size Queue depth Result
Sequential write write 1M 4 4240 MiB/s (4.45 GB/s)
Sequential read read 1M 4 6360 MiB/s (6.67 GB/s)
Random write IOPS randwrite 4k 32 145k IOPS, 568 MiB/s
Random read IOPS randread 4k 32 46.3k IOPS, 181 MiB/s
Mixed 70/30 randrw 4k 16 76.6k read + 32.8k write IOPS

Cleanup

Test files live in a normal directory: delete when done, nothing device-level to undo.

$ rm -rf /home/dat/fio-test

Takeaways

  • Always --direct=1: without it you’re benchmarking page cache, not the SSD.
  • Large block + low queue depth (bs=1M, iodepth=4) gives throughput numbers.
  • Small block + high queue depth (bs=4k, iodepth=32) gives IOPS numbers.
  • --time_based --runtime=N gives stable, comparable results for random workloads; sequential jobs can just run to completion.
  • Test files on the target filesystem measure exactly what your applications experience. Raw device paths (/dev/sdX) are unnecessary for this and destructive if misused, skip them.