Read Data into GPU Memory in Python with GPUDirect Storage

10 min read

Came here from Instagram, LinkedIn, X or YouTube? Here is the full source list you came for. Everything below is built from these, nothing else.

The tutorial

• NVIDIA Technical Blog: How to Read Data into GPU Memory in Python with NVIDIA GPUDirect Storage, by Andrew Hoyle, Oct 22 2026

By the end of this tutorial you will have a short Python script that reads a 1 GiB file from NVMe straight into a GPU buffer, plus a way to prove which path the bytes actually took. That second part is the one most people skip, and it is the part that matters.

Here is why this is worth your evening. NVIDIA's post cites the 2025 Stanford AI Index: compute for notable AI models has been doubling roughly every five months, and the datasets used to train LLMs roughly every eight months. Faster GPUs do nothing for you if they sit waiting on storage.

GPUDirect Storage (GDS) gives you a direct path from supported storage into GPU memory, so data does not have to stop in CPU memory first. That also cuts CPU usage across the system. The catch is that the main way in is the cuFile C API, while nearly half of all new AI projects on GitHub were built mainly in Python as of August 2025.

KvikIO closes that gap. It is NVIDIA's Python and C++ library for file I/O, sitting on top of cuFile. I will walk you through it step by step.

Key takeaways

• KvikIO reads a file into a CuPy array in about five lines of Python, on top of cuFile.
• There are two separate decisions: KvikIO picks cuFile or its own POSIX path, then cuFile picks the accelerated nvidia-fs path or its CPU-staged fallback.
• Defaults on the test system: compat mode AUTO, GDS threshold 16 KiB, task size 4 MiB.
• A 1 GiB read became 256 operations, and gds_stats showed nvfs=256, posix=0, meaning every one used the fast path.
• A successful read count proves nothing about the path. Only the cuFile stats (CUFILE_STATS_LEVEL=3) do.

How does KvikIO actually talk to GPUDirect Storage?

Think of it as three layers.

GDS is the overall technology. For file system storage, cuFile is the C library and API that applications use to reach GDS. KvikIO sits above cuFile and gives you Python and C++ interfaces, plus its own knobs for how requests run.

You hand KvikIO a file path and a GPU buffer. KvikIO then makes the first call: send this through cuFile, or use its own POSIX compatibility path, which stages data through host memory.

If it picks cuFile, it registers the file and issues the I/O against your GPU buffer. Then cuFile makes the second call. Based on system support, the file system and its config, it uses either the accelerated GDS path or its own CPU-staged compatibility path.

So reaching cuFile only tells you KvikIO chose cuFile. It does not tell you the fast path was used. Keep that in your head for the whole tutorial.

What do you need before you start?

• A Linux machine with an NVIDIA GPU and GPUDirect Storage installed.
• Storage and a file system that GDS supports.
• Permission to create, read and write files on that storage, and access to the GDS tools gdscheck and gds_stats.

NVIDIA's GPUDirect Storage documentation has the full install and platform requirements.

For reference, NVIDIA tested this on:

Part	Setup
Machine	x86_64 with an NVIDIA H100
Software	CUDA 13.3, cuFile 1.18
Storage	Local NVMe formatted as ext4
Topology	PIX link between GPU and NVMe (at most one PCIe bridge)
PCIe	IOMMU and ACS disabled to keep peer-to-peer transfers
Kernel	nvidia-fs module loaded for the accelerated path

Check how your GPU and NVMe are connected first:

nvidia-smi topo -nvme

Step 1: How do you install KvikIO?

Make a virtual environment and install the CUDA 13 build. CuPy comes along as a dependency.

python3 -m venv kvikio-env
source kvikio-env/bin/activate
python -m pip install kvikio-cu13

Confirm both packages landed:

python -m pip show kvikio-cu13 cupy-cuda13x

Step 2: How do you check the KvikIO and cuFile settings?

Before any I/O, look at what is active. Open Python and run:

import kvikio
import kvikio.cufile_driver as cufile_driver
import kvikio.defaults as defaults

kvikio.__version__
cufile_driver.libcufile_version()
defaults.get("compat_mode").name
defaults.get("gds_threshold")

That gives you the KvikIO version, the cuFile version, the compatibility mode (AUTO, ON or OFF) and the smallest request KvikIO will send to cuFile.

On NVIDIA's system it reported AUTO and a threshold of 16 KiB. AUTO means try cuFile, and fall back to the POSIX backend if cuFile is not available. Any read under the threshold skips cuFile and goes straight to the POSIX path.

You can set both from the shell with KVIKIO_COMPAT_MODE and KVIKIO_GDS_THRESHOLD, or from Python. defaults.set takes a dict for several settings, or a name and value for one:

defaults.set({"task_size": 1024**3, "num_threads": 8})
defaults.set("compat_mode", kvikio.CompatMode.OFF)

Now check the GDS side of the platform:

/usr/local/cuda-13.3/gds/tools/gdscheck.py -p 2>&1 | less -R

This lists installed GDS components, supported file systems, detected GPUs and the active cuFile config. Adjust the CUDA path if your version differs.

Step 3: How do you read a file into GPU memory with Python?

Create a 1 GiB test file of random data, written with direct I/O and flushed to disk. You only need to do this once.

dd if=/dev/urandom \
  of=/mnt/nvme1/kvikio-tutorial/kvikio-example.bin \
  bs=1M count=1024 \
  oflag=direct conv=fsync \
  status=progress

Save this as read_with_kvikio.py:

from pathlib import Path
import cupy as cp
import kvikio

file_path = Path(
    "/mnt/nvme1/kvikio-tutorial/kvikio-example.bin"
)

# Allocate a GPU buffer large enough to hold the file.
gpu_buffer = cp.empty(
    file_path.stat().st_size,
    dtype=cp.uint8,
)

# Read the file into the GPU buffer.
with kvikio.CuFile(file_path, "r") as file:
    bytes_read = file.read(gpu_buffer)

# Validate that 1 GiB was read.
assert bytes_read == gpu_buffer.nbytes

The CuPy array is the destination. The context manager opens the file and finishes the read into GPU memory.

The assert passing tells you the whole file arrived. It does not tell you how it got there. That is the next step.

Step 4: How do you confirm GPUDirect Storage was actually used?

cuFile keeps per-process statistics. Turn them on with CUFILE_STATS_LEVEL=3 and read them with gds_stats while the process is still alive.

Setting it as an environment variable keeps it to one run. You can also set it in cufile.json.

CUFILE_STATS_LEVEL=3 python3 -i read_with_kvikio.py

The -i flag is the trick. Python drops into interactive mode after the read, so the process stays alive and gds_stats can inspect its counters from another terminal.

In the per-GPU output, look at two counters:

• nvfs counts operations that went through the nvidia-fs kernel module. This is the fast path.
• posix counts operations handled by cuFile's compatibility path.

KvikIO's default task size is 4 MiB, so the 1 GiB read splits into 256 operations. NVIDIA's run showed n=256, nvfs=256, posix=0, totalling 1,024 MiB. Every operation used the accelerated path.

Step 5: How do you compare the fast path and the fallback paths?

Run the same script three ways to see both decision layers at work.

First, force KvikIO to use cuFile:

CUFILE_STATS_LEVEL=3 \
  KVIKIO_COMPAT_MODE=off \
  python3 -i read_with_kvikio.py

On the test system, gds_stats shows nvfs operations. cuFile chose the accelerated path.

Second, keep KvikIO on cuFile but force cuFile into its compatibility path:

CUFILE_STATS_LEVEL=3 \
  KVIKIO_COMPAT_MODE=off \
  CUFILE_FORCE_COMPAT_MODE=true \
  python3 -i read_with_kvikio.py

KvikIO still calls cuFile, but the operations now land under posix instead of nvfs.

Third, turn on KvikIO's own compatibility mode:

KVIKIO_COMPAT_MODE=on \
  python3 read_with_kvikio.py

KvikIO reads through its CPU-staged POSIX backend and never calls cuFile, so cuFile records nothing.

Run	KvikIO path	cuFile path	What gds_stats shows
KVIKIO_COMPAT_MODE=off	cuFile	nvidia-fs	nvfs ops
off + CUFILE_FORCE_COMPAT_MODE=true	cuFile	compatibility	posix ops
KVIKIO_COMPAT_MODE=on	POSIX	not called	nothing

Step 6: How does cuFile split up a big read?

Once KvikIO hands a request to cuFile, a second set of controls takes over. These live in cufile.json and work separately from KvikIO's settings.

Both layers can chop a large read into pieces. KvikIO goes first: KVIKIO_TASK_SIZE sets the piece size and KVIKIO_NTHREADS controls how those tasks are scheduled. Each task on the cuFile path becomes its own cuFile request. cuFile can then split each request again across its internal thread pool.

cufile.json setting	What it does
execution.parallel_io	Turns on cuFile's internal parallel processing
execution.max_io_threads	Caps cuFile worker threads per GPU queue
execution.min_io_threshold_size_kb	Target minimum size for scheduled pieces (the last remainder can be smaller)
execution.max_request_parallelism	Caps how much parallelism cuFile prepares for one request
execution.max_io_queue_depth	Caps pending work items in the internal queue

Here is how a 1 GiB read played out in NVIDIA's config: four worker threads, max_request_parallelism of 4, and min_io_threshold_size_kb of 8192 (8 MiB).

With the default 4 MiB task size, KvikIO sends 256 requests, and cuFile handles each 4 MiB request as one piece. This was the nvfs=256 run.

With a 1 GiB task size, KvikIO sends one big request. You can set that in Python:

defaults.set("task_size", 1024**3)

The CuPy buffer was never registered with cuFile, so cuFile uses its unregistered-buffer scheduling path. It caps the request at min(4 threads, 4 parallelism) = 4 pieces of 256 MiB each, well above the 8 MiB target. With room in its queue, cuFile gives three pieces to workers, runs the fourth on the calling thread, and waits for all of them.

Those 256 MiB pieces are units of scheduled work, not single storage transfers. Since your buffer is unregistered, cuFile reads into a registered GPU bounce buffer and then copies into your array. Since GDS 1.17, those bounce buffers can be configured as slabs. In this config the nvidia-fs path uses slabs no bigger than 16 MiB, so each 256 MiB piece moves as a series of smaller transfers.

What are the common mistakes with KvikIO and GDS?

Trusting the return value. bytes_read == gpu_buffer.nbytes means the data arrived. It says nothing about whether it skipped the CPU. Check nvfs in gds_stats.

Assuming cuFile means fast. KvikIO choosing cuFile is only the first decision. cuFile can still fall back to its CPU-staged path, and that shows up as posix.

Letting the process exit before checking stats. The counters are per process. Run with python3 -i so the process stays up while you read them.

Testing with tiny reads. Anything under the GDS threshold (16 KiB on the test system) goes straight to KvikIO's POSIX path and never touches cuFile.

Looking for cuFile stats with KVIKIO_COMPAT_MODE=on. In that mode cuFile is never called. Empty stats are expected, not a bug.

Reading min_io_threshold_size_kb as the transfer size. It is a scheduling target. With an unregistered buffer, actual transfers go through bounce buffer slabs, which were 16 MiB at most in NVIDIA's config.

Skipping the topology check. NVIDIA's test used a PIX link with IOMMU and ACS disabled to keep PCIe peer-to-peer working. Run nvidia-smi topo -nvme and gdscheck.py -p before you blame your code.

Where do you go from here?

• Run the script on your own GDS system and confirm you see nvfs counts, not posix.
• Try different task_size and num_threads values with defaults.set and watch how the operation count in gds_stats changes.
• Change one cufile.json execution setting at a time and rerun the stats check.
• Look at where KvikIO is already used. NVIDIA names cuDF, cuCIM digital pathology workflows, cuPhoton scientific image analysis and GPU-accelerated Presto analytics.

FAQ

What is NVIDIA KvikIO?

KvikIO is a Python and C++ library for high-performance file I/O, part of NVIDIA's CUDA-X libraries for data science. It sits above the cuFile C API and gives you idiomatic Python calls like kvikio.CuFile(path, "r") and .read(buffer), plus settings that control how requests run.

How do I know if GPUDirect Storage is working in Python?

Run your script with CUFILE_STATS_LEVEL=3 and python3 -i so the process stays alive, then inspect it with gds_stats. If the per-GPU nvfs counter matches your operation count and posix is 0, the reads used the accelerated nvidia-fs path. In NVIDIA's test that was 256 of 256.

What is the difference between KVIKIO_COMPAT_MODE and CUFILE_FORCE_COMPAT_MODE?

They control different layers. KVIKIO_COMPAT_MODE=on makes KvikIO skip cuFile completely and use its own CPU-staged POSIX backend. CUFILE_FORCE_COMPAT_MODE=true lets KvikIO call cuFile but makes cuFile use its compatibility path, which shows up under posix in the stats.

Why did a 1 GiB read turn into 256 operations?

KvikIO's default task size is 4 MiB, and 1 GiB divided by 4 MiB is 256. Each task becomes its own cuFile request. If you raise the task size to 1 GiB, KvikIO sends a single request and cuFile splits it according to its own cufile.json settings.

Run the five-line read on your own machine and check the nvfs counter before you trust any benchmark. Browse the free tools on Art of Code for everyday developer utilities, and see the latest engineering jobs if GPU work is where you want to go next.

Written by Adithya Guttha, Founder of Art of Code.