dataloaders  part 1

13 Jul, 2026

fafo exercise 1 : dumbest dataloader

• reads video and extracts raw frames
• no transfroms nothing , takes num_workers = 0 for baseline timing
• sweep num_workers, measure batch-fetch time
• toggle pin_memory, measure effect

If you load your samples in the Dataset on CPU and would like to push it during training to the GPU, you can speed up the host to device transfer by enabling pin_memory. This lets your DataLoader allocate the samples in page-locked memory, which speeds-up the transfer.

• toggle persistent_workers, measure per-epoch startup cost

With this option to false, every time your code hits a line line for sample in dataloader:, it will create a brand new set of workers to do this loading and will kill them on exit. Meaning that if you have multiple dataloaders, the workers will be killed when you are done with one instantly. If you make them persist, these workers will stay around (with their state) waiting for another call into that dataloader. Setting this to True will improve performances when you call into the dataloader multiple times in a row (as creating the workers is expensive). But it also means that the dataloader will have some persistent state even when it is not used (which can use some RAM depending on your dataset).

*inject artifical I/O latency into getitem redo the num_workers sweep

Dataloader FAFO — Phase 1 Worklog

Date: 2026-07-03 Setup: Fedora, 8 logical CPU cores, CPU-only run (no GPU in this pass) Source video: test.mp4 — 1 min clip, 2054 frames extracted at native fps Dataset: naive Dataset — reads jpg via cv2.imread, resizes to 224x224, no other transforms

1. Baseline (num_workers=0)

total=6.00s | per_batch=60.04ms

Single-process loading: main process does read → decode → resize → tensor conversion, one sample at a time, no overlap with anything. This is the number every other optimization is measured against.

2. num_workers sweep — no artificial I/O delay

Reading: Gains scale cleanly 0→4, then flatline at 4→8, then regress at 16.

Why: the machine has 8 logical cores. Once worker count approaches/exceeds that, workers start competing for CPU time instead of doing independent parallel work — context-switching overhead outweighs any benefit. PyTorch's own warning confirmed this directly: "suggested max number of workers in current system is 8."

Takeaway: __getitem__ here (imread + resize) is CPU-bound, not I/O-bound. In a CPU-bound regime, worker count has a hard ceiling around your core count — more workers past that point is pure waste, not "free" parallelism.

3. pin_memory on/off

Reading: no benefit — actually marginally slower (noise-level difference, not meaningful on its own).

Why: pin_memory speeds up host→GPU transfer. No GPU was available in this run (torch.cuda.is_available() == False), so there's no transfer to speed up — PyTorch printed this directly: "pin_memory argument is set as true but no accelerator is found."

Takeaway: pin_memory=True is not a free "always on" setting — it's conditional on actually having a device to transfer to. Re-test once GPU driver issue is resolved.

4. persistent_workers on/off (3 epochs)

Reading: small win (~7%), not dramatic at this dataset size/complexity.

Why: persistent_workers=False respawns worker processes every epoch (interpreter startup, re-importing, re-initializing dataset state per worker). At 2054 frames and a trivial __init__, that respawn cost is small relative to total work per epoch. This optimization matters more as dataset setup cost grows (heavier __init__, larger metadata index, etc.) — not something to assume is dramatic by default.

5. num_workers sweep — WITH 10ms artificial I/O latency per sample

Simulates slow storage/network reads via time.sleep(0.01) inside __getitem__.

Reading: No plateau. Scaling stays roughly monotonic well past 8 workers — 16 is still meaningfully faster than 8, unlike the no-latency sweep where 8→16 regressed.

Why: when each sample involves a wait (blocked on I/O), a worker spends most of its time doing nothing useful — CPU sits idle during the sleep. More workers means more samples can be "in flight" waiting simultaneously, and since they're not actually contending for CPU cycles (they're blocked, not computing), you can meaningfully oversubscribe past your core count.

Takeaway — this is the core lesson of Phase 1: There is no universal "correct" num_workers. It's a direct function of whether your per-sample cost is:

• CPU-bound (decode/resize/augment-heavy) → ceiling near core count, more workers past that hurts.
• I/O-bound (slow disk, network, remote storage) → keep scaling well past core count, since workers spend most time blocked, not computing.

The two sweeps side-by-side are the proof — same dataset, same machine, only difference is where the per-sample bottleneck sits.

resources:

• Read about pin memory and stuff
• Read about persistent workers