A large share of the things that break in distributed training have nothing to do with the accelerator. Process groups fail to initialize. The launcher doesn't spawn the workers you asked for. A collective hangs because one rank took a different branch. A wrapper silently stops synchronizing gradients. Many of those failures reproduce on CPU with the Gloo backend, often in a few seconds.

This post covers what you can meaningfully test on CPU, with runnable examples, and is honest about where the CPU story stops.

What Gloo gives you

Gloo is PyTorch's primary CPU collective communication backend and is available in standard Linux PyTorch distributions. For standard Linux PyTorch distributions it typically requires no separate communication runtime, and it implements the collectives that matter for the common training patterns: all_reduce, all_gather, broadcast, reduce, and barrier. It is not a performance backend and it is not feature-complete against NCCL. That is fine. You are not benchmarking. You are asking whether the code is correct and whether the plumbing is connected.

Check what your build actually has:

import torch
import torch.distributed as dist

print("Torch version:", torch.__version__)
print("Distributed available:", dist.is_available())
print("Gloo available:", dist.is_gloo_available())

On a CPU-only container this reports Gloo available and NCCL not available, which is exactly the configuration we want to test in.

Two processes, one machine

torchrun --standalone spawns multiple worker processes on a single node and wires up rendezvous for you. No cluster, no scheduler:

torchrun --standalone --nnodes=1 --nproc-per-node=2 smoke.py

The smallest useful test is a collective with a known answer:

import torch
import torch.distributed as dist

def main():
    dist.init_process_group(backend="gloo")
    rank = dist.get_rank()
    world_size = dist.get_world_size()

    tensor = torch.tensor([rank + 1.0])
    dist.all_reduce(tensor, op=dist.ReduceOp.SUM)

    expected = world_size * (world_size + 1) / 2
    assert tensor.item() == expected

    dist.barrier()
    if rank == 0:
        print("PASS: Gloo all_reduce")
    dist.destroy_process_group()

if __name__ == "__main__":
    main()

Two ranks contribute 1.0 and 2.0, both see 3.0. The assertion is the point. Checking that dist.is_available() returns True proves the API is importable; checking that all_reduce produced the right number proves two processes actually talked to each other.

DDP: assert on synchronization, not loss

The next layer up is DistributedDataParallel. Most DDP examples print the loss and stop there, which tells you nothing, because with different input per rank you expect different loss values. What you actually want to know is whether parameters agree after the optimizer step.

import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

def main():
    dist.init_process_group("gloo")
    rank = dist.get_rank()
    world_size = dist.get_world_size()

    torch.manual_seed(1234)
    model = DDP(torch.nn.Linear(4, 1))
    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

    x = torch.ones(4, 4) * (rank + 1)      # different data per rank
    target = torch.zeros(4, 1)

    optimizer.zero_grad()
    loss = torch.nn.functional.mse_loss(model(x), target)
    loss.backward()
    optimizer.step()

    weight = model.module.weight.detach().clone()
    gathered = [torch.empty_like(weight) for _ in range(world_size)]
    dist.all_gather(gathered, weight)
    for other in gathered:
        assert torch.allclose(weight, other)

    dist.destroy_process_group()

Different data in, identical weights out. Using different input on each rank is intentional: without gradient synchronization, the two models would take different optimizer steps, and the parameter comparison would expose it. If gradient all-reduce were broken or the model were never really wrapped, this assertion fails immediately. It runs in seconds on CPU and it is one of the highest-value distributed correctness tests you can put in CPU CI.

Everything up to this point is established PyTorch distributed functionality. It is still valuable Day-0 and CI coverage, but none of it is new. The next two tests exercise distributed changes introduced in PyTorch 2.14.

Make timeouts fail on purpose

Here is a failure mode CPU testing is particularly good at. When a rank stalls, the process group waits. For Gloo, the default process-group timeout is 30 minutes. In CI that means a job that is already dead sits burning a runner for half an hour before anyone finds out.

PyTorch 2.14 promotes torch.distributed.set_timeout() to a stable API, letting you change the timeout of an already-initialized process group. You can use it to verify that your timeout handling works at all, by inducing a hang deliberately:

import time
from datetime import timedelta
import torch
import torch.distributed as dist

dist.init_process_group(backend="gloo", timeout=timedelta(seconds=30))
rank = dist.get_rank()

dist.all_reduce(torch.tensor([rank + 1.0]))   # baseline: group is healthy

dist.set_timeout(timedelta(seconds=2))

if rank == 1:
    time.sleep(5)          # rank 1 arrives late, on purpose

start = time.monotonic()
try:
    dist.barrier()
    print(f"rank {rank}: FAIL - barrier should not have succeeded")
except Exception as exc:
    print(f"rank {rank}: timed out after {time.monotonic() - start:.2f}s")
    print(f"rank {rank}: {type(exc).__name__}")

Rank 0 fails at roughly 2.02 seconds with a Gloo receive timeout. Rank 1 then sees a closed peer connection, because rank 0 gave up and tore down the path. Both are correct outcomes.

Run the baseline collective before changing the timeout. Otherwise a failure tells you nothing, since you can't distinguish a working timeout from a process group that was broken from the start.

Reconfiguration in 2.14, with caveats

PyTorch 2.14 adds an experimental reconfiguration path to Gloo: _supports_reconfigure(), _get_reconfigure_handle(), and _reconfigure() in distributed_c10d. Initialize with enable_reconfigure=True, exchange opaque handles between ranks through a store, create the communicator through the reconfiguration API, then reconfigure the same process group in place and keep using it.

The core flow looks like this. It is abridged; the full runnable script, including the exchange_handles() helper that moves handles between ranks via a FileStore, is linked at the end of the post.

from torch.distributed import distributed_c10d as c10d

dist.init_process_group(backend="gloo", enable_reconfigure=True)
assert c10d._supports_reconfigure()

uuid, handles = exchange_handles(store, rank, world_size, "initial")
c10d._reconfigure(uuid=uuid, handles=handles, timeout=timedelta(seconds=30)).wait()

dist.all_reduce(tensor)   # works before

uuid2, handles2 = exchange_handles(store, rank, world_size, "second")
c10d._reconfigure(uuid=uuid2, handles=handles2, timeout=timedelta(seconds=30)).wait()

dist.all_reduce(tensor)   # and after

Collectives succeed on both sides of the in-place reconfiguration. That is real, and it works today on CPU.

Be clear about what it isn't. Reconfiguring a healthy process group is not fault tolerance. No worker was killed, no rank was replaced, world size never changed, and no training run was resumed. The API is marked experimental and may change or be removed. What this demonstrates is that the mechanism is present and functional, which is a prerequisite for elastic recovery rather than a demonstration of it.

Where CPU testing stops

Being honest about the limits is what makes the rest credible. Testing on Gloo tells you nothing about:

• CUDA or ROCm kernel behavior, or anything involving device tensors

• NCCL and RCCL semantics, which differ from Gloo in error handling and in which collectives are supported

• Multi-GPU DDP, tensor parallelism, or sharded strategies at realistic scale

• Anything crossing a host boundary: container networking, TCP rendezvous between machines, InfiniBand, RDMA, NVLink

• Performance of any kind

The natural next step, and still no accelerator required, is to run rank 0 and rank 1 in separate containers with TCP rendezvous instead of --standalone. That moves you from inter-process communication inside one namespace to real network communication, and catches a different class of bug: unreachable addresses, wrong interface bindings, firewall rules.

Put it in CI

None of this needs special hardware, which means all of it can run on every commit. A reasonable gate for any project shipping PyTorch builds or containers:

• torch.distributed.is_available() and is_gloo_available() both return true

• torchrun launches two workers and both report world_size=2

• all_reduce returns the mathematically expected value on every rank

• DDP parameters match across ranks after a training step

• A deliberately induced timeout fails at the configured deadline rather than hanging

CPU distributed testing does not replace GPU validation. It moves a large class of correctness failures earlier in the development cycle, where they are cheaper to reproduce and cheaper to fix. Test rendezvous, process groups, collectives, DDP synchronization, and timeout handling on every commit. Then spend accelerator time on the things only accelerators can tell you: NCCL and RCCL behavior, device correctness, topology, scale, and performance.

Save the accelerator hours for the questions only accelerators can answer.

Full runnable scripts: https://github.com/raravind007/distributed-pytorch-cpu-tests