YouTip

## PyTorch `torch.cuda.manual_seed_all` In deep learning, reproducibility is critical for debugging, comparing models, and sharing research results. PyTorch provides several functions to control random number generators (RNGs). `torch.cuda.manual_seed_all` is a PyTorch function used to set the seed for generating random numbers across **all available CUDA (GPU) devices**. If you are working with multi-GPU setups (such as Distributed Data Parallel training), this function ensures that all GPUs are initialized with a consistent random state. --- ### Function Definition ```python torch.cuda.manual_seed_all(seed) ``` #### Parameters: * **`seed`** *(int)*: The desired seed value. It must be an integer. #### Behavior: * If CUDA is not available, calling this function is a no-op (it will be safely ignored without throwing an error). * It sets the seed of the random number generator for all GPUs. If you only want to set the seed for the current GPU, you can use `torch.cuda.manual_seed(seed)` instead. --- ### Code Example The following example demonstrates how to check for CUDA availability, set the random seed across all GPUs, and generate a reproducible random tensor. ```python import torch # Check if CUDA (GPU) is available if torch.cuda.is_available(): # Set the random seed for all available CUDA devices seed_value = 42 torch.cuda.manual_seed_all(seed_value) # Generate a random tensor on the default CUDA device x = torch.randn(3, 3).cuda() print("Random tensor (reproducible):") print(x) print(f"Total CUDA devices configured: {torch.cuda.device_count()}") else: print("CUDA is not available on this system.") ``` --- ### Best Practices for Full Reproducibility While `torch.cuda.manual_seed_all` ensures that random operations on all GPUs are seeded, achieving complete reproducibility in PyTorch requires seeding other random number generators as well. Here is the standard configuration block used by developers to ensure deterministic behavior across CPU, GPU, and third-party libraries: ```python import random import numpy as np import torch def set_deterministic_seed(seed=42): # 1. Python built-in random module random.seed(seed) # 2. NumPy library np.random.seed(seed) # 3. PyTorch CPU operations torch.manual_seed(seed) # 4. PyTorch GPU operations (all GPUs) if torch.cuda.is_available(): torch.cuda.manual_seed(seed) torch.cuda.manual_seed_all(seed) # 5. Configure CuDNN to be deterministic torch.backends.cudnn.deterministic = True torch.backends.cudnn.benchmark = False # Initialize the seeds set_deterministic_seed(42) ``` ### Summary of Differences | Function | Scope | Recommended Use Case | | :--- | :--- | :--- | | `torch.manual_seed(seed)` | CPU & Current GPU | General seeding for single-GPU or CPU-only workflows. | | `torch.cuda.manual_seed(seed)` | Current GPU only | Seeding the active GPU device. | | `torch.cuda.manual_seed_all(seed)` | All GPUs | Multi-GPU training (e.g., DataParallel, DistributedDataParallel). |