YouTip

## PyTorch `torch.cuda.seed_all` API Reference `torch.cuda.seed_all` is a PyTorch function used to set the random seed for all available CUDA (GPU) devices. This ensures that random number generation across multiple GPUs is initialized to a deterministic state, which is crucial for reproducibility in multi-GPU deep learning workflows. This function is functionally identical to `torch.cuda.manual_seed_all`. --- ## Function Definition ```python torch.cuda.seed_all() ``` ### Behavior * If CUDA is not available, calling this function has no effect. * It automatically initializes the random number generator (RNG) state for all detected CUDA devices. * It is safe to call even if your system only has a single GPU. --- ## Code Example The following example demonstrates how to check for CUDA availability, set the seed across all CUDA devices, and generate a random tensor. ```python import torch # Check if CUDA (GPU support) is available if torch.cuda.is_available(): # Set the random seed for all available CUDA devices torch.cuda.seed_all() # Create a random tensor and move it to the default CUDA device x = torch.randn(3, 3).cuda() print(f"Random tensor:\n{x}") print(f"CUDA device count: {torch.cuda.device_count()}") else: print("CUDA is not available on this system.") ``` --- ## Key Considerations & Best Practices ### 1. `torch.cuda.seed_all()` vs. `torch.cuda.manual_seed_all(seed)` * `torch.cuda.seed_all()` sets a random seed (typically derived from `/dev/urandom` or the system clock) across all GPUs. * If you want to enforce **strict reproducibility** (i.e., getting the exact same results every time you run your script), you should use `torch.cuda.manual_seed_all(seed)` with a specific integer value (e.g., `42`). ### 2. Achieving Full Reproducibility Setting the CUDA seed alone is often not enough to guarantee fully reproducible results in PyTorch. For complete determinism across CPU and GPU operations, you should configure the following seeds and settings: ```python import torch import random import numpy as np def set_deterministic_seed(seed=42): # Python built-in random module random.seed(seed) # Numpy library np.random.seed(seed) # CPU seed torch.manual_seed(seed) # GPU seeds (both single and multi-GPU setups) torch.cuda.manual_seed(seed) torch.cuda.manual_seed_all(seed) # Configure cuDNN to use deterministic algorithms torch.backends.cudnn.deterministic = True torch.backends.cudnn.benchmark = False ```