PyTorch Layer Performance Benchmark

Dropout and MaxPool2d have the fastest forward pass times because they have zero trainable parameters and perform simple element-wise or reduction operations. At batch size 32, Dropout completes in ~0.005 ms while MaxPool2d takes ~0.026 ms. Among parameterized layers, BatchNorm2d is fastest due to its simple affine transform with running statistics.

TransformerEncoder is the slowest layer because self-attention requires computing an attention matrix of size (batch * heads * seq_len * seq_len), which grows quadratically with sequence length. At seq_len=512 with 8 heads, this means 2M attention scores per sample. Additionally, each transformer layer contains two sublayers (attention + FFN), each with layer normalization and residual connections.

Increasing batch size improves throughput (samples/second) but increases total forward pass time and memory usage. GPU utilization improves with larger batches due to better parallelism. For Conv2d, going from batch=1 to batch=128 increases total time by ~20x but throughput by ~6x. The optimal batch size depends on GPU memory and the compute-to-memory-bandwidth ratio of each layer.

FLOPs (Floating Point Operations) measure the computational cost of a layer. A higher FLOP count means more arithmetic work. However, FLOPs alone don't determine speed — memory bandwidth, kernel optimization, and GPU utilization also matter. Conv2d uses highly optimized cuDNN kernels that achieve near-peak FLOPs, while custom operations may only achieve 10-30% of theoretical throughput.

Use torch.utils.benchmark.Timer for accurate GPU timing with proper CUDA synchronization. Key steps: 1) Warm up the GPU with a few forward passes, 2) Use timer.blocked_autorange() for automatic iteration count, 3) Call torch.cuda.synchronize() before timing, 4) Report median instead of mean to avoid outliers. Alternatively, use torch.profiler.profile() with TensorBoard for detailed kernel-level analysis.