Bug or Feature$^2$: Weight Drift and Activation Sparsity

arXiv:2605.17659v3 Announce Type: replace Abstract: The design of modern neural architectures has converged through incremental empirical choices, yet the mechanisms governing their training dynamics remain only partially understood. We identify and analyze a negative weight drift induced by the interaction between standard losses and positively biased activation functions. We prove that under MSE or cross-entropy loss, the gradient with respect to positive pre-activations is non-negative in expectation at initialization, driving downstream weights toward negative values during early training. The drift is intrinsic to optimization rather than data, and persists across architectures and asymmetric activation functions (ReLU, GELU, SiLU). Coupled with ReLU, weight drift produces activation sparsity reaching nearly 90% in GPT-nano. We further analyze whether this natural sparsity is damaging and find that, on average, models sharply degrade above $\sim$70% activation sparsity. We then evaluate sparsity-inducing activation functions, which reveals activation spikes in intermediate transformer layers. Code is available at https://github.com/On-Point-RND/BugOrFeature.