Diagonally-Weighted Generalized Method of Moments Estimation for Gaussian Mixture Modeling

arXiv:2507.20459v2 Announce Type: replace Abstract: With direct computation of moments through order $L$ in $d$ dimensions, standard method of moments (MM) incurs $\Omega(d^L)$ computation and storage costs, and standard generalized method of moments (GMM) additionally requires estimating and inverting an often ill-conditioned covariance matrix of size $\Omega(d^L)$-by-$\Omega(d^L)$ to obtain the optimal weighting matrix. Pereira, Kileel, and Kolda [arXiv:2202.06930 (2022)] avoid the tensor computation cost of MM through implicit moment computation. Despite this progress, incorporating statistically principled weighting while preserving computational efficiency remains an important challenge, particularly at low signal-to-noise ratio (SNR), where sampling variances can differ substantially across moment orders. We introduce diagonally-weighted GMM (DGMM), which assigns one adaptive weight to each moment order through a structured approximation of the optimal weighting matrix while avoiding the covariance inversion, thereby balancing statistical efficiency, computational complexity, and numerical stability. For general parametric models, we prove that DGMM is consistent, asymptotically normal, and near-efficient under explicit structural conditions. For heteroscedastic low-rank Gaussian mixtures, we design a computationally efficient and numerically stable algorithm that extends the implicit moment computation to the DGMM objective, gradients, and weights and incorporates Nystr\"om approximations of the kernel sums. In low-SNR experiments with $d=100$ and $L=4$, where standard MM and GMM are infeasible, controlled weighting comparisons show that DGMM achieves consistently small parameter errors across tested initializations, including the challenging starts where alternative weightings struggle, with shorter median runtimes and fewer iterations. The code and data will be available upon publication at https://github.com/liu-lzhang/dgmm.