Exact Degeneracy Under Balanced k-Shot Sampling:Consequences for Small-Sample Discriminant Analysis on LLM Embeddings

arXiv:2609.09860v2 Announce Type: replace Abstract: Balanced k-shot sampling, drawing exactly k labeled examples per class, is the standard protocol in few-shot evaluation. We show that it induces an exact degeneracy in Kernelized Linear Principal Component Discriminant Analysis (KLPCDA), a family of seven supervised dimensionality reduction methods. The within-class operator becomes exactly a scaled orthogonal projector, so two variants have all signal eigenvalues equal and a third has a zero operator, and the truncated pseudo-inverse used to implement them removes exactly the subspace that contains all between-class information. We verify this to float64 precision on sentence embeddings and on residual-stream activations of a decoder-only model, and find that the accuracies of the truncated variants change with the number of BLAS threads, by up to 12 percentage points (57 for the variant with a zero operator). Replacing the truncation with a ridge selected within the support set, we evaluate few-shot text classification on frozen embeddings (d up to 4096) over four datasets, three embedding models, and three trained baselines. The regularized formulation outperforms the best original variant in 45 of 48 settings, but in general not a tuned linear probe or nearest centroid on bidirectional encoders. Its gains are largest on E5-Mistral, the only decoder-based embedding model, and come from regularized within-class whitening rather than from KLPCDA: shrinkage LDA outperforms both baselines in all 16 E5-Mistral settings, and pre-specified tests show that the effect depends on the embedding model rather than on its dimension. E5-Mistral's remaining gap is largely an estimation-efficiency effect: increasing k to 30-50 reduces the probe's gap to a bidirectional encoder by more than 80%. We give five guidelines limited to the conditions tested and release the code and per-seed results.