全部动态

本页 40 条
10月9日 周五
arXiv cs.LG70

Bilevel optimization for data-driven learning of Koopman embeddings using kernel-based autoencoders

arXiv:2610.12370v1 Announce Type: new Abstract: Koopman operator theory provides a linear framework for analyzing nonlinear dynamical systems and has become a major tool for data-driven modeling. A central challenge, however, is that finite-dimensional approximations computed by methods such as extended dynamic mode decomposition (EDMD) require the dictionary to be specified a priori. Recent machine-learning approaches address this limitation by learning the dictionary from data, predominantly using artificial neural network (ANN) autoencoder architectures. Although kernel methods offer an alternative with greater interpretability and tractability for theoretical analysis, they have received little attention in this setting. We introduce extended dynamic mode decomposition with kernel-based dictionary learning (EDMD-kDL), a kernel-based method for learning finite-dimensional Koopman embeddings directly from data. The method combines ideas from collocation methods and bilevel optimization to simultaneously learn a kernel dictionary and the corresponding Koopman approximation. We evaluate EDMD-kDL against state-of-the-art ANN-based approaches on a range of numerical experiments, including global sea-surface-temperature forecasting and learning directly from video data. Across all tested settings, EDMD-kDL achieves performance comparable to or better than the ANN-based methods. Moreover, in contrast to standard kernel methods, the proposed approach is scalable to large datasets by design since the size of the required kernel matrices depends on the number of collocation points rather than the size of the training dataset.

arXiv cs.LG75

Closing the Horizon Gap in Policy Optimization for Adversarial MDPs

arXiv:2610.12362v1 Announce Type: new Abstract: We consider policy optimization for online episodic tabular Markov decision processes (MDPs) with adversarial losses and bandit feedback. Policy optimization updates the policy locally at each state and avoids optimization over the occupancy-measure polytope, but its existing regret bounds are larger by a factor of the horizon $H$ than those of occupancy-measure-based algorithms. We close this gap by using regularized $Q$-functions, which allow us to control the stability of the local updates jointly over all state-action pairs rather than separately at each state. The resulting algorithm attains high-probability regret bounds of $\widetilde O(\sqrt{HS(H+A)T})$ for known transitions and $\widetilde O(HS\sqrt{AT})$ for unknown transitions, where $S$ is the number of states, $A$ the number of actions, and $T$ the number of episodes. Both bounds improve the horizon dependence of existing policy optimization bounds, and the latter matches the best-known bound. We further extend the algorithm to adversarial linear-mixture MDPs and obtain the same improvement in the horizon dependence.

arXiv cs.LG77

SplitJEPA: Learning Invariant and Variant Latent Worlds without Reconstruction

arXiv:2610.12349v1 Announce Type: new Abstract: Understanding a dynamical world calls for more than a latent state that summarizes its observations: the state should also be organized into the factors that stay shared across related observations and the factors that vary between them. For example, a robot pushing a cube to a goal should take the same action when the camera shifts or the lights dim, since nothing in the scene has moved. Existing approaches to this decomposition commonly obtain it through reconstruction, so the latent variables must first explain the entire observational world before their organization can be trusted. Joint embedding predictive architectures (JEPAs) model the latent state directly and never reconstruct, yet no existing result recovers the invariant and variant parts of the state they learn. How to learn the invariant-variant structure of the latent world without paying for its reconstruction therefore remains open. To close this gap, we introduce SplitJEPA, a JEPA that jointly recovers the latent state and its invariant and variant organization directly in representation space, without any reconstruction. We prove that, under stationary Gaussian predictive dynamics and a full-rank variation condition, SplitJEPA identifies the invariant and variant subspaces up to independent block-wise isometries, without introducing an observation decoder. Since the guarantee needs no decoder, the result extends reconstruction-free latent recovery to invariant-variant block identification. Experiments on synthetic nonlinear systems and robotic manipulation tasks support the theoretical results and show their practical value for both robustness and efficiency.

arXiv cs.LG77

环境离散扩散:在正确的时间使用错误的数据,实现数据高效学习

arXiv:2610.12340v1 公告类型:新 摘要:我们提出 RefineMix,这是一种在严重数据稀缺条件下训练离散扩散模型的框架,而数据稀缺是科学应用中的常见限制。RefineMix 在选定的扩散时间使用分布外数据,以提升泛化能力,同时不会对采样分布造成偏差。尽管这一策略已在连续扩散中得到探索,但离散扩散面临着一个独特挑战:与高斯噪声不同,掩码会保留幸存 token 中的领域信息,从而限制了在高噪声水平下使用相关数据的能力。然而,在低噪声水平下,各领域实际上互不相交的支撑集反而成为优势,使模型能够同时从域内数据和分布外数据中学习,而不会对采样器造成偏差。我们将这些直觉形式化,并对所提出的方法进行了理论分析。在实验中,跨越五种领域偏移设置,RefineMix 达到了与域内微调相当或更优的效果。在蛋白质序列生成任务中,仅使用 197 个域内样本进行微调,相较于标准微调,生成的同时具备新颖性、可折叠性并属于目标家族的蛋白质比例几乎翻倍。

arXiv cs.LG74

Composite Online-to-Nonconvex Conversion with Optimal Oracle Complexity

arXiv:2610.12328v1 Announce Type: new Abstract: We consider stochastic nonsmooth nonconvex composite optimization, which includes several important problems such as constrained optimization and the regularized training of neural networks. The objective is the sum of a possibly nonsmooth nonconvex Lipschitz function and a convex regularizer, and the function is accessed through stochastic gradients or function values. The goal is to find a point that satisfies a Goldstein-type stationarity condition designed for composite objectives. To our knowledge, no oracle complexity bound for this setting is known under first-order access, and existing complexities under zeroth-order access are suboptimal. To handle this issue, we employ the framework of online-to-nonconvex conversion, which chooses update directions by an online learner and is known to achieve optimal rates for noncomposite problems. We extend the framework to our composite scenario by introducing new losses for the learner, which contain the regularizer itself rather than its linearization and for which a variant of online mirror descent achieves low regret. We show that the resulting algorithm finds such a point with $O(\delta^{-1}\varepsilon^{-3})$ stochastic gradient queries or $O(d\delta^{-1}\varepsilon^{-3})$ function-value queries, where $\delta$ is the Goldstein radius, $\varepsilon$ is the stationarity tolerance, and $d$ is the dimension. These rates match the optimal ones for noncomposite nonsmooth nonconvex optimization, demonstrating that the additional convex regularizer does not worsen the oracle complexity. We also give rates for the smooth case and present numerical experiments.

arXiv cs.LG77

SparseDecoding:面向准确高效 LLM 推理的解码感知剪枝

arXiv:2610.12327v1 公告类型:新 摘要:大型语言模型(LLM)推理的解码阶段具有内存受限特性,会带来显著延迟。由 Hessian 引导的逐层免训练网络剪枝方法一直是解决这一问题的重要方案,因为剪枝能够减少解码过程中从内存读取的非零参数数量。然而,这类典型方法通常使用预先收集的自然序列计算 Hessian,而模型在解码过程中接收的是自行生成的 token,从而导致这两类序列之间出现分布偏移。在自然序列上计算得到的 Hessian 与在生成序列上计算得到的 Hessian 并不相同。我们观察到,这种差异会使生成过程中的激活分布偏离用于剪枝的激活分布,进一步降低剪枝模型的性能。此外,现有大多数能够带来实际加速的 LLM 剪枝方法主要针对稀疏矩阵-矩阵(SpMM)乘法,而对在解码过程中占主导地位的稀疏矩阵-向量(SpMV)运算支持有限。为解决这些问题,我们提出 SparseDecoding,这是一种面向准确高效 LLM 解码的原理性解码感知剪枝框架。具体而言,在算法层面,SparseDecoding 从稠密模型的自回归生成过程中收集逐层激活,并基于这些激活构建校准矩阵,同时排除 prefill 阶段,从而使剪枝目标与解码激活保持一致。在系统层面,我们开发了一种经过优化的 N:M 稀疏矩阵-向量内核,采用位掩码索引和固定步长遍历。针对代表性 LLM(Llama-3.1-8B、Llama-3.3-70B、Qwen3-14B / 32B)的大量实验证明,与标准固定文本校准相比,我们的方法在长文本生成基准测试中始终表现更优,并且在 A100 GPU 上实现了最高 1.48 倍的端到端实际解码加速。

arXiv cs.LG71

AdaptLSTM: Efficient Adaptive Online Learning for Cloud Workload Forecasting under Distribution Drift

arXiv:2610.12265v1 Announce Type: new Abstract: Accurate workload forecasting is critical for elastic resource provisioning in web-scale cloud services, where distribution shifts driven by viral content, product launches, and user behavior degrade offline-trained models rapidly. Naive online learning recovers accuracy but incurs prohibitive per-step compute cost. We propose AdaptLSTM, an adaptive online framework that detects drift via validation-calibrated thresholds and applies selective, targeted updates. On the Alibaba Machine Trace, AdaptLSTM recovers 54\% of Naive Online's improvement at 20\% cost ($2.7\times$ efficiency, $p=0.002$ over 10 seeds). On the more volatile Container Trace, it achieves 96\% at 20\% cost ($4.8\times$ efficiency, $+75\%$ MAE reduction over Static). Unlike classical drift detectors (ADWIN, DDM, Page-Hinkley) which fail to trigger on regression-scale error streams, AdaptLSTM fires 42 times over 301 steps and outperforms matched-budget baselines. Wall-clock profiling shows $1.33\times$ throughput gain and 45\% update-time reduction. The framework is model-agnostic: identical Pareto patterns hold for LSTM, GRU, and Transformer backbones.

arXiv cs.LG75

提升之前先分批:大图上的可扩展拓扑深度学习

arXiv:2610.12247v1 发表类型:新 摘要:拓扑深度学习将基于图的学习扩展到高阶域,例如超图、胞腔复形和单纯复形。这些域通常通过图提升过程,从输入图中的模式构建而成。全域训练会在模型执行前构建并存储完整的提升表示。在 Reddit(233k 个节点和 57.3M 条边)这类大规模且稠密的数据集上,这种全局实体化会成为严重的计算瓶颈,往往使训练无法进行。为解决这一局限,我们提出了 Cluster-TNN,这是一种与域无关的框架,通过进行局部提升来避免这一瓶颈。在预处理阶段对输入图进行划分后,Cluster-TNN 在运行时动态采样节点簇组,重建其诱导子图以形成小批次,并在每个小批次内执行所选的提升操作。保留采样节点之间的所有边,可以保留跨簇构建高阶结构所需的连通性,从而生成现有拓扑神经网络可以直接处理的拓扑小批次。在与全图执行的 21 组匹配比较中,Cluster-TNN 在每种配置下都降低了 GPU 峰值内存,平均降幅达 83.2%,同时保持了具有竞争力的预测性能。值得注意的是,据我们所知,这种内存降低首次使得在 Reddit 和 OGBN Products 等大型数据集上训练多种不同的高阶拓扑神经网络成为可能。这些结果表明,Cluster-TNN 是一种通用策略,能够突破全局域构建的限制,扩展拓扑深度学习。

arXiv cs.LG69

RIFT: Relative Isolation From Trees For Anomaly Detection

arXiv:2610.12244v1 Announce Type: new Abstract: Isolation Forest (IF) is a widely used baseline for unsupervised anomaly detection. Recent studies provide a closed-form expression for the infinite-forest limit for one-dimensional data. Inspired by the geometric interpretation of this formula, we introduce RIFT (Relative Isolation From Trees), a deterministic anomaly detection method that generates the minimum spanning tree and scores each point by the sum of the apparent sizes of tree edges as viewed from that point. For one-dimensional data, the RIFT score recovers the closed-form IF limit exactly. In higher dimensions, it provides a parameter-free generalization that is deterministic, robust to varying density and clustered anomalies and avoids the axis-parallel artifacts of IF. We further propose an ensemble variant for large datasets. Experiments on synthetic data and the ADBench benchmark demonstrate that the accuracy is comparable to IF, while the ensemble variant exhibits significantly lower variance across random seeds.

arXiv cs.LG69

AdaCast: Conditional Parameter Generation for Adaptive Time Series Forecasting

arXiv:2610.12240v1 Announce Type: new Abstract: Time-series foundation models (TSFMs) have achieved strong forecasting performance across domains. However, most adaptation methods remain static. Existing all-in-one methods learn a single set of dataset-level parameter updates and apply the same adapted model to every input. As a result, they cannot adapt the model parameters to the temporal patterns, seasonality and dynamics of each input time series. This limits their ability to produce forecasts that are tailored to heterogeneous inputs. To address this limitation, we propose AdaCast, a conditional parameter generation framework for time-series forecasting. AdaCast uses a generator to produce input-specific low-rank parameter updates for a frozen pretrained TSFM. These updates adapt the model to each input during both training and inference. Across six public benchmarks, AdaCast consistently outperforms static adaptation baseline in in-domain forecasting and improves zero-shot generalization to held-out datasets across domains. These results demonstrate that conditional parameter generation provides an effective approach for adaptive forecasting.

arXiv cs.LG80

面向未来的训练:时间序列预测测试时自适应的延迟感知审计

arXiv:2610.12232v1 公告类型:新 摘要:用于时间序列预测的测试时自适应(TTA)方法会根据不断到来的真实值更新已部署模型,或围绕该模型构建的小型适配器。然而,$H$ 步预测的标签只有在 $H$ 步之后才会生成,而真实数据管道还会引入额外延迟。我们构建了一个无数据泄漏的测试框架:只有在第 $s+d$ 步(其中 $d \ge H$)才释放预测起点 $s$ 的标签用于更新,并在四种近期 TTA 方法(TAFAS、COSA、PETSA 和 DynaTTA)的已发布代码中强制执行这一规则;这些方法均使用各自的骨干网络和检查点,并在五个基准数据集(ETTm1、ETTh2、Weather、Electricity 和 Traffic)上运行。作为参考,我们加入了两个闭式校正器:一个由递归最小二乘(RLS)滤波器组构成、再结合逐坐标中位数的校正器,它无需可调超参数,在 7 通道数据流上每步耗时 56 微秒;以及一个 ELF-style 线性校正器。在因果延迟标签条件下,结果呈现出不对称性。在 ETTm1 上,所有接受审计的方法都确实实现了自适应,但 RLS 滤波器组仍以远低于它们的成本击败了其中三种;只有 DynaTTA 能击败该滤波器组,而且仅在最小因果延迟下实现,其每次更新成本约为前者的 2,500 倍;ELF-style 校正器则击败了这四种方法。在其他四个数据集上,任何已发表方法相对于其自身冻结检查点所取得的最大统计显著提升均为半个百分点;在这四个数据集上,至少有一种已发表方法在最小因果延迟下显著差于冻结模型;而在漂移严重的 ETTh2 上,较长的标签延迟会使所有与冻结模型存在差异的适配器(包括我们的方法)显著有害。存在泄漏的下一步更新会使简单适配器的表观增益最多膨胀 110%;而骨干网络的训练方案会使冻结模型的在线误差最多相差 25 倍,这一影响超过了我们测得的任何自适应效果。我们发布了该测试框架、集成补丁以及所有缓存的运行结果。

arXiv cs.LG71

Verification with Transfer: Exact Information Frontiers and Their Price in Calls

arXiv:2610.12211v1 Announce Type: new Abstract: A verifier that accepts or rejects whole answers reveals little: under a flat prior over $k$-bit answers, zero error needs $2^k-1$ verifications. The usual remedy is to solve related source tasks, either all first, as a curriculum does, or interleaved with verification. We price this remedy in information and in calls. With an exact verifier, the least causal information that any interleaving of source calls and $n$ verifications needs to succeed with probability $s$ is a list rate-distortion function, attained by one observation before any verification. It lower-bounds the expected number of binary source calls, which designed sources meet within $1+\log_25$ calls for unique answers and within a logarithmic term in general, where no additive constant suffices. With an exact verifier and fixed sources, moving every call before the first verification preserves all hard caps on calls, although interleaving can save unboundedly many expected calls; under a noisy verifier, source-first protocols can lose unbounded factors in information and in error. For linear banks over $\mathbb{F}_2$, optimal accuracy has a closed form, and after a polynomial-time reduction the budget profile is computable in time $2^{O(h^2)}\operatorname{poly}(J,k+h)$ for $J$ sources and nuisance dimension $h$. In these banks, for zero error under a hard cap, the calls beyond the rounded-up information price are exactly those spent on nuisance. Every numbered result apart from two clauses about the planner is machine-checked in Lean 4, assuming two published results. Used as a ruler, the frontier shows a small transformer using all delivered bits at latent dimension $5$ and none at $11$ within fixed training budgets; in a test with predictions recorded before training, low XOR degree of the target bits did not suffice for their use.

arXiv cs.LG71

DataSense-Bench: The First Step Toward an AI Scientist

arXiv:2610.12190v1 Announce Type: new Abstract: As claims about recursive self-improvement (RSI) and artificial general intelligence (AGI) proliferate, we ask a simple question: do frontier AI models have a sense of data, i.e., can they reliably select the right data for training? We introduce DataSense-Bench to study this capability through the fundamental problem of data selection and performance forecasting in machine learning. We ask AI agents to select and rank candidate training subsets that can be used to fine-tune a small LLM model. Agents are allowed to inspect the data, write and execute analysis code, and run model forward passes, but can not train the model or access the actual evaluation tasks. We then fine-tune the base model on each selected subset and evaluate its post-training performance under a standardized protocol. We instantiate the benchmark in terminal problem solving and tool use, selecting trajectories from OpenThoughts-Agent and EnvScaler and evaluating on TBLite and BFCL, respectively. We then evaluate the agents along two complementary dimensions: the post-training performance of the top-ranked subset, reflecting the ability to identify high-value training data, and ranking accuracy, reflecting the ability to predict the relative performance of the selected subsets. In our experiments, selection gains over random selection are limited; agents do not reliably rank their selected groups, and ranking ability does not hold consistently across tasks: Astra identifies the best group in all three tool-use runs but in only one of three terminal runs. Analysis of execution traces on both tasks shows that agents often use similar data signals while interpreting their training value differently.

arXiv cs.LG75

Just Weather Scoring: Efficient End-to-end Nowcasting with Distributional Diffusion

arXiv:2610.12189v1 Announce Type: new Abstract: Generative diffusion models are well-suited for probabilistic precipitation nowcasting, but existing approaches often rely on separately trained compression or deterministic forecasting components and remain costly at inference due to iterative denoising. We introduce Just Weather Scoring (JWS), a single-stage, end-to-end diffusion model which addresses both issues by forecasting directly in radar space and enabling few-step generation. Radar-space modeling greatly simplifies training and inference and eliminates uncertainty arising from lossy compression. JWS combines Masked Asynchronous Diffusion, a timestep-sampling scheme that preserves clean context while adapting diffusion training to high-dimensional spatio-temporal data, with a simple scoring-rule objective that aligns training with probabilistic forecasting and unlocks few-step generation. On the SEVIR and MeteoNet benchmarks, JWS achieves state-of-the-art probabilistic forecasting performance at reduced training and inference cost. Even our smallest model remains competitive using substantially fewer parameters and more than 17x faster inference.

arXiv cs.LG79

A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization

arXiv:2610.12183v1 Announce Type: new Abstract: Black-box optimization (BBO) arises in many scientific and engineering problems where objective evaluations are expensive and limited. Recent large language model (LLM) agents offer a new way to approach BBO by combining task semantics, computation, optimization tools, and feedback-driven decision making, showing great potential due to the integration with mathematically rigorous tools. However, existing agentic BBO studies use different task domains and system configurations, making their results difficult to compare and the effects of individual design choices hard to isolate. We therefore introduce AgenticBBO-Bench, a cross-domain benchmark for agentic BBO spanning synthetic functions, hyperparameter optimization, database tuning, chip design, and molecular design under a unified finite-budget evaluation protocol. In our experiments, agentic BBO achieves higher family-averaged scores than direct LLM-based methods in all five domains and outperforms the best numerical optimizers in four. We further study three factors shaping agent performance: optimization tools, task information and prior knowledge, and the role of the LLM during search. Our results show that additional numerical tools do not consistently improve performance, task semantics are broadly useful while more specific priors are less reliable, and numerical optimizers can effectively absorb gains from search trajectories established by the agent. Finally, we introduce a five-task frontier challenge within AgenticBBO-Bench and evaluate seven LLMs under the Codex agent harness, where GPT-6 Astra and DeepSeek-V4.1-Flash lie on the Pareto frontier of performance and cost among the evaluated models. Our code is available at https://github.com/lamda-bbo/agentic-bbo.

arXiv cs.LG78

通用图异常检测是否需要真实世界训练数据?

arXiv:2610.12167v1 公告类型:新 摘要:通用图异常检测(GAD)旨在构建一种基础模型,无需重新训练或微调即可检测任意未见图中的异常。然而,充足的数据对于基础模型训练至关重要,而通用 GAD 仍面临数据短缺问题,因为真实世界的异常图稀缺且收集和标注成本高昂。为弥补这一缺口,我们提出了 AG-FORGE,一种用于自动合成异常图的异常图生成框架,探索由合成数据驱动的通用 GAD 训练的可行性。实证结果表明,合成数据能够达到与真实世界训练相当的性能,但由于现有方法的能力有限,无法进一步突破性能上限。为在训练数据规模扩大时进一步释放模型能力,我们开发了 TS-GGAD,这是一种拓扑—语义协同的通用 GAD,能够捕获互补的拓扑异常证据和语义异常证据,并结合了针对大规模合成训练设计的课程学习策略。在 14 个真实世界数据集上的大量实验表明,使用 AG-FORGE 生成的数据进行训练的 TS-GGAD 显著优于当前最先进的方法。

arXiv cs.LG78

Scalable Hierarchical Graph Generation via Soft Community Structure

arXiv:2610.12163v1 Announce Type: new Abstract: Generating large attributed graphs requires reproducing the topology, generating attributes jointly with the structure, and remaining scalable. Many real-world graphs exist as a single large graph, so a generative model has to generalize from the one graph it is fit on, without independent samples. We present Schema, which recursively decomposes a reference graph into a hierarchy of soft communities, assigning each node a membership distribution. Generation is then split into three stages, each trained independently: (1) synthesizing node attributes conditioned on soft memberships, (2) generating intra-community edges from local structural context, and (3) modeling inter-community connections over bridge nodes whose membership mass is distributed across several communities. No stage forms the full adjacency matrix, and each stage operates on a subgraph bounded by the community size. We also introduce an evaluation protocol that covers structural fidelity, memorization, downstream utility, and scalability. On four real-world attributed graphs, Schema recovers the balance between local and long-range structure more closely than any other model that generates attributes, while reproducing only a small fraction of the reference edges. It retains the downstream accuracy of the reference graph without raising it artificially above that level. Baselines that match its structural fidelity memorize the reference, while those with higher downstream accuracy either exceed the reference accuracy or fail to complete on the larger graphs. We measure scalability on six additional graphs with up to 10 million nodes.

arXiv cs.LG72

When KL Regularization Misfires in Group Policy Optimization

arXiv:2610.12161v1 Announce Type: new Abstract: Why does removing reference-policy KL regularization sometimes improve group policy optimization? This motivates studying how reference-policy information should enter group-relative updates. We analyze seven potential failure modes in the interactions between KL and rewards: residual KL updates after reward clipping, after gradient cancellation, and in groups with identical rewards; KL growth with response length and an imbalance in its relative contribution; KL concentration on a small number of tokens; and sampling noise when k1 is incorporated into rewards. We propose Zero-Sum Calibrated Policy Optimization (ZCPO), which uses relative drift measured by conditional KL to calibrate within-group reward coefficients and integrates them into the base surrogate. Mathematical reasoning experiments and ablations support this design's effectiveness in our settings.

arXiv cs.LG67

Toward Optimal Regret in Adversarial MDPs with Stochastic Hard Constraints

arXiv:2610.12153v1 Announce Type: new Abstract: We study episodic constrained Markov decision processes with adversarial losses under stochastic hard constraints. Specifically, starting from a known strictly feasible policy with margin $d$, we seek to obtain optimal regret while satisfying the expected cost constraints in every episode. In this setting, Stradi et al. (2025) show that a carefully designed mixing rule attains regret of order $\widetilde{\mathcal{O}}(\sqrt{T}/\min\{d,d^2\})$. Interestingly, they also provide a lower bound of order $\Omega(\sqrt{T}/\rho)$ for the same setting, where $\rho$ is the Slater margin of the offline problem and can be much larger than $d$. In this work, we build on their approach to obtain optimal regret dependence on these margins. Specifically, we propose MA-OPS, an algorithm that combines an optimistic search for the Slater margin with a pessimistic evaluation of the selected policies to safely learn a policy with a large feasibility margin. This policy is then used to minimize regret while satisfying the constraints at every episode. In particular, we show that MA-OPS attains regret $\widetilde{\mathcal{O}}(\sqrt{T}/\rho + 1/(d\rho))$. Finally, we provide a matching lower bound, showing that the dependence on $T$, $d$, $\rho$ in the regret bound is optimal up to logarithmic factors.

arXiv cs.LG74

状态保持约束下的贝叶斯优化

arXiv:2610.12150v1 发布类型:新 摘要:在许多工程设计问题中,目标和约束取决于状态,即由设计参数决定的偏微分方程(PDE)解。我们研究在使选定状态观测量保持在可信值附近的同时改进设计的问题,并将其称为状态保持约束。带约束的贝叶斯优化通过学习得到的可行性模型处理这类约束,但面对该问题高度各向异性的可行域时表现欠佳。我们的核心思路是预先计算控制量集合,使其线性化约束响应保持在容差范围内,从而将状态空间中的约束映射回设计空间。这种线性化定义了一个椭球体,我们可以从中高效地抽取大量分布均匀的候选点。底层的线性响应映射会在线更新,并据此重新构建椭球体。我们以关键应用——在保持等离子体边界的条件下进行 Tokamak 偏滤器优化——对该方法进行了端到端验证。

arXiv cs.LG71

Large-Scale Benchmarking of Quantum Neural Network Configurations for Financial Time Series Forecasting

arXiv:2610.12148v1 Announce Type: new Abstract: Quantum machine learning, and quantum neural networks (QNNs) in particular, are advancing fields with growing potential. Although systematic comparisons of QNN configurations have been explored primarily for classification tasks, comparatively little attention has been given to regression problems, particularly financial time series forecasting. This study presents a large-scale systematic comparative evaluation of QNN component configurations for financial time series forecasting, using the GBP/USD spot exchange rate as a case study. A grid search across encoding methods, ansatz designs, qubit counts, layer depths, and cost functions yields 1,368 distinct model configurations, each evaluated in terms of prediction accuracy, computational cost, and convergence behaviour. The results reveal unique insights into how the choice of methods influences performance, such as that gate selection and arrangement are more critical to model success than raw parameter count, and that entanglement is a system-level property of the full circuit rather than solely at the ansatz level. The best-performing QNN configuration achieves an $R^2$ score of 0.985, outperforming a classical BiLSTM baseline. Additionally, the impact of real quantum hardware noise is assessed through execution on the IQM Emerald device, revealing that gate errors and decoherence represent a significant barrier to practical deployment, with gate selection and circuit depth identified as key determinants of hardware noise resilience. Overall, the findings provide practical architectural guidance for QNN design and establish a baseline characterisation of QNN noise sensitivity on near-term quantum devices.

arXiv cs.LG70

Using Weisfeiler-Leman Features for Algorithm Selection in Constraint Optimisation

arXiv:2610.12119v1 Announce Type: new Abstract: Algorithm Selection is essential for efficient Constraint Programming. Over the years, many algorithm selectors based on machine learning methods have been successfully applied, yet traditional feature extraction methods often rely on manually decided instance-level statistics that fail to capture the underlying problem structure. In this paper we aim to bridge this gap by introducing a novel, automated feature extraction methodology that integrates graph conversion and Weisfeiler-Lehman graph kernels to generate robust structural representations of problem instances. The 1-WL test bounds the graph-distinguishing power of standard message-passing Graph Neural Networks (GNNs), and suitable GNN architectures match this bound \citep{Xuetal2018}. WL-based features offer an alternative that does not require training a GNN. Our primary contribution is a cut-based representation (\texttt{WLc}) designed to model structural partitions and provide a more nuanced predictive signal. We evaluate our approach on instances from the 2023--2025 MiniZinc Challenges across two tasks: maximizing Borda count scores and maximizing predictive accuracy. Experimental results across Support Vector Machines, Random Forests, and Multi-Layer Perceptrons demonstrate that cut-based features outperform \texttt{fzn2feat} with SVMs, while results with RFs and MLPs are closer.

arXiv cs.LG72

Credal Machine Learning for Risk-Averse Decision Making

arXiv:2610.12115v1 Announce Type: new Abstract: In many machine learning applications, it is necessary to guard against worst-case scenarios and predictions that could result in substantial losses. In principle, this can be achieved by training risk-averse predictive models that minimize loss functions such as conditional value-at-risk (CVaR), rather than relying on models that perform well on average. In practice, however, the effectiveness of this approach to risk aversion is undermined by the learner's uncertainty regarding the true loss distribution and, consequently, the true CVaR. To achieve reliable risk-aversion, we propose a method in which this (epistemic) uncertainty is represented in terms of credal sets, i.e., sets of probability distributions. More specifically, we develop an efficient yet reliable learner that produces predictions in the form of credal sets and combine it with a novel decision rule that maps each credal set to a single predictive distribution for CVaR minimization. Across classification, under distribution shift, and in reinforcement learning, our approach reliably avoids catastrophic decisions, while sacrificing little in expected performance.

arXiv cs.LG65

A Geometric Approach to Soft Actor-Critic with Zonotopes for Locomotion Learning

arXiv:2610.12113v1 Announce Type: new Abstract: Off-policy actor--critic methods control overestimation bias by taking the minimum of two critics. This uses the same aggregation rule everywhere, regardless of how the critics disagree. We propose \textbf{GeZo-SAC}, which uses auxiliary geometric representations to adapt critic pessimism to the state and action. Alongside its scalar value, each critic predicts a set of generators defining a zonotope. Probing this zonotope along sampled directions provides a geometric width, "subtracted from each critic value as a pessimistic offset, and a measure of disagreement between the two critics, aggregated with log-sum-exp. This disagreement controls how the critics are combined, moving from a width-weighted average toward the usual minimum as disagreement increases. At inference, the deployed policy is an unmodified SAC actor, since the generators are used only on the critic side during training.Across four MuJoCo-v5 locomotion benchmarks and six off-policy baselines, GeZo-SAC achieves the highest mean return on Ant-v5 and Hopper-v5 and remains competitive with other methods on the remaining tasks. Our analysis further shows that GeZo-SAC achieves the lowest average actuator work and action effort per metre among the evaluated methods, while maintaining near-zero measured overestimation frequency across all four environments.

arXiv cs.LG75

通过数据空间迭代实现少步生成

arXiv:2610.12102v1 公告类型:新 摘要:流匹配已成为训练高质量生成模型的一种可扩展范式,但从所学习的概率流中进行采样需要进行多次网络评估。蒸馏可以将这一成本降低到一次或少数几次评估;然而,单步生成通常会牺牲质量,因此少步生成成为实际应用中的主要工作模式。现有的少步方法沿概率流执行迭代计算,因而需要固定且由人工选择的时间步离散化方案。这种离散化通常依赖启发式方法进行选择,调优成本高;当不同样本或空间位置的细化难度存在差异时,它还可能形成限制。我们提出了数据空间迭代,一种彻底消除流离散化的少步生成框架。从噪声开始,共享生成器直接在数据空间中细化其预测结果,并在每次迭代中进行训练,以生成在其模型容量允许范围内的最佳样本。我们的方法只需极少改动即可与分布匹配蒸馏(DMD)相结合,从而能够在匹配的训练设置下,对不同迭代方法进行受控比较。在类条件 ImageNet 256x256 任务上,数据空间迭代优于标准离散化基线,并达到或超过通过调度搜索选出的变体,同时无需针对特定调度方案进行训练。这些结果表明,对于快速生成,数据空间迭代是离散化流空间迭代的一种简单而有效的替代方案。

arXiv cs.LG77

SCORE: Spectral Correlation Estimation for Multivariate Gaussians

arXiv:2610.12096v1 Announce Type: new Abstract: Neural network-based predictive modeling with high-dimensional structured Gaussian targets requires an efficient and numerically stable, yet expressive approximation of the covariance matrix. We propose SCORE: a scalable framework, combining scoring rule training with an expressive covariance approximation learned in spectral space. For $d$-dimensional data, the learning task is decomposed into learning the marginal distributions and learning a structured correlation matrix, which enables dense dependencies with linear storage and $\mathcal{O}(d\log d)$ cost. We utilize the closed form Gaussian kernel score for training, which remains defined even for degenerate covariances and admits bounded gradients during optimization. We characterize kernel scores under invertible transforms and prove exact invariance under unitary transforms. At population level, our two-level objective recovers the true marginals and projects the target correlation onto the representable class; finite-sample PAC bounds show that the errors of the two stages enter additively. We evaluate our model on a variety of tasks with a commonly assumed Gaussian domain: Time-series forecasting, monocular depth estimation, and spatial weather prediction, showing improved performance at lower computational cost.

arXiv cs.LG75

利用梯度进行昂贵模拟器的贝叶斯推断

arXiv:2610.12076v1 公告类型:新 摘要:基于微分方程的模拟器广泛应用于科学和工程领域。它们常用于基于模拟的推断,根据对模拟器输出的现实世界观测结果评估输入参数的后验分布。然而,当单次模拟器评估的计算成本高昂时,推断会变得极具挑战。在这种情况下,研究人员采用了基于贝叶斯优化的主动学习方法,并结合高斯过程代理模型,以最大化从有限模拟预算中获取的信息。近年来,模拟器输出相对于输入参数的梯度越来越容易获得,但这些梯度很少被用于推断。尽管我们只需学习模拟器的输入—输出关系,梯度信息仍可提供额外的宝贵信号,用于指导主动学习过程。在昂贵模拟器的场景中,这一点尤其重要,因为样本效率至关重要。本文展示了如何将梯度信息融入高斯过程代理模型,从而在有限模拟预算下加速基于贝叶斯优化的推断。结果表明,利用梯度信息能够显著提高收敛速度。对于反向模式微分,在考虑额外计算成本后,推断效率的提升仍然得以保持。相比之下,对于正向模式微分,推断速度的提升不足以抵消计算成本。这些结果表明,梯度增强代理模型主要适用于参数数量超过输出维度的问题,因为在此类问题中反向模式微分更为高效。

arXiv cs.LG74

MPGE:用于分子分类解释的多视角图解释器

arXiv:2610.12039v1 公告类型:新 摘要:图神经网络(GNN)根据化学图数据预测分子性质,但预测准确性并不能解释图信息如何支持某个具体决策。能够保持预测结果不变的紧凑依据,并不一定能揭示哪些变化会逆转决策,或模型能够容忍哪些修改。我们提出多视角图解释器(MPGE),针对冻结分类器统一事实支持、反事实敏感性和示例容忍性。事实视角最初称为原型(PT),旨在寻找一个紧凑的保留边集合,使标签和所需置信度保持不变。反事实(CF)解释旨在寻找有界的、能够改变预测结果的删除操作;示例(EXE)解释旨在寻找非平凡的有界删除操作,同时保持标签和置信度不变。一个共享的约束形式将预测行为、紧凑性和编辑成本联系起来,同时通过独立目标生成这三种视角。我们对 CF-GNNExplainer 的图分类扩展学习对称的边排序,并验证离散候选解,同时记录未成功的搜索。一个独立的 BBBP 片段后端返回经过 RDKit 清理的分子。我们在 MUTAG、Mutagenicity、AIDS、COX2_MD 和 BBBP 数据集上,使用语义覆盖率、条件质量、稳定性和运行时间评估主要的 GCN 实现。跨数据集来看,成功的事实掩码平均保留输入边的 8.6%--15.5%;有界反事实覆盖率为 4.8%--67.6%,示例保持覆盖率为 98.9%--100.0%。探索性控制实验揭示了硬投影和保留节点信息的影响。定量比较和分子可视化刻画了模型的支持、敏感性和容忍性,但并未将其视为经过验证的化学机制。

arXiv cs.LG71

CausalDreamer:通过潜在解耦学习预测性世界模型

arXiv:2610.12016v1 公告类型:新 摘要:用于控制的世界模型必须捕捉环境中哪些方面会响应智能体的动作,以及哪些方面与奖励相关。Dreamer 4 等生成式世界模型由视频标记器和动力学模型组成:视频标记器将每一帧编码为一个潜变量,动力学模型则经过预训练,根据过去的潜变量和动作预测未来的潜变量。然而,标记器采用重建目标进行训练,没有动作或奖励监督,因此其潜变量没有明确的机制来区分可控、不可控、与奖励相关以及与奖励无关的信息。我们提出 \textit{CausalDreamer},在冻结标记器的同时,将其潜变量重新编码为由两条轴上的四组信息构成的因子化表示:一条轴是可控性,只有其中两个可控组接收动作信息;另一条轴是奖励相关性,通过根据两个与奖励相关的组预测奖励来学习。随后,我们对预训练的动力学模型进行微调,使其预测该因子化表示。我们在 20 个 MMBench2 任务上使用基于模型的预测规划,对 \textit{CausalDreamer} 及其所基于的预训练世界模型进行评估:其中 10 个是训练期间见过的干净任务,另外 10 个是未见任务;未见任务中有 6 个是干净任务的变体,改变了背景、物体或迷宫布局,另有 4 个是全新的环境。我们对回报进行归一化,使采取均匀随机动作的策略得分为 0,专家得分为 1。在干净任务上,\textit{CausalDreamer} 的归一化得分比预训练世界模型高 14%(0.199 对 0.175);在变体任务上高 25%(0.307 对 0.246)。然而,在新环境中,两个模型的得分均未明显高于随机策略。此外,我们的分析表明,该因子化表示能够将与奖励无关的变化(例如背景变化)与其奖励相关组区分开来。

arXiv cs.LG77

表格基础模型的测试时计算:机制、收益与局限

arXiv:2610.12005v1 公告类型:新 摘要:哪些形式的测试时计算能够改善强大的预训练表格基础模型(TFMs)的预测?我们从三个轴向对此进行了系统研究:适应、聚合和上下文构建。评估涵盖 TabArena 基准测试中的现代 TFM,并辅以在 OpenML 上的宽表和大规模表格实验。在适应方面,我们提出了 DiagScale,一种对角查询—键相似度更新方法。该方法仅训练模型参数的 0.003%–0.03%,却在三个独立预训练的骨干模型上取得了与全量微调相当的增益。在聚合方面,候选池的构成和选择策略都很重要。TabPFN-3 已经对同一数据的不同预处理变体进行预测并取平均,继续增加此类预测带来的收益会逐渐递减。在包含 96 种配置的更大候选池中,贪心选择相较于默认预测器可将误差降低 2.4%,但均匀平均反而会增加误差。在上下文构建方面,注意力引导的检索改善了 TabPFN-3 在部分大型表格上的预测,并支持使用超出完整上下文记忆容量限制的来源池。我们测试的上下文扩展方法未带来一致的改进。总体而言,我们的结果表明,适应和选择性聚合能够带来稳定的基准级增益。在同一骨干模型上结合适应与聚合还能获得进一步提升,但其所需计算量显著高于默认推理。这些权衡促使人们根据可用的计算预算选择相应策略。代码可在 https://github.com/kanghui-learning/test-time-compute-for-tabular-foundation-models 获取。

arXiv cs.LG72

The Polytopal Neural Network

arXiv:2610.12004v1 Announce Type: new Abstract: Understanding how deep neural networks process information remains a central challenge. Existing interpretability methods often compromise structural fidelity, rely on prespecified corpora, or explain models post-hoc. We propose Polytopal Neural Networks (PNNs), a framework that extracts distinct layer-wise aspects by enforcing a polytope-based structure that is used directly in subsequent information processing. We scale our approach using learned corpus representations and an amortized simplex inference procedure and highlight how the framework also gives a direct route to vector quantized (VQ) training. In PNNs, observations are explicitly described by their alignment with layer-specific aspects. Empirical results show that imposing polytopal constraints on neural network representations preserves meaningful structures in the latent space with minimal degradation in performance, favorable compressed representations when compared to VQ representations in unsupervised learning, while also providing a performant new approach to VQ deep learning training. Our findings suggest that deep networks can enforce interpretable polytope-based representations, offering a principled path toward more transparent AI systems with minimal performance compromise.

arXiv cs.LG75

Agentic-TTT: Training test-time policy for test-time training

arXiv:2610.12002v1 Announce Type: new Abstract: Test-time training (TTT) adapts an LLM's parameters using signals derived from test inputs, and can make striking improvements in pre-specified settings such as IMO competitions or designated open problems. By turning deployment experience into parameter updates, TTT provides a direct mechanism for model-level self-improvement. Yet TTT is not universally beneficial: each TTT algorithm works in different settings, and applying an ill-suited method could waste test-time compute or even damage model performance. Therefore, such parameter-level self-improvement requires agency: the model must decide when TTT is warranted, which algorithm to invoke, and whether an existing skill can be reused. To fill this gap, we introduce Agentic-TTT, which learns a test-time policy to govern those decisions. Agentic-TTT turns TTT procedures into callable tools, treats accumulated skills as an evolving deployment environment, and trains its policy using the observed utility gains from its decisions. On our benchmark, Agentic-TTT nearly doubles the utility over the backbone model, learns to trade off utility against compute, and generalizes to domains unseen during training. Together, these results point toward autonomous self-improvement: models that can decide how to learn from their own deployment experience.

arXiv cs.LG68

Example-driven Parametrisations for Bayesian Shape Optimisation

arXiv:2610.11984v1 Announce Type: new Abstract: Bayesian optimisation is the natural tool for shape design when objectives are expensive and non-differentiable, but it needs a compact yet expressive parameterisation of the search space. Hand-crafting one is a complex endeavour requiring domain expertise, and often yields implicit infeasible regions, artificial bounds, and coupled, unordered coordinates. We instead learn the parameterisation from a collection of existing designs, applying principal component analysis to the deformations between shapes. The result is a linear, interpretable search space in which the number of components explicitly trades expressivity against dimensionality. Across aerofoils, wings, and radio-frequency cavities, spanning 2D geometry to 3D aerodynamics and electromagnetics, we show improved sample efficiency and the ability to explore beyond the confines of hand-crafted baselines.

arXiv cs.LG72

Stochastic Grouping Conformal Prediction for Effective Subgroup Reliability

arXiv:2610.11957v1 Announce Type: new Abstract: Conformal prediction offers a distribution-free coverage guarantee, making it especially attractive for clinical applications. Standard conformal prediction, however, provides such guarantees only at the population level, and its prediction sets can exhibit coverage disparities across clinically important subgroups. A natural remedy is to calibrate within predefined groups. However, this can require access to sensitive subgroup attributes and is prone to a worst-group bottleneck: protecting the most difficult subgroup can inflate prediction sets for all, increasing cognitive burden on decision makers. To this end, we propose Stochastic Grouping Conformal Prediction (SGCP), a conformal framework for subgroup-reliable uncertainty quantification. It learns a stochastic grouping map that allows each sample to draw calibration information from others with similar calibration behavior, yielding a local score law that boosts reliability across subpopulations. We prove that SGCP retains the standard coverage guarantee. Experiments on synthetic and real-world benchmarks show that it consistently reduces subgroup coverage gaps while achieving smaller or comparable prediction set sizes relative to existing baselines.

arXiv cs.LG78

基于树的模型中的区间值 SHAP

arXiv:2610.11953v1 发布类型:新 摘要:沙普利值是最受欢迎的特征归因解释方法之一。针对基于树的模型,人们已经开发出高效的沙普利值计算与估计方法;对于表格数据集而言,这类模型代表了当前最先进的水平。然而,已知沙普利值可能因微小且现实的变化而表现出(高度)不稳健性。本文提出一种基于不精确 Dirichlet 模型(IDM)的方法,用于分析决策树和随机森林中沙普利值的稳健性。具体而言,当少量未标注实例被随机引入树的叶节点时,我们通过量化和分析区间值沙普利值来实现这一目标。区间值沙普利值可以遵循处理不完整数据时的常见原则来定义:悲观原则和平均原则。我们推导出各种理论结果,从而实现区间值沙普利值的高效计算。我们还表明,所提出的方法可以直接推广到班扎夫值的情形。随后,我们通过多个案例研究和实验,展示所提出的区间值沙普利值的行为,以及其在消除无信息特征偏差方面的应用。

arXiv cs.LG79

Puffin:从粗粒度观测中概率学习空间细节

arXiv:2610.11914v1 公告类型:新论文 摘要:高分辨率社会经济变量对城市规划、公共卫生、灾害响应和资源分配等应用至关重要。然而,在实践中,这些变量往往只能以较低的空间分辨率进行观测。我们提出了 Puffin,这是一种用于统计分解的概率框架,以高分辨率卫星嵌入作为协变量,提升粗粒度总量数据的空间分辨率。Puffin 并非为每个细分辨率子区域预测单一数值,而是学习概率分布,并通过考虑聚合关系的似然函数进行训练。在推断阶段,Puffin 以观测到的区域总量为条件调整这些预测,并将总量分配至各子区域。由此得到的精细尺度估计既与观测到的聚合总量一致,又具备经过校准的不确定性,且训练时无需细分辨率标签。我们使用德国和美国的人口普查、就业及选举数据,对 Puffin 在人口、就业岗位及其他计数变量上的表现进行评估,并研究统计分解在不同地区、国家和目标变量上何时成功、何时失败。

arXiv cs.LG66

Automated Assembly Instruction Generation from CAD Models Using Grounded Large Language Models: A Human-in-the-Loop Framework

arXiv:2610.11896v1 Announce Type: new Abstract: Assembly documentation is a downstream manufacturing artifact that is still usually authored by interpreting CAD models by hand. Structured product data and large language models are both available, yet studies of CAD interpretation, assembly sequence planning, instruction writing, and human oversight have largely proceeded separately. This paper formulates CAD-grounded assembly instruction generation: the production of natural-language assembly procedures constrained by structured engineering information extracted from CAD models. The proposed framework maps a STEP assembly to a typed ProductGraph intermediate representation, derives a precedence order by deterministic topological sorting, realizes each step as language conditioned only on selected graph context, attaches per-step visual documentation, and applies rule-based and model-assisted checks. PDF export remains disabled until a human reviewer resolves every quality flag. The case study establishes endto-end feasibility on a built-in six-part reference assembly: the pipeline preserves a reported assembly order and carries quantity, material, and torque into an exported manual page. Generalization and geometric validation remain open empirical questions. The contribution is an architecture that separates engineering state, deterministic reasoning, grounded language realization, verification, and human release.

arXiv cs.LG70

Understanding Latent-Dimension Scaling in Dynamical-System Learning through Spectral Reliability

arXiv:2610.11866v1 Announce Type: new Abstract: In deep learning, approximation theory motivates increasing representation size. We ask whether this benefit extends to dynamics learning through autoregressive prediction. We analyze the learned time evolution through the eigenstructure of Koopman operators, using relative residuals to detect spurious eigenpairs arising even as one-step error falls. For bounded Koopman operators, we show that minimal residuals over learned dictionary spaces converge pointwise to their full-space counterparts as these spaces approximate the observable space in $L^2$. Our hypothesis is that Koopman spectral reliability helps explain how consistently rollout error decreases with increasing dimension. We compare two models of a shared Koopman autoencoder trained alternately for reconstruction and latent evolution, using latent-prediction loss (one-step prediction errors in latent coordinates) or spectral-residual loss (relative residuals of candidate eigenpairs). Across six chaotic systems, both models reduced median windowed rollout error from smallest to largest dimension. The spectral-residual model achieved lower medians than the latent-prediction model for all systems and dimensions, and its median fell by a larger factor in every system. Its median decreased monotonically with dimension in four systems, against one for latent prediction. Against four baseline families, its mean valid prediction times were nearly always longer. At the largest dimension under two-stage training, we compared eigenvalue positions with each learned dictionary's residual contours. Spectral-residual eigenvalues concentrated in low-residual regions, whereas latent-prediction eigenvalues also appeared in high-residual regions, consistent with the hypothesis.

arXiv cs.LG72

GRPODropout: Less is More for Online Reinforcement Learning Rollouts

arXiv:2610.11854v1 Announce Type: new Abstract: Reinforcement learning (RL) methods such as GRPO substantially improve large language model reasoning but often suffer from policy entropy collapse: the loss of sampling diversity weakens exploration and limits further improvement. Existing methods address this issue either through algorithm-level interventions, such as reward modification and entropy/KL regularization, or through token-level reweighting. We investigate a complementary perspective: entropy collapse can also be mitigated by changing which generated rollouts contribute to policy updates. Under the same sampling budget, not all rollouts contribute positively to an update, and selectively excluding some can improve learning. To address this, we propose GRPODropout: before the standard update, we use a simple strategy that selectively removes a small number of high-probability positive-advantage rollouts and recenters the retained advantages. To motivate this design, we develop a rollout-level theoretical analysis that guides method design and threshold selection. The method changes only rollout usage, and adds negligible computational overhead. Experiments show higher accuracy than original GRPO and higher actor entropy while using fewer rollout samples for updates, illustrating "less is more." This work provides insight into RL rollout usage: removing some rollouts can improve performance. Code is available at https://github.com/hexuandeng/GRPODropout/.

arXiv cs.LG71

DADP:动态活动依赖剪枝,一种受反向 Hebbian 启发的结构化剪枝方法

arXiv:2610.11853v1 公告类型:新 摘要:现代神经网络通常存在严重的过度参数化问题。这种冗余会在训练和推理过程中带来大量的计算与内存开销。现有剪枝方法依赖事后幅值阈值或静态初始化启发式方法。因此,它们通常需要为每一层手动设定稀疏度目标,或进行代价高昂的再训练循环。我们提出 Dynamic Activity-Dependent Pruning(DADP),一种受生物学启发的结构可塑性机制。在训练过程中,DADP 通过累积突触前激活值与突触后误差梯度的乘积来衡量连接的重要性。DADP 使用单一的全局阈值,而不是固定的分层稀疏度预算,从而能够在网络深度方向动态分配稀疏度,并自然地实现神经元级和通道级剪枝。在 MLP、VGG-16、ResNet-18、BiLSTM-CRF 和 MiniBERT 架构上,DADP 的性能达到或优于 Magnitude、SNIP 和 RigL;在 ResNet-18 上达到 99% 稀疏度时,仍保留 73.67% 的准确率(稠密基线:76.06%)。最后,基于矩阵的 Shannon 熵和有效秩测量结果证实,在极高稀疏度下,DADP 仍能保持潜在特征的多样性,而不会发生表示坍塌。