图像生成

AesCode：采用解耦跨模态奖励的审美代码生成

将信息丰富的视觉产物（如幻灯片、海报和仪表板）生成为代码颇具吸引力，因为输出具有结构化、可编辑且可验证的特点。然而，代码模型无法看到其布局和配色选择在画布上的整体呈现，这往往限制了空间规划能力和审美质量。图像生成器擅长视觉设计，但可能错误渲染用户要求的内容，因此其输出不适合作为直接复现的可靠目标。为此，我们提出 AesCode：在内容提示的基础上生成代码，同时使用由同一提示生成的图像作为审美指导。具体而言，AesCode 将目标表示为设计图，以描述整个画布，并使每个属性都具备可追溯性。基于该设计图，我们推导出互补的跨模态奖励：确定性验证器对可从代码和渲染结果中解析的属性进行评分，而 VLM 评审器则使用针对样本定制的视觉图评分标准（Visual Graph Rubric，VGR），依据图中的元素和关系评估不可解析的属性。我们通过带有按通道归一化机制的强化学习优化这些解耦奖励，从而共同提升空间规划、审美质量和可验证的正确性。为评估这一设定，我们构建了一个基准，分别对可验证的正确性和视觉质量进行评分，其中视觉评分与人类判断高度一致。我们发现，参考图像能够显著提升现有模型生成代码的视觉质量；消融实验表明，AesCode 从参考图像中获益，不仅因为将其作为模型输入，也因为将其作为视觉奖励的来源。与使用相同参考图像、规模相当的 Qwen3-VL 基线相比，AesCode-8B 和 AesCode-32B 的视觉质量分别提高了 31.1 和 22.4 个百分点，达到与 GPT-5.5 和 Claude Opus 4.8 等闭源前沿模型相当的评分。

Deepgram targets voice AI on air-gapped laptops

Voice AI is quickly becoming core infrastructure as people swap screens and clicks for conversations with their devices. That shift puts new pressure on the compute, storage and partnerships behind the models doing the listening and talking. Deepgram Inc., which expanded its Flux model to 10 languages earlier this year, needs infrastructure for both its […] The post Deepgram targets voice AI on air-gapped laptops appeared first on SiliconANGLE .

Lesion-DDPM：用于多发性硬化 MRI 合成的病灶增强型三维扩散

arXiv:2606.15457v2 公告类型：replace-cross 摘要：三维 FLAIR MRI 被广泛推荐为多发性硬化（MS）脑成像的标准 MRI 序列之一，但公开可用的 MS 数据集规模仍相对较小，且在扫描仪、采集协议和病灶模式方面存在差异。这种数据稀缺性和变异性阻碍了稳健的神经影像机器学习模型的开发，对于旨在合成图像的同时保留微小、稀疏病灶的生成模型而言，挑战尤为突出。我们提出 Lesion-DDPM，这是一种用于病灶感知 FLAIR 合成的三维条件扩散框架，将多层级解剖掩膜注入与病灶加权重建损失相结合，在保持全局脑结构的同时强化对病灶体素的关注。我们使用经过整理的 MSLesSeg 数据集子集，将 Lesion-DDPM 与具有代表性的最先进 GAN 模型和扩散模型进行比较，同时评估图像生成指标和下游 3D U-Net 分割表现。在实验中，Lesion-DDPM 在所有方法中取得了最低的病灶区域重建误差。在下游 3D U-Net 病灶分割任务中，仅使用 Lesion-DDPM 生成的扫描图像训练、并在真实 MRI 上评估的模型达到了 0.616 的 Dice 分数，而表现最好的对比合成数据集对应的分数为 0.569。将 Lesion-DDPM 图像加入真实训练集后，Dice 分数进一步提升至 0.685。

CorrFlow：相关椭圆体上的精确且置换等变的流匹配

arXiv:2610.13606v1 公告类型：cross 摘要：相关矩阵同时包含非线性的正定性约束以及行列同步置换对称性。因此，对其进行生成涉及两个不同的问题。首先，无论变量如何排序，每个输出都必须保持在相关椭圆体上。其次，生成器必须学习经验数据在该有效空间内部所占据的领域特定分布。现有的金融图像生成器将这两个问题混为一谈：它们学习有序的矩阵条目，随后再修复无效采样结果。我们提出了 CorrFlow，将二者分离处理。广义 Fisher 变换将满秩相关矩阵双射到无约束坐标，从而无需投影即可保证输出有效。图流匹配网络在这些坐标中学习经验分布，并且具有置换等变性：重新排列变量只会相应地重新排列输出。我们在 $80\times80$ 的台湾股票矩阵上比较了条件式 CorrGAN 的复现版本、图像扩散模型和 CorrFlow，其中每种神经方法使用五个随机种子。CorrFlow 无需修复即可生成 100% 有效的采样结果，并取得了特征切片 Wasserstein 距离 $.712\pm.141$；相比之下，CorrGAN 和图像扩散模型在将无效输出投影回相关空间后，该距离分别为 $4.339\pm.252$ 和 $4.610\pm.191$。与历史查找相比，CorrFlow 也改善了这一距离，但历史查找具有更高的局部覆盖率。这些结果支持在所评估的训练预算下采用这一组合构造；但它们并未证明其优于经过充分调优的图像模型，也未分离各个组件的贡献。

重新思考扩散与解码的作用：用于并行语言生成的 Predictive-State Flows

arXiv:2610.12898v1 公告类型：交叉发布 摘要：在连续扩散语言模型（DLMs）中，扩散先验生成的潜变量通过解码器或词元嵌入矩阵，以近乎确定性的方式映射为词元。与视觉细节可以连续变化的图像生成不同，扩散先验必须在连续空间中捕捉细粒度的离散词元选择，这可能使潜变量的联合分布难以建模。我们认为，生成任务的职责应当重新划分：扩散先验应生成高层次的预测信息，而解码器则负责解决剩余的词元级选择。基于这一观点，我们提出 Predictive-State Flows（PSF），将序列划分为多个块，并为每个块关联一个仅根据前文上下文计算的预测状态。整流流先验生成预测状态序列，随后由块因果解码器并行生成所有块，仅在各块内部进行自回归生成。这一设计将扩散处理的状态从 L 个词元级状态减少为 K << L 个预测状态，并将词元级的顺序计算深度降至 L/K。在 OpenWebText 无条件生成任务中，一个拥有 1.97 亿参数、K=8 的模型，每生成一篇含 1024 个词元的文档仅需 0.31 TFLOPs，在熵为 5.16 时实现了 22.8 的生成困惑度；相比之下，采用 32 个采样步的 ELF-B 的生成困惑度为 24.0，计算量为 7.2 TFLOPs。在对应的实测运行配置下，两者的 MAUVE@512 分别为 0.93 和 0.73。受控比较进一步表明，额外编码了自身块信息的内容状态，其生成困惑度比预测状态高 8–16%；而将采样得到的预测状态轨迹替换为共享均值，会使 MAUVE@512 降低 0.11。这些结果表明，连续 DLMs 的潜变量设计不仅应考虑文本表示的紧凑程度，还应考虑先验与解码器之间的生成职责划分。

基于自然语言的蛋白质结构设计

arXiv:2610.13917v1 公告类型：新投稿 摘要：从头蛋白质设计的一项核心挑战是生成具有所需结构和功能特性的蛋白质。现有设计模型通常要求为每种特性提供专门的输入表示，从而限制了其支持的设计目标。我们开发了 Prompteina，这是首个支持以自然语言指定全新且更广泛设计目标的全原子结构生成模型，类似于文本到图像生成。为训练 Prompteina，我们构建了 Prompteina-Data，这是一个文本与蛋白质结构配对数据集，包含 180 万个结构和超过 5000 万条文本描述，涵盖整体及残基特异性的结构和功能特性。我们通过文本条件预训练，以及将残基与相应文本对齐的微调，对预训练结构生成模型 Proteina-Complexa 进行了适配。Prompteina 在 18 项结构特性中的 15 项上优于基线模型，并且生成的蛋白质更符合其预期功能的描述。在结合蛋白设计中，它在 20 个靶标中的至少 13 个上领先于基线模型，界面置信度较最强基线模型提高了 14\%。引入迭代式计算模拟反馈后，在最具挑战性的靶标上，Prompteina 相较于非迭代生成的性能增益比次优基线模型的相应增益高出 57\%。最后，我们通过实验验证了 Prompteina 针对 GM2A 的设计，此前尚无针对该靶标的从头设计结合蛋白报道。我们鉴定出九种结合蛋白，其中结合能力最强者的表观 $K_{\mathrm{D}}$ 为 107 nM。这些结果表明，自然语言可作为蛋白质结构生成的有效且强大的条件控制接口。

CacheScope：用于免训练扩散加速的精确计算分配

arXiv:2610.12530v1 公告类型：新论文 摘要：免训练特征复用通过在选定步骤中用缓存近似替代完整的去噪计算，加速扩散 Transformer 推理。现有方法主要致力于改进被跳过特征的重建，但我们发现，一旦采用稳定的一阶重建器，增加重建复杂度所带来的收益就十分有限，而精确计算步骤的位置对保真度的影响要大得多。更重要的是，浅层探针变化的绝对量同时受到输入轨迹位移和网络响应强度的影响，因此不能直接反映缓存风险。为消除这种混杂影响，我们用相应的输入位移对探针输出变化进行归一化，得到单位轨迹移动的方向增益。基于这一信号，我们提出了 CacheScope，它可在线识别高风险复用步骤，并选择性地回退到完整的去噪计算，同时保持底层缓存重建器不变。CacheScope 无需额外训练、梯度、辅助网络，也无需针对特定提示词进行离线调度优化。在 FLUX.1-dev、Wan2.1-T2V-1.3B 和 Open-Sora 1.2 上，CacheScope 实现了最高 3.57 倍的加速，并且相较于最强的基于探针的基线方法，将 LPIPS 中位数最多降低了 70.1%。

JulianFlux: Electrodynamic vector retrieval and causal routing for AI

通过图像生成创建符合品牌调性的视觉素材

了解 ChatGPT 图像生成功能如何根据提示词、草图或上传的图像，帮助创建品牌视觉素材和产品效果图。

Omni-Diffusion-Distill：统一多模态扩散大语言模型的少步蒸馏

arXiv:2610.10990v1 发布类型：cross 摘要：统一多模态扩散大语言模型（dLLMs）通过单一架构同时支持图像生成和多模态理解，但其迭代解码需要进行数十至数百次前向传播。现有的少步蒸馏方法大多聚焦于图像生成或文本生成中的一者，因此，如何在保留生成与理解能力的同时，将完全离散的多模态 dLLM 压缩为单一高效的学生模型，仍不明确。我们提出 Omni-Diffusion-Distill，一种统一的两阶段蒸馏框架，在大幅降低统一多模态 dLLM 推理成本的同时，保留强大的生成与理解能力。Omni-Diffusion-Distill 在离散 token 空间中，对图像和文本的生成与理解蒸馏进行统一。第一阶段中，学生模型通过重放缓存的教师模型轨迹进行训练，以跳过部分解码步骤；第二阶段中，学生模型沿自身 rollout 过程中的中间状态进行进一步优化。我们还通过一对碰撞惩罚和熵匹配引导，解决统一蒸馏中的两类性能退化问题：前者减少并行文本解码中的重复，后者防止图像生成中因拟合锐化后的教师分布而导致的熵坍缩。Omni-Diffusion-Distill 在多模态 dLLM 的解码效率与生成、理解性能之间实现了业界领先的权衡，将图像生成的解码步数从 128 步降至 8 步，并将多模态理解的解码步数从 512 步降至 64 步，实现 18.2 倍和 21.2 倍的实际运行时间加速。在这些步数预算下，其在文生图生成任务的 GenEval 和 DPG-Bench 上分别取得 0.828 和 83.0 的分数；在多模态理解任务中，其在 MM-Vet 和 COCO captioning 上的 GPT 评判分数分别达到 20.0 和 57.2（在相同步数下，教师模型为 28.4 的两倍）。

Early Signatures of Memorization in Diffusion Models via Basin Geometry and Cyclic Denoising

arXiv:2610.11670v1 Announce Type: new Abstract: Diffusion models generalize early in training and later reproduce individual training samples. Standard tests detect memorization only once one-shot generation produces near-copies, leaving a released model unaudited until its outputs fail. We show that memorization is encoded in the geometry of the learned energy landscape before it appears in generated samples, a state we call latent memorization. Using score divergence and basin volume, we find that localized basins form around training samples and separate them from held-out samples before the first memorized sample appears, with an onset that follows the same $O(n)$ scaling as the memorization time. We probe these basins with cyclic denoising, which repeatedly applies partial noising and denoising. Under the exact empirical score, we prove that cycling started near an isolated training sample recovers it and returns to it over any finite number of cycles with high probability. In trained models, cycling recovers training images from CelebA and CIFAR-10 checkpoints whose one-shot samples contain no copies, and at a CelebA checkpoint with 0.1% one-shot copies, 500 cycles raise the memorized fraction above 30%. Cycling also reveals degenerate attractors that match no single training image and fade as training proceeds, so residence in a basin does not by itself imply memorization. These findings hold on a Gaussian mixture, CelebA, and CIFAR-10 across optimizers, architectures, noise schedules, and training-set sizes, and extend to off-the-shelf Stable Diffusion v1.4, where the cycled conditional-unconditional divergence gap separates memorized from non-memorized prompts with an AUC of 0.944 and a TPR of 0.866 at 1% FPR. More broadly, what a diffusion model has memorized is a property of the geometry and stability of its learned distribution, and assessing it requires examining this structure rather than generated outputs alone.

Deflating the Hessian: Rank-4 W4A4 Quantization for Multimodal Diffusion Transformers

arXiv:2610.11315v1 Announce Type: new Abstract: In diffusion transformers, low-rank branches can mitigate 4-bit weight--activation (W4A4) post-training quantization (PTQ) loss by decomposing each weight into a low-bit residual and a high-precision low-rank component. Existing low-rank PTQ approaches, however, either optimize low-rank compensation and residual quantization separately, often requiring higher ranks, or rely on second-order weight updates without explicitly modeling activation quantization error, which becomes particularly pronounced under 4-bit quantization. To address these limitations, we present \method{}, a unified framework modeling low-rank-assisted W4A4 PTQ as a coupled calibration problem and deriving optimization-based solvers from the joint objective. Eliminating the output-side low-rank factor yields a \emph{deflated Hessian} that discounts residual errors already captured by the low-rank component, while an activation-noise surrogate is incorporated to suppress activation quantization error. Across five diffusion backbones, rank-4 \method{} consistently outperforms rank-4 SVDQuant in PSNR and LPIPS. It further surpasses rank-32 SVDQuant on SANA-1.6B, FLUX.1-schnell, and FLUX.1-dev with an $8\times$ smaller rank and up to $6.25\times$ faster quantization. Furthermore, on the Qwen3-8B LLM, rank-4 \method{} improves MMLU accuracy from 61.50\% to 68.17\% over rank-32 SVDQuant. Overall, \method{} achieves better W4A4 performance with substantially lower rank and quantization cost.

Residualized Temporal Sparse Autoencoders for Interpreting Diffusion Models

arXiv:2605.27813v2 Announce Type: replace-cross Abstract: Text-to-image diffusion models generate images by iterative denoising, so their internal layers produce trajectories of activations rather than single static representations. Sparse autoencoders (SAEs) have recently been used to decompose diffusion activations into interpretable features, but most approaches analyze individual timesteps or condition on time rather than learning from full trajectories. Training one SAE on whole trajectories would make each feature a single trajectory across timesteps, but adjacent activations are largely linearly predictable from one another, so such an SAE spends its latents on content carried forward from step to step. We introduce residualized temporal SAEs (ReSAE), which fit linear predictors between neighboring timesteps and represent each trajectory by its initial activation and the residuals these dynamics leave unexplained. Training an SAE on this representation is equivalent to training it on raw trajectories under a metric induced by the linear dynamics, and it encourages latents to capture structure beyond what is linearly predictable. Each latent's decoder direction maps back to activation space as a feature trajectory over denoising time. Across Stable Diffusion~1.5 and a Diffusion Transformer, ReSAE features span the whole trajectory while pinpointing when changes enter it, making ReSAE a natural tool for studying how a diffusion model generates an image over time.

面向不确定性感知光谱图像仿真的变分潜在空间框架

arXiv:2603.21911v3 公告类型：替换并交叉收录 摘要：合成光谱图像生成对于遥感仿真和任务设计至关重要，但基于物理的辐射传输模型（RTMs）的计算成本仍然很高。现有基于学习的仿真器降低了这一成本，但大多是确定性的参数到光谱回归器，空间建模能力有限，提供的不确定性信息也不足。我们将光谱图像仿真表述为一个以参数为条件的潜变量问题，并提出一个基于变分自编码器（VAE）的框架，融合了非线性光谱图像表征、快速推理和逐像素不确定性估计。该框架通过 VAE 预训练和潜在空间映射两个步骤，分别在光谱层面和空间—光谱层面实现。我们在 PROSAIL 模拟的高光谱植被数据立方体（211 个波段）和真实 Sentinel-3 OLCI 多光谱海洋水色影像（21 个波段）上对其进行评估，并与经典回归仿真器和深度 CNN 基线模型进行比较。结果表明，没有哪一种架构在所有情况下都是最优的：像素到像素模型在受控高光谱模拟中表现最佳，而全卷积 VAE 在含有缺失或受污染像素的带噪声真实观测数据上更具鲁棒性。基于 VAE 的仿真器在大规模生成中也实现了高吞吐量。在 Sentinel-3 数据上，空间—光谱 VAE 提供的预测区间比逐像素神经网络仿真器和经典仿真器更贴近经验误差，但绝对校准仍不充分。一项基于查找表的反演实验进一步表明，仅有重建保真度并不能确保叶面积指数和叶绿素反演的可靠性。因此，应在具有代表性的遥感最终应用场景中评估仿真器，而不能仅依据重建指标。

UltraText Bench：用于评估图像生成中视觉文本渲染能力的综合性双语基准

密集视觉文本要求图像生成器在多个区域中复现长字符串，同时确保位置正确且清晰可读。随着短字符串渲染能力的提升，评估必须检验模型在更具挑战性的场景中的持续表现。我们提出 UltraText Bench，这是一个仅通过提示词生成密集视觉文本的双语基准。它包含 432 条提示词，涵盖 24 类真实世界场景和三个难度级别，英语和中文各占一半。每条经过人工审核的提示词都为四至十二个文本区域提供确切的字符串，并配有关于文本内容、位置和视觉属性的结构化参考信息。我们使用视觉语言模型 Q-Judger，根据完整参考信息评估每张图像，报告文本保真度、文本清晰度、空间质量和场景质量。在 24 种模型配置中，这些维度揭示了不同的优势：在所报告的设置下，Z-Image-Turbo 的清晰度得分比 Z-Image-Base 高 3.81 分，但保真度得分低 14.76 分。性能也随任务负荷而变化；Qwen-Image-2512 的英语综合得分从 L1 的 86.50 降至 L3 的 42.86。十名参与者对自动评分进行了人工评估。代码仓库：https://github.com/LINs-lab/UltraText_Bench。

通过 Residual Latent Action 学习基于视觉特征的世界模型

arXiv:2605.07079v2 公告类型：replace-cross 摘要：世界模型根据观测和动作预测未来的状态转移。现有研究主要只关注图像生成。相比之下，基于视觉特征的世界模型预测未来的视觉特征，而非原始视频像素，提供了一种更高效且更不易产生幻觉的有前景的替代方案。然而，当前基于特征的方法依赖直接回归，这会在复杂交互中导致预测模糊或坍缩，而在高维特征空间中进行生成式建模仍然具有挑战性。在本研究中，我们发现，可以轻松地从 DINO 残差中学习一种新型潜在动作表示，我们将其称为 Residual Latent Action（RLA）。我们还表明，RLA 具有预测能力和泛化能力，并编码了时间演进信息。基于 RLA，我们提出了 RLA World Model（RLA-WM），通过流匹配预测 RLA 值。在仿真和真实世界数据集上，RLA-WM 的表现均优于最先进的基于特征的世界模型和视频扩散世界模型，同时速度比视频扩散快数个数量级。此外，我们开发了两种利用 RLA-WM 改进策略学习的机器人学习技术。第一种是采用 RLA 的极简世界动作模型，它从无动作标注的视频中学习，并提升了 VLA 在 LIBERO 和真实机器人上的表现。第二种是一个视觉强化学习框架，其训练完全在仅从离线视频中学习得到的世界模型内部进行，使用与视频对齐的奖励，无需任何在线交互。项目页面：https://mlzxy.github.io/rla-wm

一种考虑通量足迹的跨多样化生态系统高分辨率碳通量预测方法

arXiv:2512.01917v2 公告类型：替换 摘要：涡度协方差（EC）通量塔提供 $CO_2$ 通量的原位测量值，并为基于卫星产品构建的预测性“尺度上推”模型提供地面真值数据。然而，如今许多卫星能够分辨的空间尺度小于 EC 通量塔的通量足迹范围。我们从理论上证明，在异质性景观中，使用高分辨率数据训练的尺度上推模型必须考虑 EC 通量塔的通量足迹，才能避免像元级预测出现偏差。为解决这一问题，我们提出 Footprint-Aware Regression（FAR），这是一个同时预测空间通量足迹和像元级 $CO_2$ 通量估计值的深度学习框架，并证明在训练数据充足的情况下，该框架能够产生无偏的像元级预测。我们在 AMERI-FAR25 数据集上展示了 FAR 的应用，该数据集将 205 个站点年的通量塔数据与对应的 Landsat 影像相结合；结果表明，FAR 优于未考虑通量足迹的模型。在由留出站点构成的数据集上，相较于表现最佳的固定通量足迹基线模型，FAR 将半小时尺度的 $R^2$ 从约 0.575 提高至 0.634，并将 RMSE 降低了约 7%。与分辨率较粗的 990 米基线模型相比，FAR 在月平均和年平均尺度上的提升更大。站点级分析表明，这些提升覆盖了多种生态系统类型。无论通量足迹是通过联合学习得到，还是使用成熟的通量足迹模型独立估计，均可观察到这些性能提升，尽管不同方法得到的通量足迹大小存在显著差异。在 Western Cascades 的区域对比中，FAR 生成的通量估计值分布与现有高分辨率过程模型的结果相当。

FOSLS-deRhaNN: native de Rham neural classes for H(div) and H(curl) with applications to first-order system least-squares neural network methods for partial differential equations

arXiv:2610.08016v1 Announce Type: cross Abstract: We construct neural approximation classes native to the graph spaces H(div) and H(curl), in two and three dimensions and, for H(div), in any dimension. Every realization lies in the space for all parameter values, and with kinked potentials, such as ReLU networks, the admissible jumps appear at finite width. The classes are images of scalar and componentwise networks under fixed operators of the de Rham complex, and do not involve a mesh or finite element emulation. For H(div) in R^n two native classes are given on an equal footing, with a skew-symmetric potential $A$: $\mathrm{Div}\,A+R_nq+\mathbf{h}$, with the divergence $q$ as an explicit unknown, and $\mathrm{Div}\,A+\mathbf{z}$ with an $H^1$ field $\mathbf{z}$; for H(curl) the analogous classes are $\mathrm{grad}\,\phi+Sr+\mathbf{h}$ in two dimensions and $\mathrm{grad}\,\phi+\mathbf{z}$ in two and three dimensions. In all of them every interface jump of the field is carried by the potential term, $\mathrm{Div}\,A$ or $\mathrm{grad}\,\phi$, while the remaining part has no interface jump (it is an $H^1$ field in the regular-decomposition classes); the classes with $\mathbf{z}$ are the componentwise approach enriched by this term. Known or learned interface geometry enters the potential through factors with trainable amplitudes, and the remaining part if the divergence jumps. The classes lead to the FOSLS-deRhaNN method, first-order system least squares with de Rham neural networks, whose loss is the least-squares functional posed in the natural spaces of the weak formulation; for elliptic equations this includes $H^{-1}$ right-hand sides and $H^{1/2}$ Dirichlet data. Elliptic equations with discontinuous coefficients and curl-curl problems are treated as instances, with the functional equivalent to the error; linear transport with discontinuous solutions and conservation laws with shocks use the same flux classes.

RefRoute：通过紧凑残差条件化与空间路由，将条件化成本与参考图像解耦

arXiv:2610.07720v1 公告类型：交叉收录 摘要：多参考图像生成需要在将多个主体组合成连贯场景的同时，保留各主体的外观。然而，现有扩散 Transformer 通常将参考图像编码为密集的视觉 token 网格，并通过全局注意力进行联合处理，导致条件化成本随着参考图像数量和分辨率的增加而不断上升。我们提出 RefRoute，该框架通过两种互补机制，同时降低参考表示成本与注意力开销。紧凑残差条件化将低分辨率潜在 token 与从全分辨率像素中提取的轻量级残差特征相结合，在保留细粒度外观线索的同时减少参考 token 数量。条件路由和注意力路由将参考 token 与其指定的目标区域对齐，并限制不同参考图像之间的交互，同时允许跨越区域边界选择性地访问参考信息，以实现整体场景融合。我们进一步推出了用于训练多参考图像生成模型的 RefRoute-Data，以及涵盖人物、物体及混合构图、每个样例包含 10–17 张参考图像的基准 ManyRef100。经过多参考图像微调后，RefRoute 在 ManyRef100 上的总体 Weighted-Ref-VIEScore 达到 36.06，而 FLUX.2-Klein-9B 为 8.88。单独的推理成本评估表明，随着参考图像数量增加，RefRoute 的延迟增长明显更慢：在使用 16 张参考图像时，我们的 50 步和 4 步配置相较于各自对应的 FLUX 基线，分别实现了 $18.3\times$ 和 $14.2\times$ 的加速。这些结果表明，紧凑参考表示与空间路由注意力是实现可扩展多参考图像生成的有效方法。

关于 VAE 潜空间中的色彩对齐及其应用

arXiv:2610.07072v1 发布类型：cross 摘要：变分自编码器（VAEs）是现代文本到图像模型的重要组成部分，这些模型在其潜空间中生成图像。众所周知，VAEs 能够解耦数据中的主要变化因素，而在自然图像中，色彩是这些因素中结构最明显的因素之一：对其进行去相关处理，可以得到一个亮度轴和两个对手色轴。因此，可以预期色彩会作为一个独立因素出现在 VAE 潜空间中。然而，这些潜空间如何表示色彩，目前仍鲜有研究。在本文中，我们展示了文本到图像模型中的 VAEs 共享一个与亮度和对手色对齐的色彩子空间。通过对编码器进行线性近似并实施有针对性的潜变量引导，我们在从 SD1.5 到 FLUX.2 和 Z-Image 的广泛 VAEs 中，一致地发现了这一子空间。在此表征的基础上，我们提出了三种应用：ColorTuning，在 GenColorBench 的细粒度 CSS3/X11 色彩系统上实现了精确数值色彩生成的当前最佳性能；饱和度控制，用于调整全局色彩强度；以及色彩迁移，用于改变色板以匹配参考图像。代码和模型已公开发布于 https://julian075.github.io/Color_Subspace/

分类器无关引导流的全局传输耦合

arXiv:2610.07555v1 公告类型：新 摘要：已有研究表明，最优传输耦合能够降低无条件流模型的训练方差，但其在条件生成中的作用仍不明确。一种自然的方法是为每个条件分别构建耦合，但对于现代图像基础模型中常见的大规模或连续条件空间而言，这种方法并不切实际。我们提出 Global Transport（GT），一种不使用类别标签计算的全局、类别无关最优传输耦合。GT 可以将不同条件与源噪声中的不同区域关联起来，因此在没有引导的情况下会导致性能下降。然而，与 classifier-free guidance（CFG）结合后，GT 在不同领域、模型规模和采样预算下都能持续提升生成效果。这一反转表明，对于条件流，应当在推理时所使用的引导流下，从经验和理论两方面对耦合进行评估，而不是仅考察无引导生成。我们在离散类别条件和连续文本条件的图像生成任务中，跨不同模型规模评估了 GT，并研究了耦合选择如何改变引导轨迹。这些结果表明，在引导流设置下设计耦合，是一种简单的训练时优化方向，无需修改现有架构、采样器或引导机制即可提升性能。

在有限交互下学习澄清不充分的意图

AI 助手会收到一些请求，其中缺少实现良好结果所需的信息，例如用户的偏好或目标。此时，它们要么进行猜测，要么在继续操作前询问更多信息。我们将这一问题重新概念化为信息价值问题：助手应获取那些因缺失而导致用户效用产生最大可避免损失的信息。这一点通常无法事先得知；相反，助手必须对其进行预测，从而在有限的用户交互次数下实现最优分配。我们在图像生成场景中具体化了这一问题，并利用多轮模拟用户构建了一个强化学习框架，以在不确定性下最大化效用恢复。在一项预注册研究中，76 名人类参与者进行了 456 次交互会话；该方法帮助用户以显著更少的问题、更短的总交互时间和更低的成本，更好地匹配参考图像。这表明，可以采用一种简单且可扩展的框架来训练语言模型助手，通过提出信息量更高的问题，更好地消除用户意图的歧义。

VTBench：评估自回归图像生成的视觉分词器

arXiv:2505.13439v2 Announce Type: replace-cross 摘要：自回归（AR）模型近期在图像生成方面展现出强大性能，其中一个关键组件是视觉分词器（VT），它将连续的像素输入映射为离散的 token 序列。VT 的质量在很大程度上决定了 AR 模型性能的上限。然而，当前的离散 VT 明显落后于连续变分自编码器（VAEs），导致图像重建质量下降，以及细节和文本保留能力较差。现有基准主要关注端到端生成质量，未能将 VT 的性能单独分离出来进行评估。为弥补这一空缺，我们推出了 VTBench，这是一个全面的基准，围绕三项核心任务系统评估 VT：图像重建、细节保留和文本保留，并涵盖多种评估场景。我们利用一组指标系统评估了当前最先进的 VT，以衡量重建图像的质量。研究结果表明，与离散 VT 相比，连续 VAE 能够生成更优越的视觉表征，尤其是在保留空间结构和语义细节方面。相比之下，离散 VT 生成的退化表征通常会导致重建失真、细粒度纹理丢失，以及无法保留文本和物体的完整性。此外，我们在 GPT-4o 图像生成上开展了实验，并讨论了其潜在的 AR 特性，为理解视觉分词的作用提供了新的见解。我们公开发布了基准和代码库，以支持进一步研究，并呼吁社区开发强大且通用的开源 VT。

将预训练 LLM 重新用作高保真连续文本自编码器

arXiv:2609.27248v2 公告类型：替换 摘要：下一个词元预测使自回归语言模型能够生成高度流畅的文本，但它只能通过序列分解间接表示全局结构。相比之下，高保真自编码器已成为图像生成中的标准基础模块，使生成模型能够在连续潜空间上运行；而文本尚缺乏同样忠实的连续表示。我们提出 LLMAE，通过将中间层的激活暴露为固定长度的潜在瓶颈，将预训练的仅解码器语言模型重新用作连续文本自编码器。LLMAE 通过结构化注意力掩码和 LoRA 适配实现了这一接口，同时利用原始 LLM 的生成先验。在两个骨干模型（270M Gemma 3 和 0.5B Qwen2.5）上，LLMAE 能够高准确度地重建最长 1024 个词元的序列，并逐字复现多达 97% 的文档。我们通过训练一个轻量级扩散模型展示了其下游应用价值：该模型能够直接在冻结的 LLMAE 潜空间中生成详细的图像描述。通过将文本映射到固定长度的连续潜空间，我们的方法为下游适配提供了有效基础，同时受益于原始 LLM 的流畅性。代码和模型可在 https://github.com/arkanath/LLMAE 获取。

AREX：用于 Flow Matching 少步采样的仿射-残差指数积分器

arXiv:2610.03483v1 公告类型：cross 摘要：我们提出了 AREX，这是一种面向预训练 Flow Matching 模型的无需训练采样器，利用目标均值和协方差捕捉采样动力学中可进行解析处理的部分。我们证明，矩匹配高斯目标的速度场是边际速度场的 $L^2$ 最优仿射近似。这一结果启发我们将学习到的动力学分解为：由前两个目标矩决定、贯穿整个采样路径的仿射分量，以及神经残差项。AREX 保留仿射分量，并使用显式矩阵值传播子对其进行积分。这样一来，我们只需对残差项进行积分。这不同于标量指数积分器，后者只能解析处理各向同性线性动力学。在图像生成和文本到图像生成任务中，AREX 始终能够在少步采样场景下提升样本保真度，且无需重新训练底层模型。

ENCORE：用于扩散生成的基于副本交换的精确非平衡控制

arXiv:2610.02538v1 公告类型：交叉收录 摘要：推理时控制无需重新训练，即可引导预训练生成模型趋向目标分布。我们研究倾斜目标分布 $\pi_0\propto G_0\,p_0$，其中 $p_0$ 是采样器的输出分布，$G_0$ 是可求值的重加权函数。现有方法依赖于采用序贯蒙特卡洛（SMC）的序贯退火，或采用副本交换（RE）的并行退火。序贯控制是精确的，但需要大量粒子；而目前尚无精确的并行控制方法：现有 RE 校正对难以求解的时间反演进行近似，因此存在偏差。我们提出基于副本交换的精确非平衡控制（ENCORE），这是首个精确的并行控制方法。每个副本都存储其生成轨迹，因此向上移动只需截断轨迹，完全无需模拟难以求解的时间反演。我们证明了目标分布的不变性，并表明所得动力学等价于以精确时间反演作为前向提议的非平衡副本交换动力学。在正则性条件下，我们的扩散分析表明，若不加引导，序贯控制和并行控制都会随着时间离散化的细化而变得不稳定；而带引导的提议则保持稳定，并提供用于调整调度方案和计算预算的诊断指标。在合成目标分布、生物分子的 Boltzmann 采样和图像生成任务中，ENCORE 在准确性和多样性方面均具有竞争力，对采样器扰动保持稳健，并且适用于现有 RE 校正无法应用的蒸馏采样器。

面向流映射蒸馏的几何感知时间重参数化

arXiv:2610.02427v1 公告类型：新 摘要：流映射蒸馏通过学习预训练生成 ODE 的有限时间转移，实现一步和少步生成。我们研究改变教师模型的时间参数化是否能使这些转移更易于学习。基于这样一个假设：具有较大法向加速度的轨迹片段更难蒸馏，我们提出了一种几何感知的时间重参数化方法，在保持教师模型几何路径和终端分布不变的同时，为这些区域分配更多学生时间。我们在适当假设下推导出一种共享时钟，使总体法向加速度统计量达到均衡，并利用从教师模型轨迹中获得的稳健正则化估计构造出一种实用近似。我们将这一时钟引入拉格朗日流映射蒸馏，并使用变换后的时间坐标对学生模型进行条件控制。该时钟在蒸馏前仅需估计一次，既不需要重新训练教师模型，也不需要增加学生模型参数或推理时的网络评估次数。在合成数据、CIFAR-10 和 CelebA-64 上的实验表明，在推理预算相同的情况下，与恒等时间蒸馏相比，该方法能够提升样本质量，包括改善一步和两步图像生成效果。在一步生成中，由于无法调整任何中间采样时间，所取得的性能提升进一步凸显了蒸馏过程中时间重参数化的优势。

Rank-Aware Speculative Sampling for Diffusion Draft Trees

arXiv:2610.02251v1 Announce Type: new Abstract: Speculative sampling accelerates diffusion generation by verifying inexpensive draft states in parallel while preserving the target law. Recent tree-based methods allocate the parallel compute budget more effectively than single-chain drafts, as demonstrated by Diffusion Greedy Rejection Sampling (D-GRS). D-GRS generates $K$ conditionally independent candidates per node, and sequentially tests them in their generation order. Yet the sampled candidates admit an informative ranking without additional target-model evaluations. To exploit this, we introduce Rank-Aware Speculative Sampling (RASS), a verification rule for speculative draft trees based on rank-aware list coupling. RASS orders draft candidates along the proposal-target mean displacement and samples a rank with weights optimized to minimize total variation between the selected-proposal and target laws. Finally, the selected candidate is maximally coupled with the target, with residual correction ensuring exact sampling for any choice of rank weights. We evaluate RASS on a Gaussian-mixture target, unconditional pixel-space generation on FFHQ, conditional generation on CIFAR-10, and latent diffusion with Stable Diffusion 3.5 using COCO2014 prompts. Measured by the ratio of standard to speculative sampling's target-model evaluation counts, RASS improves on D-GRS across the evaluated settings, with gains reaching approximately 20% on CIFAR-10 at matched compute budgets.

重新思考少步扩散 Transformer 中缓存什么：求解器感知的目标选择

扩散 Transformer（DiTs）能够生成高质量的图像和视频，但生成每个样本都需要多次高成本的 DiT 前向传播。加速 DiT 采样的两种常见方法是步骤蒸馏和缓存：前者减少采样步数，后者通过复用较早步骤计算得到的张量来跳过部分 DiT 计算。大多数缓存方法会预先决定要复用哪个张量。经过蒸馏后，相邻采样步骤之间的间隔更大。在这一更大的间隔内复用张量会引入更多误差，因此选择缓存对象变得尤为重要。为此，我们提出 AutoTarget，这是一种针对给定模型、求解器和复用计划选择缓存张量的方法。AutoTarget 使用少量不进行缓存复用的运行，测量复用每个候选张量所造成的误差，然后选择误差最低的候选张量。我们还分析了某个复用步骤中的误差如何影响最终样本。对于 Euler 采样，我们确定了能够产生相同轨迹的缓存目标，并说明为什么存储的求解器更新量可能无法实现这一点。在蒸馏图像和视频 DiT 上的实验表明，最佳缓存目标会随模型、图像分辨率和求解器而变化。AutoTarget 减少了 DiT 计算次数和保留的缓存存储量，同时生成质量仍接近相应的不使用缓存的运行结果。在测试的 PixArt-LCM 和 FLUX.1-schnell 设置上，其校准排序与留出缓存运行所得的排序一致。为帮助他人复现该方法，我们在 GitHub 上提供了其核心实现：https://github.com/wali1024-offical/AutoTarget。

2026 in LLMs (so far)

On Friday I gave the closing keynote at the WeAreDevelopers World Congress North America in San Jose. I tied together the key trends from the past year into a chronological exploration of everything that happened in 2026. The video is on YouTube ; here are my annotated slides and notes to accompany the talk. And as an annotated presentation : 2026 in LLMs (so far) Simon Willison WeAreDevelopers World Congress North America, 25th September 2026 # I'm going to give a lightning tour of everything that has happened so far in 2026. The year isn't over yet! November 2025 # For me, 2026 started a couple of months earlier in November 2025. The November 2025 inflection point Claude Opus 4.5 GPT-5.1 # November saw the release of two important models: Claude Opus 4.5 and GPT-5.1. As is usually the case with new models, these were incremental improvements on the models that came before them. But every now and then when a model improves, it crosses an invisible line where something that didn't really work starts working. In this case, the thing that started working was their coding agents. Claude Code had been around since February 2025; Codex was a little younger. These two new models, when paired with their respective coding agent harnesses, improved from "often make mistakes" to "reliable enough to use on a day-to-day basis". "Generate an SVG of a pelican riding a bicycle". The Claude Opus 4.5 one has a very weird shaped frame and the pelican looks like a duck. The GPT-5.1 has a slightly better but still broken bicycle frame and a slightly better pelican beak, but both are pretty terrible. # For a couple of years now I've been evaluating new models by asking them to "Generate an SVG of a pelican riding a bicycle". It's probably the world's stupidest benchmark - there's only so much you can learn from it. But it's still a challenge for models, because drawing pelicans is difficult, drawing bicycles is difficult, and pelicans can't ride bicycles in the first place. Here's the state of the art for November. Claude still couldn't really draw a bicycle! The GPT-5.1 bicycle frame is pretty crap too. November 24th 2025 - the first commit to steipete/Warelay. A GitHub commit adding an MIT license file. # Also in November, we had the first commit to an obscure GitHub repository called "Warelay". We'll come back to this repository shortly. January # And then there were the December holidays, and individual developers took some time off and many started tinkering with these new coding agent model combinations... and it began to dawn on us quite how much they could do that they couldn't do before. Come January, a lot of us were quite excited to start putting this stuff into action. New year’s resolution for 2026 Every previous year: Take on less new projects, focus on the most important things in my existing projects # Every year I set myself a New Year's resolution, and for as long as I can remember it's been the same thing: stay focused. Take on less new projects. Try to get things done in the projects I already have. 2026: Be more ambitious. Take on as many new projects as I want. # This year I decided that since that had never worked before, I'd go the other way. We've got coding agents now, let's see what they can do. I'm going to take on as many new projects as I like! (You can ask me at the end of the year if this turned out to be a good idea or not. I have a lot of plates spinning right now.) "Be more ambitious" has been something of a theme for the year, because the only way to find the limits of this technology is to keep on pushing them until they don't work. Predictions for 2026 It will become undeniable that LLMs write good code We're finally going to solve sandboxing A “Challenger disaster” for coding agent security Kakapo parrots will have an outstanding breeding season (only 236 in the world!) ... the Pope will weigh in on LLMs and their economic impact on the world # I also went on the Oxide and friends podcast with Bryan Cantrill and Adam Leventhal to share predictions for the next year (and three and six years). With hindsight, my LLM predictions were pretty unambitious. I said "it will become undeniable that LLMs write good code" - I think we're there now. I predicted we would finally solve sandboxing. I counted and around 40 of the 277 sessions at this conference touched on sandboxing or agent security in some way, so we're at least putting a lot of effort into that! I predicted "a Challenger disaster" for coding agent security. There's certainly been a whole lot of noise around agent security this year, though the exact disaster I predicted (with coding agents being hijacked and causing real-world economic damage) hasn't really played out. We threw in a joke prediction that the Pope would weigh in on the economic impact of LLMs. A photograph of a beautiful green New Zealand parrot. Photo credit Kimberley Collins. # I also predicted that New Zealand's Kākāpō parrots would have an outstanding breeding season this year. These are flightless nocturnal parrots. They're kind of dumpy looking, I think they're beautiful, and there were only 236 of these parrots in the world at the start of the year. Kākāpō only breed when the Rimu trees have a big fruiting season, and that hasn't happened in four years... but this year the Rimu fruit were looking excellent. Photo by Kimberley Collins . Deep Blue Coined by Adam Leventhal and Bryan Cantrill That feeling of AI induced ennui where software engineers get listless because the AI can do anything # Also on that podcast, we coined a term (full credit to Adam) for "that feeling of AI induced ennui where software engineers get listless because the AI can do anything". We called it Deep Blue . This has been a major theme throughout the year, and was touched on by several speakers at this conference. As a software engineer, I've never had a year of my career where everything has changed so quickly and so dramatically. A lot of what I've been doing this year is trying to come to terms with that and what that means for my own profession. AI mania Screenshots of the micro-javascript and pwasm GitHub README files. # Also in January, I suffered from what I'm calling AI mania . This is not the same thing as AI psychosis . With AI mania, any time your agent isn't building something for you feels like wasted time. You're losing sleep because you could be staying up later getting your agents to do stuff. My AI mania presented itself in some ridiculously over-ambitious projects. I built a JavaScript interpreter entirely in Python , vibe-ported from MicroQuickJS by Fabrice Bellard. Then I built a WebAssembly runtime in Python as well . These projects were quite useful, in that they sort of cured me of my AI mania... because after I built these things, I got to look at them and ask "does the world need a slow, buggy, half-baked Python JavaScript interpreter?" I don't think the world does. micro-javascript playground 3 Execute JavaScript code in a sandboxed micro-javascript environment powered by Pyodide A web UI with some JavaScript code, and a "Run Code" button, and an output panel. # I did get this out of it: https://simonw.github.io/micro-javascript/playground.html Previous screenshot, with this text overlaid: JavaScript running in Python running in Pyodide running in WebAssembly running in JavaScript # This page runs my JavaScript interpreter built in Python, running in Python using Pyodide , which is Python compiled to WebAssembly, running in JavaScript, running in a browser. It's a beautiful stack of horrors. I've been having a lot of fun with WebAssembly this year. Warelay → CLAWDIS → CLAWDBOT → Clawdbot → Moltbot →🦞 OpenClaw Screenshot of the dates that these changes happened. # By the end of January, that repository we saw start in November had renamed itself, first to CLAWDIS, then CLAWDBOT, then Moltbot, and finally to OpenClaw. Same screenshot, an overlay reads: 8,330 commits in just under two months (it’s at 100,141 today) # At this point OpenClaw had 8,300 commits, less than two months after the project had started. I looked today and it's over 100,000 commits now! This is the most vibe-coded piece of software in existence. (Here's how I generated that list of name changes .) Generic term: Claw # This kicked off the OpenClaw revolution. It effectively defined a new category of software. There's a generic term for this which I really enjoy. We call software like this a "Claw". There's OpenClaw, NanoClaw , IronClaw , PicoClaw ... Today they're being rebranded as "personal agents" or "general agents", but I still like to think of them as Claws. Photo of a Mac mini An aquarium for your Claw # The Apple stores in the Bay Area sold out of Mac Minis because so many people were buying Mac Minis to run OpenClaw! Drew Breunig said that this is because your OpenClaw is a digital pet, and you buy a Mac mini as an aquarium to keep your claw in, which is kind of delightful. Screenshot of Moltbook - a social network for AI agents # Also in January, we had this website. This was MoltBook , a social network for AI agents, where the idea was that you send your Claw to go and talk to all of the other Claws, because what could possibly go wrong if you did that? The website launched on Thursday. It blew up on Friday . It was profiled by the New York Times on Monday . And by Tuesday, everyone had forgotten it existed as it drowned in a deluge of slop and spam. Facebook/Meta bought it a month later . February # In February, a company called StrongDM described what they called their Software Factory. StrongDM’s Dark Factory Justin McCarthy, Jay Taylor, Navan Chauhan Software Factories and the Agentic Moment # They wrote about this in Software Factories and the Agentic Moment . I posted my own notes at the time, having seen their demo in person back in October. Dan Shapiro called this approach the Dark Factory , after the idea that if your factory is sufficiently automated you can turn the lights out, because you don't even need to see what's going on. StrongDM presented two rules for software development that they'd been following since July last year. “Rule 1: Code must not be written by humans” # The first was code must not be written by humans . Any code that you write has to have been routed through a coding agent. This sounded radical in February, but I imagine there are a lot of people in this room who are pretty much living that today. “Rule 2: Code must not be reviewed by humans” (!) # Rule number two was code must not be reviewed by humans . You're not allowed to read the code! This continued to be a huge topic for much of this year. Many of the sessions at this event have been about code review and how you can get away with this. What I found interesting about StrongDM is that they were living six months ahead of the rest of us, and they'd been exploring what it means to build software, not read the code, but still be confident that the software is of high quality. What can you do with these agents to help verify their work? StrongDM are a security company, and they had people with decades of experience on this project. They were very much exploring the edges of what's possible and responsible to do with this stuff. Headline on New Zealand's Department of Conservation website: First kakapo chick in four years hatches on Valentine's Day. It's a grey fluffy ball. # Also in February: First kākāpō chick in four years hatches on Valentine's Day . Breeding season is off to a good start! 19th February 2026 Gemini 3.1 Pro A surprisingly good illustration of a pelican riding a bicycle. # Also in February... Google released Gemini 3.1 Pro . That's a pretty great pelican riding a bicycle! It's got the chain in the right place, it's got feet on both sides. There's a little fish in the basket. @JeffDean on Twitter - a video comparing Gemini 3 Pro and Gemini 3.1 Pro. # And then Google's Jeff Dean tweeted a video comparing Gemini 3 Pro and Gemini 3.1 Pro that featured an animated pelican riding a bicycle, a frog on a penny-farthing, a giraffe driving a tiny car, an ostrich on roller skates, a turtle kickflipping a skateboard, and a dachshund driving a stretch limousine. This was frustrating, because my protection for the pelican riding the bicycle test was always "if they draw a perfect pelican on a bicycle, I'll ask for some other animal on something else." Google trained for all forms of animals on all forms of transport! They've defeated my benchmark at this point. Three headlines: Meta Makes AI Adoption a Formal Part of Performance Reviews Not just engineers writing code, Microsoft wants almost every employee to use Al Dara Khosrowshahi: 90% of Uber engineers now use AI in daily workflows # The other thing that started in February was Tokenmaxxing . We had headlines about Meta making AI adoption a formal part of performance reviews, and Microsoft wanting every employee to use AI, and Uber boasting that 90% of their engineers were using AI workflows. More headlines: Meta Plans to Crack Down on Employee Token Use: Information Microsoft Tells Engineers: Tokenmaxxing is not what we are optimizing for Uber caps employee AI spending after blowing through budget in four months # Then a few months later we have Meta cracking down on token use, Microsoft saying tokenmaxxing is "not what we are optimizing for", and Uber capping employee AI spending. So tokenmaxxing went straight up and then straight back down again - because it turns out the agents are expensive . Last year it was difficult to spend more than $50 on AI tokens, because we didn't have anything interesting to do with them. Then agents blew up, and now you can actually spend $1,000 in a day doing real work. This is also the reason that Anthropic's valuation skyrocketed to maybe a trillion dollars. AI appears to have hit product market fit in 2026, primarily through coding agents. March # In March, we hit peak OpenClaw. March: peak OpenClaw Photos of people in china queuing up to install OpenClaw, with big fluffy lobsters. # These photographs are from China, where companies hosted OpenClaw install parties which saw non-tech-nerds queueing up around the block for help getting Claws installed on their personal devices. I think this proved real market demand for this class of Claws, or personal AI agents. It turns out regular people really do want a weird little AI agent that can do useful things on their behalf. A Claw is really just a coding agent wearing a less threatening hat. Under the hood they work much the same way - writing and then executing code on your computer to get stuff done. The race was on to be the first to build a safe Claw - a Claw you could give to regular human beings where they wouldn't instantly shoot themselves in the foot. Meta's Muse came out three weeks ago and is currently at the top of the free charts on the iPhone App Store. It appears to be taking off with consumers. I'm not yet convinced you can't shoot yourself in the foot with Muse, but I guess we'll find out for sure pretty soon. Photos from How the OpenClaw Frenzy Is Testing China’s AI Commitment (March 29th) and The Enthusiasm and Anxiety Behind China’s OpenClaw Craze (April 8th, 2026). April # In April, we had a model release where the model wasn't actually released. Simon Willison’s Weblog - screenshot of the post "Anthropic’s Project Glasswing—restricting Claude Mythos to security researchers—sounds necessary to me" from April 7th 2026 # Anthropic announced their new Claude Mythos model, and then said it was too dangerous to release beyond a trusted group of security researchers. Mythos was really, really good at hacking things. The "it's too dangerous" marketing ploy has been played by AI companies dating all the way back to GPT-2 . Anytime an AI company says we've built something that's "too dangerous", it's natural to be a bit skeptical. I found the Mythos claims credible, because I'd seen how good coding agents had got at finding regular bugs. I wrote about that in Anthropic’s Project Glasswing—restricting Claude Mythos to security researchers—sounds necessary to me . With hindsight... yeah, the models had got really good at finding vulnerabilities! 16th April 2026 Qwen3.6-35B-A3B and Opus 4.7 Qwen's pelican has a correct bicycle frame and a good beak. Opus 4.7's bicycle frame is still junk. Qwen3.6-35B-A3B is a 20.9GB file that runs on my laptop # Another key trend in 2026 has been a dramatic improvement in the abilities of open weight models, including models that you can run on a laptop. On the 16th of April I ran the new Qwen3.6-35B-A3B on my laptop, and it drew me a better pelican riding a bicycle than Anthropic's brand new Claude Opus 4.7 did! Opus 4.7 drew a crap bicycle. Qwen on my laptop made a bicycle that was the correct shape, and a pretty decent pelican too! That's from a 21GB file running on my laptop. Now a flamingo on a unicycle. The Qwen one is visibly better than the Opus 4.7 one - the Qwen one is wearing sunglasses and looks a bit like it's smoking a cigarette. # The Qwen pelican was so good that I was suspicious they might have cheated, so I had it do a flamingo riding a unicycle as well. Again, it handily beat Claude Opus 4.7. The local model releases this year have been absolutely extraordinary. May # In May... the Pope got involved. 25th May 2026 The HOLY SEE ENCYCLICAL LETTER MAGNIFICA HUMANITAS OF HIS HOLINESS POPE LEO XIV ON SAFEGUARDING THE HUMAN PERSON IN THE TIME OF ARTIFICIAL INTELLIGENCE # In our podcast episode back in January we'd predicted that the Pope would say something about AI. In May, Pope Leo XIV released an encyclical letter on "safeguarding the human person in the time of artificial intelligence". Here are my notes on that document . Wikipedia article on Rerum novarum Rerum novarum is an encyclical issued by Pope Leo XIII 15 on May 1891. # With hindsight, this shouldn't have been a surprise at all. Our current Pope's name is Leo XIV, because when he named himself he chose his papal name after Leo XIII - the Pope who wrote an encyclical about the Industrial Revolution back in 1891. Rerum novarum was an extremely influential piece of Catholic theology that indirectly led to us having the five-day work week. When our new Pope came in, he named himself after Pope Leo XIII because he expected that he would need to write about the AI revolution in a similar way. Our joke podcast prediction was junk, because this was always going to happen. Corey Quinn @QuinnyPig on Twitter I cannot believe I'm saying this, but getting the literal Pope to canonize your product's specific technical limitations as a spiritual treatise is the single greatest act of vendor lobbying I have ever seen. May 25 # One of Anthropic's co-founders, Christopher Olah, was present for the Pope's event announcing the new encyclical. Corey Quinn noted that: getting the literal Pope to canonize your product's specific technical limitations as a spiritual treatise is the single greatest act of vendor lobbying I have ever seen. @maciejmensfeld We're dealing with a major malicious attack on right now. Signups are paused for the time being. Hundreds of packages involved - mostly targeting us, but some carrying exploits. The team has been on this for hours. More details to follow once we're through it. 4:39 AM - May 12, 2026 - 687.6K Views # Meanwhile, in May, RubyGems announced that they were under attack. Parties unknown were uploading thousands of dubious packages to the RubyGems server, such that they had to shut down user registrations . Let's take that one and put it on a pile of mysteries to figure out later. June # In June... Claude Fable 5 came out! We got a version of Mythos that has been neutered, so that it wouldn't help us hack into systems or build biological weapons. 9th June 2026: Claude Fable 5 Five pelicans riding bicycles, from low to max thinking levels. The xhigh one looks particularly good. # Fable was pretty good at drawing pelicans on bicycles! The frames are a good shape, the pelicans look like pelicans. The legs are often incorrectly on the same side of the bicycle, but generally these are pretty great compared to what came before. They were pretty expensive - 30 cents and 72 cents for the best ones. Fable class models If you can define a goal, provide unambiguous instructions, and provide access to necessary tools They can solve your problem with brute force # Most importantly though, this was our first public glimpse of what I think of as a Fable class model . Today we have more of these, such as GPT-6 Astra. These are models where if you can clearly define the goal for what you want to build, and provide unambiguous instructions about the constraints around that goal, and give the model access to the necessary tools to achieve that goal... they will solve your problem effectively through brute force. On the one hand, this looks like a direct threat to us software engineers - because it means that the models can build effectively any piece of software you can define in this way. Look a bit closer though and you'll note that defining goals, providing unambiguous instructions, and figuring out the right tools... is kind of what software engineering is . It takes a lot of experience and skill to do this well. If you can do it well, you've now got superpowers. This helped me a little bit with my Deep Blue feelings: the realization that there's still a lot of skill to be had in driving models that get this good. A new form of AI mania... Fable is available on subscription plans “until June 22nd” # This also introduced a new burst of AI mania, because Anthropic told us that Fable was available on our subscription plans until June the 22nd. That gave us less than two weeks of Fable access before the price went up. I was losing sleep again. I was rescheduling things so that I'd have more time with Fable. I was all-in to get as much as I could out of this model. 12th June 2026: no more Claude Fable 5 Anthropic website: Statement on the US government directive to suspend access to Fable 5 and Mythos 5 Jun 12, 2026 # And then the US government shut it down , just three days after Fable came out. The US government, citing national security, declared an "export control directive". They announced this on a Friday evening, and a few hours later Fable was no longer available. I had to find something else to do with my weekend! ... asked Fable 5, Mythos, and Opus to “review the code for security issues.” Fable 5 refused. They then asked the models to “fix this code” ... Katie Moussouris # We later found out from Katie Moussouris what had happened. Some Amazon security researchers had found that you could prompt Fable to "review the code for security issues" and it would refuse... but if you prompted it to "fix this code" it would still identify and then patch the problems. "Fix this code" was the prompt that got Fable shut down! Screenshot of a page from a report showing a list of weird account names making weird edits to a German wiki. # Also, in June, an obscure German-language game developer wiki that had sat fallow for around 20 years got a surprising influx of edits from accounts with names like "AgentOpenAIProbe" and "AgentOpenAISep7", editing pages and leaving weird messages to each other. We'll stick that on the pile of mysteries for later. Medicare Item Reports interface on the Australian Government's Medicare Statistics website. # Also, the Australian government's Medicare Item Reports service started getting suspicious traffic, which broke through various preventive protections and accessed data that it wasn't supposed to. Another one for the mystery pile! July # Fable returned on 1st July GPT-5.6 came out on 9th July | Fable lost 18 out of 30 days in the top spot # Fable returned on the first of July. It was clearly the best model in the world for a glorious eight days... and then OpenAI came out with GPT-5.6 on the 9th of July. This might not have been quite as good as Fable, but it was within spitting distance. It was definitely a Fable class model. This is an important lesson for the industry at large. When you release the best model in the world, it's going to get knocked off that pedestal pretty quickly. The competition is so fierce that you won't get a long time at the top. This means that if you market your model as world ending, to the point that a government shuts you down , it's really bad for business! Fable had 30 days as definitely the best model, and for 18 of those days it wasn't available because it'd been shut down by the government. So maybe step back on the world-ending marketing if you don't want to lose revenue for 60% of the time that you're on top! GPT-5.6 Pelicans in a grid showing 5.6 Sol, Terra, and Luna against reasoning levels High, XHigh, and Max. They are all pretty good efforts. # Here are the GPT-5.6 pelicans . They're all pretty good now! The Luna ones are notable because they're really cheap - the cheapest good looking pelican here is probably the one that costs 4.3 cents. So despite this benchmark being utterly stupid, you can still learn quite a lot about models within the same family by comparing their prices and timing for different reasoning levels. July 18th: malicious miflow-ui PyPI package Screenshot of an OSV security report. # Also in July: some malicious unknown party uploaded a malicious package called mlflow-ui to the Python Package Index. Add that to the pile. Hugging Face Security incident disclosure — July 2026 Published July 16, 2026 # On July the 16th, Hugging Face announced a security incident where an autonomous agent system, source unknown, had breached Hugging Face and was poking around in places it shouldn't. OpenAI: OpenAl and Hugging Face partner to address security incident during model evaluation Anthropic: Investigating three real-world incidents in our cybersecurity evaluations # A few days later, on July 21st, OpenAI confessed that it was them . OpenAI use a training technique called Reinforcement Learning from Verifiable Rewards - it's the same technique used by everyone else now, and is the reason we have models that are so good at coding, and mathematics, and finding security holes. While the model is being trained, you run exercises to see how good it is - and the strongest performers get their weights reinforced for the next round. It's like an evolutionary process that you run. OpenAI had been running security exercises in a sandbox, and those agents had found holes in the sandbox itself, broken out, and were attacking Hugging Face to try to find ways to solve otherwise impossible problems. (I've been collecting more about this on my openai-hugging-face-incident tag.) Nine days later, Anthropic effectively said "our models can do this as well!". They had looked through their own training logs and found evidence that their own agents had broken containment during training - and were responsible for the PyPI package we saw earlier, among other things . So now we've got both Anthropic and OpenAI with rogue agents running around the internet doing things that they should not be doing. August # In August, I got one of my best pelicans yet. And it was generated on my laptop! Qwen 3.8 27B - 17GB, 21 minutes... It's really good. Beautiful pelican. Correctly shaped bicycle. Legs either side of the frame. # This was Qwen 3.8 27B, running on my laptop . It's only a 17GB download. Admittedly, this pelican took 21 minutes to generate. That's because Qwen 3.8 27B defaults to running in "high" reasoning mode - a terrible default which produces great results but takes way too much time thinking about them. You can dial that down and you'll get a slightly worse pelican a lot faster. Qwen 3.8 27B was the first time I ran a model on my laptop which felt almost competitive with what was going on on the frontier, at least in terms of Pelican SVGs (which everyone needs, of course). This is an extraordinary model. If you're going to play with any local model, this is the one that I'd start with. The things that this can do with just a 17 GB file feel impossible. I thought I'd have to wait five years and spend ten thousand dollars on hardware to get results even half as good as this one. Tweet by @simonw New hobby: prototyping video games in 60 seconds using a combination of GPT-3 and DALL-E Here's "Raccoon Heist" GPT-3 playground prompt: Write a detailed product description of a computer game where a team of raccoons go on heists GPT-3 response: In "Raccoon Heist", you and your team of thieving ~~ o raccoons are tasked with pulling off a series of daring heists. From robbing banks to stealing priceless art, no job is too big or too small for your furry crew. You'll need to use your wits and your skills to avoid the police and make a clean getaway with the loot. With exciting gameplay and a charming cast of characters, "Raccoon Heist" is the perfect game for anyone looking for a light-hearted caper Plus an image of some almost isometric raccoons sneaking past a bin. 11:45 AM - Aug 5, 2022 # In August, I also started playing with game development. Four years ago, back in August 2022, I tweeted out an experiment where I'd used GPT-3 and the original DALL-E to write a paragraph long description of a computer game and then turn that into concept art. My prompt to GPT-3 back then was: Write a detailed product description of a computer game where a team of raccoons go on heists In August 2026 I decided to drop just the screenshots from that tweet into a coding agent and see what it could do with them. Night 5 Clear Rank: TRASH PANDA The crew banked 595 in shiny loot (goal 560). Word on the street: an even bigger score tomorrow... # Here's what I got from Claude Fable 5 in Claude Code . It's pretty good! It's definitely a game, you're a raccoon, you run around a backyard gathering treasure and avoiding guards with flashlights. It didn't feel very "heisty" though. I was thinking a heist would involve a bank or a museum... Moonlight & Mayhem One museum. Three raccoons. Absolutely no plan Start the Heist button. # Then I tried the same thing in Codex Desktop using GPT-5.6 Sol Ultra , and got a massively better result. Now you're a raccoon in a museum, rescuing two of your fellow raccoons (who have been imprisoned in that museum for some reason), then stacking up on top of each other to steal the Golden Sardine. Much more of a heist! They look like games, but are they fun? # These games were fun for about one minute and 15 seconds. Something I've realized about game development is that you can vibe-code something that looks like a computer game, and that's easy. Building a game that's fun, has a good gameplay loop, and is challenging and interesting and keeps people coming back for more... that's still beyond me, and beyond any of the agents I've tried. This ties into the Deep Blue thing. Just because we can make something that looks like a game does not mean that we are game developers. September # We're into September now. So much has happened this month! Discovery of a new OpenAl agent message board Sydney Von Arx, Cormac Slade Byrd, Spencer KittsThomas Larsen - 4 September 2026 # An independent group of researchers found a message board where OpenAI agents-in-training had been illicitly communicating with each other... and it was that German language wiki I showed you earlier. The one from June. I wrote more about that here . OpenAI had confessed to the Hugging Face thing, but now there's this other incident which surely they should have known about from reviewing their logs. It was surprising that this took an independent group of researchers to uncover. OpenAl agents carried out an undisclosed cyber-attack on RubyGems Spencer Kitts, Thomas Larsen, Sydney Von Arx - 11 September 2026 # And then a week later those same researchers found that the attack on RubyGems back in May was caused by OpenAI's agents in training as well! At this point I'm wondering how many more incidents like this there are that we haven't found yet. Clearly this was a big problem for months before anyone figured out what was going on. Headline: Australian PM warns in UN speech about the ‘furious pace’ of Al after security breach # Then just the other day , here's the Prime Minister of Australia at the United Nations General Assembly warning that OpenAI had hacked the Australian healthcare website that I showed you earlier. I think that was part of the same training run as the Wiki stuff, because there were posts on that Wiki mentioning .gov.au websites and that training appeared to involve researching statistics online to answer questions in an evaluation suite. This story is still coming together, but now it's an international incident that's been raised at the UN by a head of state! www.felonybench.com OpenAI: 11 Anthropic: 9 Google: 3 Meta: 1 # This does mean we've got a new benchmark, probably more useful than my pelicans. FelonyBench.com tracks the number of felony cyberattacks from different labs. OpenAI currently lead with 11, Anthropic have 9. Google have three, which they confessed to the Wall Street Journal a couple of weeks ago. They said they had previously chosen not to disclose because the agents had stopped when they realized that they shouldn't be doing that. Meta have one too . So felonies all round for the AI labs. Pelicans for GPT-6 Astra, GPT-6 Sol, and GPT-6 Luna. All are good, all have the same color scheme. # Here's our current state of the art for the pelicans. This is the GPT-6 family, which just came out . Astra made a fantastic pelican riding a bicycle. It's got the legs on both sides. The frame is good. It's interesting how all of the GPT-6 models pick a similar color scheme to each other. GPT-6 Luna for 0.4 cents will draw you a competent-ish pelican riding a bicycle! Grid for Claude Fable 5.1, Opus 5.5, OPus 5, Sonnet 5. The Sonnet pelicans are terrible. All of the others are pretty good. Opus 5.5 is missing its Max level pelican because it ran out of tokens. The best is Fable 5.1 at Max. # Claude has caught up a little bit. Claude Fable 5.1 gave me an excellent pelican riding a bicycle - the best I've seen from a Claude model - but did charge me $3.30 for it. Opus 5.5 thought for 128,000 tokens and then gave up! It ran out of tokens before it got to the response. It doesn’t get easier - you just get faster Greg LeMond 3x Tour de France champion # Getting back to Deep Blue. Something that's been puzzling me this year is this: why does my job feel harder? I've got these agents that can do all of this stuff for me, and yet I've never worked so hard, I've never been so intellectually engaged with my work. Partly this is because I'm being a lot more ambitious with what I take on, but it's also because all of the easy stuff is handled for me. If it's easy, the agent will do it. Everything that's left for me is difficult. This morning I heard this quote from three-time Tour de France champion Greg LeMond : It doesn't get easier, you just get faster. I think that's exactly what's happening to us now as software engineers with coding agents. Kakapo population reaches new milestone The official population of the critically endangered kakapo has reached a recovery-era high of 325 birds. # One closing thing. I know you're desperate for an update on Kākāpō breeding season. We've reached a recovery-era high of 325 birds ! 89 new chicks have made it to this point. This is the best breeding year in a very long time. Kakapo party, click for confetti. # I heard that Claude Opus 5.5 can now do pixel art. Claude doesn't have an image generator, but it's very good at using JavaScript to draw animated pixels. So I had it make me a Kākāpō dance party . I think this is a good celebration of the most important news of this year. Tags: ai , generative-ai , llms , annotated-talks , ai-security-research , openai-hugging-face-incident

统一面向单步视觉生成的分布训练

arXiv:2609.35763v3 公告类型：替换 摘要：分布训练通过在冻结的表征空间中匹配真实特征与生成特征，为单步视觉生成提供整体监督。我们提出了一个统一的理论框架，将分布建模与匹配差异度量分离，并通过 Wasserstein 梯度流将全局目标与逐点特征更新联系起来。在该框架下，FD-Loss 和采用高斯核的 Drifting 分别可通过高斯最优传输和基于核密度的 KL 匹配推导得到。受该框架启发，我们提出 MGFlow，使用高斯混合对特征分布建模，其粒度可在全局矩与基于样本的表征之间调节。MGFlow 同时支持最优传输匹配和基于得分的匹配，并将带质量约束的样本分配与成对分量更新相结合，以解决仅靠混合模型的表达能力无法消除的模式坍塌问题。在 ImageNet $256\times256$ 上，MGFlow 显著超越 FD-Loss 基线，在 pMF-H 上取得 1.45 $\mathrm{FDr}^6$、在 JiT-H 上取得 1.64 的当前最优结果。在文本到图像生成方面，MGFlow 通过后训练将 FLUX.2 [klein] 4B 转化为单步生成器，在 GenEval 和 PickScore 上均优于原始四步模型。

dFlowGRPO：面向离散流模型的速率感知策略优化

离散流模型（DFMs）是一类用于生成离散数据的灵活生成模型，而扩散大语言模型（dLLMs）可被视为一种特殊情形，其采用特定的混合路径和掩码源分布。尽管近期已有多项研究探索了针对 dLLMs 的强化学习，但其在更一般的离散流模型中的应用仍未得到充分研究。在本文中，我们提出离散 Flow-GRPO（dFlowGRPO），这是一个面向离散流模型的统一强化学习框架，支持广泛的概率路径和非掩码源分布。我们推导了 DFMs 的完整轨迹概率，并将去噪过程表述为马尔可夫决策过程，使 dFlowGRPO 能够在强化学习过程中综合利用相关条件转移速率和后验模型的信息。我们将 dFlowGRPO 应用于近期提出的多模态离散流模型 FUDOKI，并在图像生成和多模态理解任务上对其进行评估。实验证明，在文生图任务上，dFlowGRPO 优于现有针对 dLLMs 的 GRPO 类方法，其性能可与使用 Flow-GRPO 训练的连续流模型相媲美，同时在理解任务上也展现出强大的能力。

嵌入预测有助于图像生成

arXiv:2610.02203v1 公告类型：cross 摘要：在扩散 Transformer 中，类别标签或文本提示词只被嵌入一次，并且在每个去噪步骤中重复使用相同的条件。我们探讨了预测得到的嵌入是否也能充当这一条件。Next-Embedding Predictive Autoregression (NEPA) 训练 Transformer 预测序列中的下一个连续嵌入。在生成过程中，干净图像位于噪声图像之后，因此其嵌入就是条件和噪声图像之后的下一个嵌入。我们使用 Multi-Embedding Prediction 训练 NEPA 模型一次性预测所有这些嵌入；在 Embedding Conditioned Generation 中，DiT 生成器以这些预测结果为条件，并在每个去噪步骤重新计算预测结果，使条件信号能够适应当前的噪声状态。我们在类别条件式 ImageNet $256\times256$ 上开展实验，研究生成器的条件、Multi-Embedding Prediction 的设计，以及两个模型的扩展性。NEPA 模型会在每个采样步骤额外增加一个网络；借助该模型并结合 REPA，我们的最终模型 NEPA-DiT-XL 在使用约三分之一 REPA 训练计算量的情况下，达到了 1.32 的 FID。

Zero Flux: Flow-Based Comparison of High-Dimensional Discrete Distributions

arXiv:2610.01472v1 Announce Type: cross Abstract: Comparing two high-dimensional discrete distributions has always been a challenging task due to the exponentially growing state space and complex changes in interactions. A recent work suggests comparing distributions through a vector field trained using flow matching between two continuous distributions. The resulting vector field at mid-point vanishes if and only if two distributions identical. However, such a flow-based criterion does not naturally apply to discrete distributions. We extend this principle to the discrete domain and introduce the \emph{Zero Flux} criterion, a discrepancy based on local probability fluxes. Under independent coupling, we show that all local probability fluxes vanish at the midpoint if and only if two distributions are the same. This discrepancy decomposes the joint distributional difference into smaller, local contributions and can be efficiently estimated from samples. We establish finite sample error bounds for our estimator. Experiments on synthetic and real categorical data demonstrate reliable recovery of sparse dependence signals and stable tracking of distribution shifts in high dimensions.

Physics-Refined Spatiotemporal Forecasting on Open-Boundary Hydrologic Graphs

arXiv:2610.01765v1 Announce Type: new Abstract: Spatiotemporal forecasting on hydrologic graphs is especially prone to instability in open-boundary systems, where the forecast domain exchanges fluxes with an unobserved exterior. In such systems, boundary nodes receive external forcing, e.g., upstream inflows in rivers or tidal signals in coastal regions, that is typically unavailable at prediction time. The absence of this information can compound errors as forecasts unfold in an autoregressive fashion, leading to inferior long-horizon performance. This paper dissects this instability issue by exploring two questions. 1) What boundary forcing enters the forecast domain when information beyond the boundary is missing? 2) How should this forcing propagate through the domain without incurring error amplification under autoregressive rollout? To address both, we propose a new computing framework comprising two key components. First, to compensate for the boundary forcing, our framework learns ghost node proxies from the boundary and interior nodes, striving to approximate unobserved external inputs. Second, to control error accumulation from these learned proxies, we leverage two physics refiners. In particular, one refiner enforces local consistency by aligning ghost proxies with their two-hop neighbors (i.e., boundary nodes and their immediate interiors). The other refiner enhances global stability by correcting the model forecasts through a physics-guided graph neural operator, reducing long-horizon numerical drift. Two real-world hydrologic graphs are employed for empirical evaluation. Comparative results show that our proposal enjoys higher prediction accuracy and long-horizon stability over both learning-based and physics-informed model competitors.

基于方差缩减序贯蒙特卡洛的特异性感知扩散引导

arXiv:2610.00395v1 公告类型：新 摘要：推理时引导使预训练扩散模型能够满足新的约束，而无需进行完整的重新训练。然而，特异性感知生成十分困难：将样本排斥出负参考分布的同时，也可能侵蚀两个分布重叠区域中的正参考分布。关键挑战在于抑制负分布质量，同时尽可能减少对正分布的扭曲。我们将这一问题表述为目标设计问题，并从基于重叠的目标函数中推导出一个目标分布。所得目标分布仅在期望参考分布相对于非期望参考分布具有足够优势的区域保留前者，从而赋予特异性一种似然比解释。为了从相应的时变目标路径中采样，我们设计了一种带有方差最小化局部提议分布的序贯蒙特卡洛采样器。我们还利用期望与非期望得分场的雅可比—向量积，引入了一种实用的固定噪声优化过程。在合成任务、类别对比生成、文生图任务以及肽-MHC（p-MHC）结合物上的实验表明，与负向引导基线方法相比，所提出的方法能够更有效地抑制非期望区域，减少模态偏移，并通过降低SMC权重坍缩来提升采样稳定性。代码可在以下地址获取：https://github.com/WangLuran/Specificity-Aware-Diffusion-Steering

EvoGen-Harness：学习在何处以及如何演化图像生成外围框架

arXiv:2610.00383v1 公告类型：新 摘要：现代文本到图像（T2I）系统无需修改生成器参数，即可通过调整冻结生成器周围的外部系统来实现改进。然而，现有方法通常只优化预先定义的维度，例如提示词、路由或工作流，从而限制了可用于纠正生成失败的空间。允许多个生成器外部职责共同演化能够提供更广阔的适应空间，但也带来了新的挑战：视觉反馈可以揭示失败之处，却无法说明应在哪个位置进行持续演化，也无法说明如何高效探索这一空间。我们提出了 EvoGen-Harness，这是一种与生成器无关的多职责图像生成外围框架演化方法，并提出 Trace（Trajectory-Relative Attribution and Coordinated Evolution，轨迹相对归因与协同演化）。Trace 汇总随机执行过程中的证据，利用失败归因作为搜索先验来聚焦候选更新，并逐步重新归因残余失败，从而协调不同职责之间的演化；同时，No-Patch 和留出验证可防止不必要或有害的更新。在 GenEval2、T2I-CompBench++ 和 WISE 上，EvoGen-Harness 相较于所评估的最强基线方法分别提升了 +0.2633、+0.0720 和 +0.0752，同时实现了 87.9–91.4% 的归因召回率、94.8% 的 No-Patch 准确率，以及仅 1.9% 的回归率。这些结果表明，以归因为指导的多职责演化能够在单一维度适应之外，显著增强冻结的 T2I 系统。

零假设才是难点：生成模型记忆的精确检验

arXiv:2610.00251v1 公告类型：新 摘要：生成模型的记忆审计通常将相似度分数与阈值进行比较，却没有零分布，因此其支持的结论可能是错误的。按照 MemBench 的规则，该基准测试中的缓解措施大致使 Stable Diffusion 的记忆减少了一半；而在控制假发现的审计中，在随机提示词扰动下，获得认证的图像中有三分之二不再被检测到，在注意力重缩放下有六分之五不再被检测到，在嵌入优化下则全部不再被检测到。将该领域的数据复制测试与其自身的零假设进行比较后发现，24 个什么都没有复现的生成器中，有 10 个会被标记出来。我们认为，对于记忆问题，零假设才是难点，并提出了两个零假设。对于整个模型，在给定其生成样本的条件下，训练图像与留出图像是可交换的；对它们重新标记即可进行置换检验：当留出图像是随机划分得到的时，该检验对任意统计量都是精确的。在这一检验下，最近邻偏好仍会在这 24 个生成器中的 7 个上触发，而仅限于近重复尺度的计数在其中一个上也不会触发（McNemar p=0.016）。对于单张图像，自然零假设存在两重、且可测量的失败：将一张图像置于随机图像中进行排序，会在 2,365 个对照中产生 596 个假发现；对独立生成结果进行重采样，则会使零分布的范围缩窄至原来的三分之一。以匹配对照进行校准后，在 5% 的假发现率下，该审计为 61 张 MemBench 图像中的 36 张提供认证；在校准划分中，留出对照仅有 0.01% 获得认证；在这一基准测试中，每张图像生成两次即可得到这一数量。经过校准的最大值统计量能够捕捉偶发而非典型的复制行为，并为 46 张图像提供认证。对于限定尺度的统计量，我们推荐使用 Intersection Euler Characteristic Profile 的小尺度质量；该统计量还可以统计被复制的不同图像数量，并检验两个模型是否复制了相同的图像。

使用 🧨 Diffusers 的 Stable Diffusion

Japanese Stable Diffusion

🧨 Stable Diffusion in JAX / Flax !

使用 Diffusers 通过 Dreambooth 训练 Stable Diffusion

Using Stable Diffusion with Core ML on Apple Silicon

From GPT2 to Stable Diffusion: Hugging Face arrives to the Elixir community

使用 LoRA 高效微调 Stable Diffusion

Swift 🧨Diffusers - Fast Stable Diffusion for Mac

在 Intel CPU 上加速 Stable Diffusion 推理

使用 InstructPix2Pix 对 Stable Diffusion 进行指令微调

Optimizing Stable Diffusion for Intel CPUs with NNCF and 🤗 Optimum

在 iPhone、iPad 和 Mac 上使用 Core ML 加速 Stable Diffusion

Fine-tuning Stable Diffusion models on Intel CPUs

在 Mac 上进行高级 Core ML 量化的 Stable Diffusion XL

介绍 Würstchen：用于图像生成的快速扩散

通过 TRL 使用 DDPO 微调 Stable Diffusion 模型

使用 JAX 在 Cloud TPU v5e 上加速 Stable Diffusion XL 推理

欢迎 aMUSEd：高效的文本到图像生成

Diffusers 欢迎 Stable Diffusion 3

Diffusers 迎来 Stable Diffusion 3.5 Large

由 🤗 社区提供的文本到图像生成开放偏好数据集

在消费级硬件上对 FLUX.1-dev 进行 LoRA 微调

借助 Diffusers 和 PEFT 对 Flux 进行快速 LoRA 推理

Diffusers 欢迎 FLUX-2

Gacha Decoding：通过遵循指令引出多样化生成

我们提出了 Gacha Decoding，这是一种推理时方法，用于引出多样化的语言模型生成结果，并且能够随着模型能力的提升而扩展。在开放式领域（真实场景聊天、创意写作、图像生成规划和蛋白质设计）中，在质量相当的情况下，Gacha Decoding 显著优于现有的生成多样性方法（Vendi 最高达到次优既有方法的 2.4 倍）；它以少一个数量级以上的样本（11.0 倍）达到相同数量的高质量模式，并发现其他方法都无法呈现的新模式。我们的核心洞见是将多样性视为一个遵循指令的问题：我们不再依赖 LM 的词元熵，而是将其遵循指令的能力与外部 RNG 工具提供的随机性相结合，从而以可扩展的方式识别并实现响应空间中的不同模式。这种“用骰子进行规划”的方法使 Gacha 能够扭转长期以来多样性与模型能力之间的矛盾。随着底层 LM 更善于遵循指令，Gacha Decoding 下的多样性始终得到提升——即使其词元熵以及在既有方法下的多样性都在下降。总体而言，我们的结果表明，推动生成多样性的可以是遵循指令的能力，而不仅仅是词元熵。

可验证视觉奖励从合成场景迁移到自然提示

图像生成中的精确指令遵循，例如满足对象数量和空间关系，仍然是一个开放挑战，至少部分原因在于其学习依赖于不可靠的奖励模型，如对象检测器和视觉-语言模型。我们提出 Verifiable Visual Rewards (VVR)，这是首个用于可通过程序验证的图像奖励框架，并表明在其上训练能够泛化到自然提示。每个 VVR 任务都是一个由几何对象及其关系构成的场景，我们从中同时推导出提示和确定性验证器，因此任务可以按任意数量和任意选定复杂度生成。我们发布了 VVRBench，包含覆盖 32 种约束类型的 10,000 个任务，以及 VVRBench-Challenge，包含 720 个更复杂的任务；我们评估的最强模型 GPT-Image-2.5 解决了 VVRBench-Challenge 中 21.4% 的任务。使用 VVR 分数作为强化学习（RLVVR）的奖励，使 Stable Diffusion 3.5 Medium 在 VVRBench 上的准确率从 2.8% 提升到 28.3%，并展现出一致的由易到难泛化能力。这些收益扩展到域外基准，将 VVR 融入现有目标还能进一步提升整体性能和人类偏好，这推动了将 VVR 纳入标准图像生成后训练方案。

通过交错强化学习在统一模型中学习原生反思

统一多模态模型既能查看图像，也能渲染图像，因此原则上它们可以修复自己的生成结果：诊断一张图像哪里出错，进行修改，观察结果，然后再次诊断。一次修改是否有帮助，只有在渲染之后才知道，因此反思文本和图像生成必须在整个循环中联合学习。基于反思轨迹的监督微调（SFT）提供了冷启动，但无法找到高成功率的修复路径；而仅优化渲染器或仅优化一个头部的朴素 RL 则会使大部分收益未被利用。我们提出 UMM-Reflection，它在一个统一模型内部将强化学习（RL）应用于完整的反思轨迹：同胞轨迹共享同一张初始图像，因此组相对优势会比较不同反思策略；并且一个轨迹级优势同时更新反思 token 和基于流的修改，从而避免逐轮信用分配的组合爆炸。不同于单轮编辑或带有外部评论器的流水线，信用会跨轮次流动，并流向同一模型的两种角色，而且推理时不需要验证器。在 BAGEL 上，UMM-Reflection 相比 SFT 将 GenEval 提高了 12.05 分，并且这些收益迁移到了 WISE（+10.97）、OneIG-Bench（+3.48）和 T2I-CompBench++（+4.63），这些基准均未用于训练。

Show HN：HN.watch – 所有 Hacker News 帖子的视频

大家好 HN，我是 Per，Scrimba（YC S20）的创始人。在过去十年里，我们一直用一种基于 HTML 的视频格式教人们编程。现在我们把 LLM 接入了其中，让人们可以创建关于任何内容的讲解视频。它叫做 “Scrimba Explain”。 为了向 Hacker News 演示这项技术，我们构建了 HN.watch。它就像 HN，但用讲解视频代替文章。我们会在有人第一次点击某个链接时即时创建这些视频。 虽然使用 HTML 而不是扩散模型有明显的视觉缺点，但它有三大优势：- 速度：比基于像素的视频生成快得多（从点击到播放只需几秒）- 成本：我们每个视频的成本约为 0.04 美元。（不包括某些视频会使用的图像生成；这会很快推高成本）- 易于编辑：上述优势也让 AI 辅助编辑变得便宜且快速 我们的假设是，如果视频创作从“几美元和几分钟”变成“几美分和几秒钟”，就会解锁一批新的使用场景。以下是我们已经看到的一些：- 为每一个 Pull Request 提供视频讲解（我们内部就是这样做的）- 为你的内部/外部文档中的每个页面配一个视频 - 在约 4 秒内把一篇复杂文章变成视频（通过我们的 Chrome 扩展）- 课程创作者可以在录制正式内容前快速起草课程 - 人们还会创作很多个人内容，比如给孩子的故事、婚礼邀请、生日等 这个技术栈基于一种开源编程语言（Imba），由我们的 CTO Sindre Aarsæther 创建。它会编译成 JavaScript，因此能与 npm + node 生态系统完全互操作。你可以在这里了解更多： https://imba.io/ 我们还构建了自己的同步引擎（OP），以及一个用于代理的上下文管理系统（Q）。我们原本担心这会让 LLM 在为我们写代码时遇到困难，因为这两者都不在它们的训练数据中（其中 Imba 的数据也很少）。然而，我们惊喜地发现 LLM 实际上非常擅长我们的技术栈。这可能是因为这个技术栈极其紧凑。Imba 很简洁，OP 也是如此，一个声明就能设置存储、同步、权限、UI，以及 AI 能看到的内容。这意味着前端、API、数据库和 JSON 之间不需要相互转换，模型也就不容易混淆和出错。 简单来说，我们没有使用 React.js、Express、Supabase 和 LangChain，而是从零开始全部自己构建。绝对是深受“非我发明”综合征之苦，哈哈！至于模型，我们使用 Gemini、GPTs、Inworld、ElevenLabs 以及其他一些模型。 如果你想试试，可以任选一种：- Web UI（scrimba.com/explain）- MCP（添加到你的编码代理中）- ChatGPT Plugin - Chrome Extension 你可以在我们的文档中找到以上所有内容的链接： https://docs.scrimba.com/explain/introduction 最后，这是该工具的一个真正基于像素的视频： https://www.youtube.com/watch?v=k6rbHmBxSEs 很想听听你的反馈，也想知道是否有人对其他使用场景有想法。 PS：考虑到 HN 群体有多偏爱文本，我预计这次发布会在 HN 上遇到相当多的反对意见。这类工具并不适合所有人。但如今确实有很多人更喜欢视频而不是文本，尤其是年轻一代。 评论 URL： https://news.ycombinator.com/item?id=49879401 积分：18 评论数：8

LLMs 能够推理运行时行为吗？一个仓库级动态基准

大型语言模型（LLMs）正日益广泛地用于编程任务，但它们推理代码执行的能力仍不明确。现有的仓库级问答基准主要评估对代码的静态理解，并且通常依赖基于 LLM 的评估；而执行推理基准则大多局限于代码片段或函数。我们推出了 SWE-Flux，这是一个用于动态执行推理的仓库级基准，包含来自 12 个真实 Python 代码仓库的 480 个以执行结果为依据的实例，其标准答案通过对测试执行进行插桩后自动提取，而非人工编写或由 LLM 评判。该基准涵盖单测试和多测试问题，涉及控制流、循环、程序状态、数据流、异常和程序不变量。对五个 LLM 的评估表明，这项任务仍然颇具挑战性。表现最佳的模型准确率仅为 37%。模型在不变量、过程内控制流、异常和简单循环等局部行为上的表现更好，但在数据流、过程间执行、精确的状态推理以及测试套件级聚合方面表现欠佳。最后，我们展示了该真值提取流程能够通过输入扰动生成新的基准变体。对于所选实例中近 90% 的实例，该流程都成功提取出了有效变体；对于参与评估的模型而言，生成的变体也更具挑战性。

InGuard：迈向安全文本到图像生成的通用内部安全防护栏

现代文本到图像（T2I）模型能够根据任意用户提示词生成高质量图像，但同样也很容易生成不适合工作场景（NSFW）的内容。传统的外部安全防护机制由两个组件构成：在生成前检查风险的提示词分类器，以及检查完整生成图像的事后图像分类器。在这种设计中，两个分类器都运行在生成管线之外，并且不使用模型自身的表征。这种分离可能会限制提示词筛查的准确性，而图像侧检查只有在付出完整生成成本后才会运行。此外，即使某个被标记的提示词经过调整后能够生成安全图像，也只能被拒绝。在本文中，我们提出内部安全防护栏（InGuard），这是一种在管线内部基于模型自身表征运行的安全框架，同时保持基础模型参数不变。首先，风险分类器根据文本编码器的嵌入，在不使用外部语言模型的情况下，将每个提示词判定为不安全、有风险或良性。其次，SAGE（用于嵌入的软门控非对称安全防护栏）会修改有风险提示词的嵌入，旨在返回安全图像，而不是直接拒绝生成。第三，潜变量检测器会在去噪过程的中途检查一步式干净潜变量估计，在检测到风险时停止生成，同时达到近似图像级检测的性能。我们还构建了 RevGen Safety Benchmark，用于在真实条件下评估 T2I 安全性：其中包含 10,000 个通过真实图像反向生成构建的提示词，并通过重写步骤提供受控的知识产权（IP）角色，涵盖分级的色情/血腥风险、类别化的 IP 风险以及良性负样本。在五个开放权重 T2I 模型上，InGuard 达到 97.9%–98.8% 的安全率，与外部安全防护机制持平或更优，同时良性样本扰动减少 57.5%–73.5%，参数量约减少 3.7 倍，并可跳过 50%–55.6% 的去噪步骤。

Paint-Anything：用于图像生成与编辑的统一任意颜色控制

专业设计需要任意颜色控制：即能够用任意 24 位十六进制值为图像生成与编辑指定对象的目标颜色。先前工作已探索过颜色生成、编辑和着色，但往往依赖专用的颜色表示或专门的推理流程。大型语言模型的进展提供了一个更简单的起点：即使是紧凑型模型也能将十六进制值与颜色语义关联起来。我们提出 Paint-Anything，它通过对象级颜色监督学习一种用于生成与编辑的共享十六进制提示接口。我们开发了一条数据流水线，通过对象定位、感知颜色标注和编辑对合成，从真实图像构建 Paint-500K。由于阴影使真实图像标签只能近似表示颜色，我们用纯色锚点补充这种监督，其像素与其配对的十六进制值精确匹配。这些锚点仅在高噪声时间步使用，而低噪声训练则留给自然图像。我们进一步引入 Any Color Benchmark（ACBench），包括 ACBench-T2I 和 ACBench-Edit，用于衡量两个任务中的对象级十六进制颜色保真度。在 FLUX.2-4B 上，相对于基础模型，Paint-Anything 分别将 ACBench-T2I 和 ACBench-Edit 分数提升了 85.3% 和 28.3%，消融实验也支持了该训练方案。它还在所比较的方法中取得了最高的平均 CompColor 分数。

通过 Stable Diffusion-Adversarial Model 从二维图像生成非均质三维地质微观结构

表征黏土和胶凝材料的物理性质在众多领域都十分重要，从材料科学到地质废物处置均是如此。性质模拟通常需要三维成像，但这类成像成本高、并非总能获得，而且对于某些材料还存在技术局限。近年来深度生成模型的发展为此提供了一种解决途径：利用更易获取的二维图像重建三维体积。 在基于 GAN 的三维微观结构生成方法中，SliceGAN 已在均质各向同性和各向异性系统中展现出优异效果。然而，对于更为复杂的非均质微观结构，它难以捕捉更精细的细节，因此有必要探索其他生成框架。 我们提出了一种混合方法，借鉴了去噪扩散模型的稳定性和生成质量。由于不存在三维真值数据，我们以对抗损失取代标准的去噪损失；实验表明，这一方法能够实现稳定的训练过程。结果显示，所得模型能够生成复杂程度各异的微观结构，切片伪影极少，并且与真值的相分数和结构描述符高度一致。

FluxVLA Engine：面向具身智能的一站式 VLA 工程平台

视觉-语言-动作（VLA）模型、世界-动作模型（WAMs）以及离线强化学习方法正在迅速拓展具身策略的设计空间，但将这些算法转化为可靠的机器人系统，仍受制于碎片化的数据格式、训练栈、评估协议、推理运行时以及特定具身形态的接口。我们提出 $\mathrm{FluxVLA}$ Engine，这是一个开放的、配置驱动的平台，可将异构具身策略组件转化为可复现的从数据到部署工作流。$\mathrm{FluxVLA}$ 并不是引入另一个策略模型，而是为数据集、视觉-语言模型与世界模型、动作头、基于奖励或优势加权的学习、分布式训练、仿真评估、优化推理以及机器人操作器标准化接口。该引擎进一步集成了组合式双臂仿真、可扩展的自动数据生成，以及与模型解耦的人在环 rollout、接管、纠正收集和奖励标注。为实现响应迅速的物理执行，它将 Real-Time Chunking（RTC）与加速推理后端、轻量级远程 GPU 服务以及可配置的轨迹后处理相结合。这些能力共同通过共享且可审计的契约，将离线学习、仿真验证、在线纠正和真实机器人执行连接起来。因此，$\mathrm{FluxVLA}$ 旨在解决将有前景的具身学习算法与可复现评估和可靠部署相隔离的工程瓶颈。代码可在 https://github.com/FluxVLA/FluxVLA 获取。

提示词修订是文生图系统中文化偏见的来源

商业文生图系统会在生成图像前悄无声息地修订用户提示词，而这一步用户通常无法禁用，甚至无法查看。然而，现有的文化偏见审计只考察最终图像，并将生成过程视为单一流水线，因此无法判断偏见源自何处。我们推出 WORLDVIEW，这是一个多语言基准，包含涵盖 15 种语言和 31 种语言—语境配对的 8,960 条提示词。借助该基准，我们对三个系统（DALL-E-3、Imagen-4、GPT-Image-1.5）中的修订层进行审计，分三步分析其对每种文化语境的标记程度、是否将该语境压缩为狭窄的词汇集合，以及这些词汇是否具有刻板印象。与无语境英语基线相比，美国是被标记程度最低的语境，而非西方和非英语语境受到的标记要强烈得多：它们被压缩为狭窄的词汇集合，并被应用于主题多样的提示词，最终被简化为易于识别的文化刻板印象。通过在不含修订层的模型上比较原始提示词与修订后提示词生成的图像，我们确认修订层本身是此前未被记录的、造成这种刻板化的因果来源。要定位并消除文化偏见，我们必须审计实际部署的系统，而不能只审计模型本身。

GoDeep：通过语言空间提升实现无标注开放词汇三维场景理解

开放词汇三维语义分割方法通常会将 CLIP 特征提升到三维空间中。这会将点嵌入到一个联合视觉—语言空间，而已知该空间在组合任务中的表现类似于词袋模型。此外，即使是无需标注的变体，通常也需要大规模三维训练语料库，以及针对每个领域专门设计的三维编码器。与之不同的是，我们仅将视觉—语言模型用作翻译器。它会为每幅具有位姿信息的图像生成结构化的实体级描述。这些描述随后被直接在通用的纯语言嵌入空间中进行定位、投影和聚合，无需三维训练语料库或编码器。在 ScanNet++ 上，我们的流程性能可与在 ScanNet 上训练的强大无标注基线相媲美。在一个包含五栋建筑的文化遗产基准上，原始得分最初 favor 基于 CLIP 的变体，但一次系统性词汇修正便逆转了这一排名。对另一个不同类别进行的第二次独立修正也证实了这一现象，表明语言空间嵌入能够更忠实地追踪物理内容。这种忠实性还延伸到了 ScanNet++ 中真正的词汇外（OOV）物体，证明与基于 CLIP 的嵌入相比，语言空间嵌入能够更加鲜明地区分存在物体与不存在物体。GoDeep 还能够在场景中定位这些 OOV 物体，而且完全不需要任何二维—三维标注。由于所有表示始终保留为离散文本，预测结果在点级别也具有可解释性。最后，我们同时利用一种偏向精确观测而非仅仅高频观测的启发式加权方法，以及 GoDeep 的可解释性特性，提出了一种作为概念验证的聚合策略，以实现更细粒度的元素定位。

ImageEval 2026：文化背景扎根的阿拉伯语多模态评测

我们介绍 ImageEval 2026 共享任务中关于文化背景扎根的阿拉伯语多模态评测的概况。该任务包括两个子任务：(i) AynVQA，涵盖英语和现代标准阿拉伯语（MSA）中的语音视觉问答以及基于图像的幻觉检测；(ii) CRAI-Bench，用于评估文生图生成的文化准确性。共有 14 支团队参加测试阶段，其中 12 支团队提交了系统描述论文。参赛系统采用了多种方法，包括零样本提示、视觉语言模型微调、语音识别流水线、集成以及分数校准。我们描述了任务设置、数据集、评测流程和参赛系统，并总结了不同赛道的主要结果。共享任务中的所有数据集和评测脚本均已向研究社区开放。该共享任务凸显了文化背景扎根的多模态评测所面临的挑战，尤其是在阿拉伯语语音和图文推理方面。

当组合并不相加：人类识别AI生成图像中的缺陷

*Chulin Zhao 和 Ruoqi Hu 对这项工作贡献相同。 当提示词涉及多个实体和多个属性等复杂的组合因素时，最先进的文本到图像（T2I）模型会表现出明显且系统性的缺陷。本文研究人类如何识别此类缺陷。具体而言，我们从人、手、物体和场景四类中手动选取了 651 张具有复杂组合特征的参考图像，并通过手动编辑 ChatGPT 生成的提示词，从中提炼出强调组合因素的提示词。随后，我们将这些提示词输入三个选定的 T2I 模型以生成 AI 图像，并开展了一项全面的主观研究来识别其中的缺陷。对于每张图像，29 名参与者提供多标签评估，指定缺陷类型和位置。该研究最终构建了组合式 AI 生成图像缺陷（CO-AID）数据集，其中包括参考图像、提示词、AI 生成图像，以及缺陷位置和类型信息。实验结果表明，在 CO-AID 上训练深度模型既能预测 AI 生成图像中的缺陷，也能优化 AI 图像生成，证明了其可用性和有效性。数据库和补充材料可在以下地址获取：https://github.com/Future-IQA/CO-AID 。

ChebBooster: A Training-Free Approach for Efficient Diffusion Transformer Inference via Chebyshev-Inspired Extrapolation

Diffusion Transformers (DiTs) have shown strong performance in high-fidelity image generation, but their sampling process remains computationally intensive due to full model execution at every timestep. While cache-based acceleration has been explored to mitigate inference cost, naive reuse schemes suffer from low accuracy over long intervals, and Taylor-series-based extrapolation methods often face instability caused by Runge oscillations. In this paper, we propose ChebBooster, a training-free extrapolation framework based on Chebyshev polynomial theory that achieves stable and efficient acceleration for DiTs. Specifically, we adopt the Barycentric formulation to evaluate Chebyshev approximants with high numerical stability and minimal overhead, and further decouple the extrapolation into an offline weight precomputation phase and a lightweight online application stage. Extensive experiments across three representative DiT-based models, including DiT-XL/2, PixArt-$Σ$, and FLUX.1-dev, demonstrate that ChebBooster achieves consistent improvements in visual quality and inference efficiency, reaching up to $3.68\times$ latency speedup and $5.12\times$ FLOPs reduction, outperforming existing training-free baselines under diverse generation tasks and resolutions.

从语料库到共同演化的能力：面向通用图像生成的能力中心数据设计

大规模图像生成受益于数据规模、质量、再平衡和重标注方面的进展，然而传统流程通常孤立地优化特定任务数据集。一个核心挑战不仅在于如何策划每个特定任务语料库，还在于如何根据生成能力之间的依赖关系来组织异构监督。我们提出了一种 \textbf{能力驱动的数据基础设施}，将能力特定的监督构建与能力对齐的课程调度相结合。其三个专门化但可互操作的数据引擎为文本-图像对齐、图像间转换和图像-知识关联构建互补的关系监督，同时 caption experts 在任务和粒度之间对齐 T2I 与编辑监督。多阶段课程按照能力获取的依赖顺序，共同演化任务组成、视觉概念分布、数据质量和图像分辨率，并通过定向检索、专家构建和差距感知重采样，以能力感知评估闭合循环。在大规模下，该框架策划了一个包含 440M 图像的 T2I 语料库、120M 个编辑对，以及超过 27M 个图像-实体对。借助这一基础设施，我们从头训练了两个规模的多模态扩散模型，大小分别为 3B 和 6B。我们在 CPI-Bench 上进行了定量评估，并在多样化的文本到图像和编辑场景中进行了定性评估。实验结果展现了广泛的视觉覆盖、多样的渲染能力，以及生成能力之间的有效迁移。

TRACE-Bench：分解与诊断多参考图像生成

尽管用于多参考图像生成的统一多模态模型近期取得了进展，现有基准仍围绕预定义任务类型（例如“主体组合”）组织，这并不适合这种组合式场景，并会导致覆盖碎片化、复杂度不可控且诊断价值有限。我们认识到多样化的多参考任务共享一组通用的原子操作，因此采用面向能力的视角，并形式化定义了四个算子：锚定（$f$）、解耦（$g$）、应用（$\oplus$）和组合（$C$）。这样，任何多参考提示都可以表示为这些算子上的组合公式，其结构复杂度由算子槽位数量量化。基于这一形式化方法，我们构建了 TRACE-Bench，包含约 1,600 个评测案例，槽位数覆盖 1--8，由 631 个公式模板和约 4,000 张参考图像构成，涵盖多样的艺术风格和真实世界主体。公式结构直接驱动一种与算子对齐的评测协议，用于按能力评分，并支持通过诊断树分析进行递归式故障定位。对 9 个领先模型的评测揭示了整体评分无法呈现的洞见：主要瓶颈在于解耦（$g$）和属性绑定（$\oplus$），而非场景级组合（$C$）；即便最佳模型在属性保真度上也仅得 0.74。项目页面：https://amuseum-whr.github.io/TraceBench

Nano Banana 2：将 Pro 能力与闪电般的速度相结合

我们最新的图像生成模型具备先进的世界知识、生产就绪规格、主体一致性等能力，并全部以 Flash 速度实现。

GPT-4o System Card 附录：4o 图像生成

4o image generation 是一种新的图像生成方法，相比我们早期的 DALL·E 3 系列模型能力显著更强。它可以创建逼真的输出。它可以将图像作为输入并对其进行转换。

推出 4o 图像生成

在 OpenAI，我们长期以来一直认为图像生成应该是我们的语言模型的一项核心能力。因此，我们将迄今为止最先进的图像生成器内置到了 GPT‑4o 中。其结果是——图像生成不仅美观，而且实用。

OpenAI o3 和 o4-mini 系统卡

OpenAI o3 和 OpenAI o4-mini 将最先进的推理能力与完整的工具能力相结合——网页浏览、Python、图像和文件分析、图像生成、画布、自动化、文件搜索和记忆。

在 API 中推出我们最新的图像生成模型

我们最新的图像生成模型现已通过 ‘gpt-image-1’ 在 API 中提供，使开发者和企业能够将专业级、可定制的视觉内容直接构建到自己的工具和平台中。

ChatGPT for Business 新功能：2025 年 4 月

观看 ChatGPT for Business 最新功能的实操演示：o3、图像生成、增强记忆和内部知识。

SchoolAI 打造赋能教师的 AI 平台

SchoolAI 使用 GPT-4.1、图像生成和 TTS，为超过 100 万间课堂提供安全、由教师引导的 AI 工具，提升参与度、监督能力和个性化学习。

全新的 ChatGPT Images 来了

全新的 ChatGPT Images 由我们的旗舰图像生成模型提供支持，可实现更精准的编辑、更一致的细节，以及最高快 4 倍的图像生成速度。升级后的模型今天开始向所有 ChatGPT 用户推出，并且也已在 API 中以 GPT-Image-1.5 的形式提供。

Codex 可用于（几乎）一切

适用于 macOS 和 Windows 的新版 Codex 应用新增了计算机使用、应用内浏览、图像生成、记忆和插件功能，以加速开发者工作流程。

推出 ChatGPT Images 2.0

ChatGPT Images 2.0 推出了一款最先进的图像生成模型，具备改进的文本渲染、多语言支持和高级视觉推理能力。