Evaluating Rollout-Verified Reinforcement Fine-Tuning of a Reasoning Model for Multi-Zone Variable-Air-Volume Control

arXiv:2607.27914v2 Announce Type: replace Abstract: Heating, ventilation, and air-conditioning systems account for a large share of building energy use. High-performance control often requires building-specific modeling or training, limiting scalability. We investigate whether a frontier reasoning model -- a large language model trained to use additional inference-time computation -- can coordinate multi-zone variable-air-volume control without building-specific training, and whether rollout-verified reinforcement fine-tuning can improve a smaller, locally deployable model. Five controllers were evaluated over five summer days in a physics-based four-zone emulator. Across three runs, GPT-5 used 6.8-7.4% less electricity than a Guideline 36-based baseline, but used less outdoor air and had lower CO$_2$ compliance. GPT-5 provided a capability reference for fine-tuning the smaller model. A trained reinforcement learning controller used 4.5% less electricity than the baseline while improving temperature and CO$_2$ compliance. To assess this controller's critic as a lower-cost verifier, we compared its estimates with deterministic rollout returns. Across 470 candidates, their correlation was 0.9995, yet the critic selected the best action in only 7 of 30 states. High overall correlation did not ensure reliable ranking at the same state. We fine-tuned gpt-oss-20b using rollout returns as rewards. One 200-step run produced no sustained improvement in the returns of sampled actions and no improvement in closed-loop performance. Analysis of the model's reasoning traces and quantitative tests showed that its next-state predictions of temperature and CO$_2$ were unreliable both before and after fine-tuning. Because next-state prediction is important for decision-making, these findings motivate testing whether supervised fine-tuning on next-state prediction improves subsequent rollout-verified reinforcement fine-tuning.