Steering an LLM AutoResearch Loop with Domain Context: A Case Study with Vehicle Energy Models

Research Software Engineers (RSEs) often work with Machine Learning (ML) models integratedinto scientific software stacks, and incremental model improvements are valuable but expensivein human time and attention. Our team maintains RouteE-Powertrain, which predicts vehicleenergy consumption and is integrated into RouteE-Compass for energy-aware routing.We investigated whether a Large Language Model (LLM) agent could help accelerate thediscovery process for model improvements by taking on certain elements that are challengingfor us, like context switching and managing limited time budgets. We began with an existingautomated research harness (Andrej Karpathy's autoresearch) where an agent canrepeatedly modify the model and then evaluate against a held-out dataset. This harness foundlarge improvements in our error metrics, but upon further inspection, the winning model wasunusable in our application domain. Specifically, it used unobservable future road network linkstates that are not compatible with our energy-aware route planning application and yielded avery large compute cost at inference time, leading to prohibitive computational overhead forroute planning. We also identified the need for additional agent instruction around logging,single-variable experimentation, token accounting, and trial budgets to support improvedanalysis. We accumulated these discoveries into a modified harness that would guide the agentwithin our narrower optimization space.We then ran two sessions over the same modified harness, each ~50 trials, ~8 hours wall time,~$35 token cost. The first session included our complete modified harness, whereas the secondsession excluded only the domain-specific guidance. Both sessions were then audited in anenvironment where only valid input features were used, for one million model evaluations. Theunguided experiment chose a final model of an ensemble of 75% GBDT/25% CNN and reporteda better overall RMSE score, but when audited, the model performed even worse than thebaseline and took 951 seconds to run one million evaluations (0.95ms/eval). The domain-guidedexperiment produced a 2x128 MLP with a properly constrained feature set which reduced ournetwork link-level RMSE by 49% and completed the audit in 1.4 seconds (0.0014ms/eval).Grounding the agent with explicit deployment constraints ensured that all improvementsremained usable in our application, highlighting that human-in-the-loop domain context is criticalfor this type of automated discovery.We have open sourced the modified harness, along with the archive of the two experimentsessions, with the hope that this will be a valuable resource for other RSE teams whomaintain ML models.