Speaker
Description
The performance of deep learning models is sensitive to the settings of various hyperparameters, most importantly learning rate and initialization scale. Hyperparameter tuning becomes expensive at scale, motivating the modern paradigm of hyperparameter transfer, a strategy for inferring near-optimal hyperparameters in larger models from their smaller counterparts, which are cheaper to optimize. Though hyperparameter transfer is increasingly common in, e.g., large language models trained on textual data, applications to science remain relatively sparse. Furthermore, little progress has been made towards hyperparameter transfer in graph-based algorithms, which are particularly important for modeling relational data common in scientific datasets. We present systematic prescriptions for hyperparameter transfer in Graph Neural Networks (GNNs) and set-based models generalized to higher-order interactions, which we dub Coupled Particle-Edge Networks (CPENs). We derive and empirically validate learning rate transfer parameterizations for training via SGD, Adam, and AdamW, with applications spanning key benchmarks and scientific datasets including jet identification in particle physics and stellar stream identification in astronomical datasets. Our results indicate that proxy-tuning a small model leads to robust performance at scale, paving the way for applying large GNNs and CPENs to scientific data.