GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shifts
Abstract
Geometric deep learning (GDL) has gained significant attention in scientific fields, for its proficiency in modeling data with intricate geometric structures. However, very few works have delved into its capability of tackling the distribution shift problem, a prevalent challenge in many applications. To bridge this gap, we propose GeSS, a comprehensive benchmark designed for evaluating the performance of GDL models in scientific scenarios with distribution shifts. Our evaluation datasets cover diverse scientific domains from particle physics, materials science to biochemistry, and encapsulate a broad spectrum of distribution shifts including conditional, covariate, and concept shifts. Furthermore, we study three levels of information access from the out-of-distribution (OOD) test data, including no OOD information, only unlabeled OOD data, and OOD data with a few labels. Overall, our benchmark results in 30 different experiment settings, and evaluates 3 GDL backbones and 11 learning algorithms in each setting. A thorough analysis of the evaluation results is provided, poised to illuminate insights for GDL researchers and domain practitioners who are to use GDL in their applications.
AI-Generated Overview
Here’s a brief overview of the extracted text presented in the form of bullet points:
-
Research Focus: The study aims to bridge the gap in understanding how Geometric Deep Learning (GDL) models perform under the common challenge of distribution shifts in various scientific applications.
-
Methodology: The authors propose GeSS, a benchmark that evaluates GDL models across diverse scientific fields (particle physics, materials science, biochemistry) and distribution shift types (conditional, covariate, concept shifts) using multiple experimental settings.
-
Results: The experiments reveal that no single method performs optimally across all distribution shift scenarios. The effectiveness of various algorithms depends on the nature of the shift and the availability of out-of-distribution (OOD) information.
-
Key Contributions: This work provides a comprehensive benchmark (GeSS) and systematic evaluation methodologies for GDL models in scientific contexts, with findings that illustrate how shifts in distribution affect model performance.
-
Significance: The findings contribute to a better understanding of how GDL models can be adapted and improved in scientific applications facing distribution shifts, offering insights to both researchers and practitioners.
-
Broader Applications: The developed datasets and benchmarks can facilitate further advancements in GDL algorithms, potentially influencing various scientific domains that utilize complex geometric data structures.