Machine Learning resource consumption has exploded due to the Generative AI boom, with many companies racing to develop unique capabilities around foundational models to become the next big thing. One unfortunate by-product is the overconsumption of GPUs leading to stock-outs on many clouds as well as contributing to higher carbon footprint across on-premise and multi-cloud. This session is to show a way which complies with Green Software patterns of predictive power consumption to train models in a sustainable and cost-optimized way while leveraging Kubernetes for orchestration, scalability and high availability while avoiding unnecessary retraining and massive amounts of experiments.