The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop: SC22 SuperCompCloud: 6th International Workshop on Interoperability of Supercomputing and Cloud Technologies
Authors: Simon Caton (University College Dublin); Matt Baughman (University of Chicago); Christian Haas (Vienna University of Economics and Business Administration); Ryan Chard and Ian Foster (Argonne National Laboratory (ANL)); and Kyle Chard (University of Chicago, Argonne National Laboratory (ANL))
Abstract: Since 2009, Amazon has offered its unused compute capacity as AWS Spot Instances. For the first eight years of spot, pure market dynamics and high pricing variability created an ideal environment for time-series prediction. Following a pricing-scheme change in 2017, this extreme variability was removed as pricing is artificially smoothed for the end-user, therefore making it significantly easier to accurately predict prices. Nevertheless, the literature demonstrates ongoing efforts to accurately predict spot prices. To show prediction in the modern spot market is unnecessary, we train nearly 2.2 million ARIMA models on new and old data to demonstrate an order of magnitude improvement in accuracy for models trained on new data. Further, we show this new ease of price prediction makes spot instances ideal for large-scale, cost-aware cloud computing, as cost estimation is now trivial. Accordingly, we demonstrate that even naive prediction approaches waste less than $360 for 1,000,000 core hours.