Abstract Whole sky spectral radiance distribution measurements are difficult and expensive to obtain, yet important for real-time applications of radiative transfer, building performance, physically based rendering, and photovoltaic panel alignment. This work presents a validated machine learning approach to predicting spectral radiance distributions (350–1780 nm) across the entire hemispherical sky, using regression models trained on high dynamic range (HDR) imagery and spectroradiometer measurements. First, we present and evaluate measured, engineered, and computed machine learning features used to train regression models. Next, we perform experiments comparing regular and HDR imagery, sky sample color models, and spectral resolution. Finally, we present a tool that reconstructs a spectral radiance distribution for every single point of a hemispherical clear sky image given only a photograph of the sky and its capture timestamp. We recommend this tool for building performance and spectral rendering pipelines. The spectral radiance of 81 sample points per test sky is estimated to within 7.5% RMSD overall at 1 nm resolution. Spectral radiance distributions are validated against libRadtran and spectroradiometer measurements. Our entire sky dataset and processing software is open source and freely available on our project website.