Tree-ring chronologies are widely used to reconstruct high-to low-frequency variations in growing season temperatures over centuries to millennia. The relevance of these timeseries in large-scale climate reconstructions is often determined by the strength of their correlation against instrumental temperature data. However, this single criterion ignores several important quantitative and qualitative characteristics of tree-ring chronologies. Those characteristics are (i) data homogeneity, (ii) sample replication, (iii) growth coherence, (iv) chronology development, and (v) climate signal including the correlation with instrumental data. Based on these 5 characteristics, a reconstruction-scoring scheme is proposed and applied to 39 published, millennial-length temperature reconstructions from Asia, Europe, North America, and the Southern Hemisphere. Results reveal no reconstruction scores highest in every category and each has their own strengths and weaknesses. Reconstructions that perform better overall include N-Scan and Finland from Europe, E-Canada from North America, Yamal and Dzhelo from Asia. Reconstructions performing less well include W-Himalaya and Karakorum from Asia, Tatra and S-Finland from Europe, and Great Basin from North America. By providing a comprehensive set of criteria to evaluate tree-ring chronologies we hope to improve the development of large-scale temperature reconstructions spanning the past millennium. All reconstructions and their corresponding scores are provided at www.blogs.uni-mainz.de/fb09climatology.