Conservation of traditional olive groves through effective monitoring of their health state is crucial both at a tree and at a population level. In this study, we introduce a comprehensive methodological framework for estimating the traditional olive grove health state, by considering the fundamental phenotypic, spectral, and thermal traits of the olive trees. We obtained phenotypic information from olive trees on the Greek island of Lesvos by combining this with in situ measurement of spectral reflectance and thermal indices to investigate the effect of the olive tree traits on productivity, the presence of the olive leaf spot disease (OLS), and olive tree classification based on their health state. In this context, we identified a suite of important features, derived from linear and logistic regression models, which can explain productivity and accurately evaluate infected and noninfected trees. The results indicated that either specific traits or combinations of them are statistically significant predictors of productivity, while the occurrence of OLS symptoms can be identified by both the olives’ vitality traits and by the thermal variables. Finally, the classification of olive trees into different health states possibly offers significant information to explain traditional olive grove dynamics for their sustainable management.