Abstract Numerous occupant behavior models, typically stochastic ones, have been built to quantify the effect of occupants on building energy simulation, which however are supposed to go through a rigorous validation, or evaluation process to claim that they are qualified to be used in the application. Current contributions relevant with occupant behavior modeling in literature have been resorting to statistical metrics rather than those orienting on their application, causing the disputation in interpreting the accuracy of the occupant behavior models when they are actually used in simulation. Furthermore, the stochasticity of occupant behavior models necessitates statistical evaluation of the model accuracy, instead of the comparison between actual data and a single-run simulated data. This paper proposes a methodological framework for model evaluation from an application-oriented perspective, with lighting behavior as a case study. The metrics for evaluating lighting behavior models are identified according to different application scenarios, while the comparison between the measurement and simulation is done by introducing the statistical hypothesis testing. Analogous analyses on other behaviors can be conducted in the future, which altogether would foster more rigor and application significance in occupant behavior model evaluation.