The reliability of the transmission grid is challenged by the integration of intermittent renewable energy sources into the grid. For model-based reliability studies, it is important to have suitable models available of renewable energy sources like wind and solar power. In this study, we investigate to what extent the power output of wind farms can be modeled with discrete Hidden Markov Models (HMMs). The parameters of the HMMs are inferred from measurement data from multiple turbines in a wind farm. We use these models both for individual turbine output and for total aggregated power output of multiple turbines. When modeling individual turbine output, the hidden process in the HMM is instrumental in capturing the dependencies between the output of the different turbines. It is important to account for these dependencies in order to correctly capture the upper quantiles (90%, 95%, 99%) of the distribution of the wind farm aggregated power output. We show that despite their simple structure, HMMs are able to reproduce important features of the power output of wind farms. This opens up possibilities to model and analyze these features with methods and techniques stemming from the field of Markov models and stochastic processes.