With currently more than nine million hectares, olive tree cultivation has spread worldwide, table olives and olive oil as the main products. Moreover, a number of by-products and residues derived from both tree cultivation and the process of industrial olive oil production, most having no practical applications, are obtained yearly. This paper reviews the research regarding these by-products, namely biomass from olive tree pruning, olive stones, olive pomace and wastewaters obtained from the process of olive oil production. Furthermore, a wide range of compounds has been identified and can be produced using a broad definition of the term biorefinery based on olive tree biomass. As an example, this paper reviews ethanol production as one of the main proposed applications, as well as research on other value-added products. Finally, this paper also assesses recent technological advances, future perspectives and challenges in each stage of the process.