Aggregate composition has a pivotal role in ensuring the quality of pavement materials. The use of waste materials to replace the aggregate composition of asphalt pavement leads to green, sustainable, and environmentally friendly construction, which ultimately preserves nature by reducing the need to harvest materials from natural sources. Using the Marshall mix design, the main objective of this paper is to investigate the effects of waste palm oil clinker (POC) as fine aggregates replacement on the properties of stone mastic asphalt (SMA) mixture. Six groups of asphalt mixtures were prepared using different percentages of palm oil clinker content (0%, 20%, 40%, 60%, 80%, and 100%). To determine the Marshall properties and select the optimum binder content, asphalt mixture samples with different percentages of asphalt binder content (5.0%, 5.5%, 6.0%, 6.5%, and 7.0%) were prepared for each group. The results showed that the palm oil clinker was appropriate for use as a fine aggregate replacement up to 100% in SMA mixture and could satisfy the mix design requirements in terms of Marshall stability, flow, quotient, and volumetric properties. However, the percentage of palm oil clinker replacement as fine aggregate has merely influenced the optimum binder content. Furthermore, there were improvements in the drain down, resilient modulus and indirect tensile fatigue performances of the SMA mixture. In conclusion, the use of POC as fine aggregates replacement in SMA mixture indicates a good potential to be commercialized in flexible pavement construction.