Understanding the drivers of deforestation and forest degradation and the agents of such drivers is important for introducing appropriate policy interventions. Here, we identified drivers and agents of drivers through the analysis of local perceptions using questionnaire surveys, focus group discussions, and field observations. The Likert scale technique was employed for designing the questionnaire with scores ranging from 1 (strongly disagree) to 5 (strongly agree). We found nine direct drivers of forest deforestation and forest degradation, namely illegal logging (4.53 ± 0.60, ± is for standard deviation), commercial wood production (4.20 ± 0.71), land clearing for commercial agriculture (4.19 ± 1.15), charcoal production (3.60 ± 1.12), land clearing for subsistence agriculture (3.54 ± 0.75), new settlement and land migration (3.43 ± 0.81), natural disasters (3.31 ± 0.96), human-induced forest fires (3.25 ± 0.96), and fuelwood for domestic consumption (3.21 ± 0.77). We also found four main indirect drivers, namely lack of law enforcement, demand for timber, land tenure right, and population growth. Our analysis indicates that wood furniture makers, medium and large-scale agricultural investors, charcoal makers, land migrants, firewood collectors, and subsistent farmers were the agents of these drivers. Through focus group discussions, 12 activities were agreed upon and could be introduced to reduce these drivers. In addition to enforcing the laws, creating income-generating opportunities for locals along with the provision of environmental education could ensure long-term reduction of these drivers. The REDD+ project could be an option for creating local income opportunities, while reducing deforestation and forest degradation.