Climate change continues to pose threats to fisheries and fishery-dependent communities globally. Vulnerability to climate change is a function of exposure, sensitivity, and adaptive capacity. Sensitivity is largely determined by the differences in socio-economic conditions among communities, and conflicts over resources often exacerbate this sensitivity. This study aims to understand factors affecting the sensitivity to climate change. The objectives are twofold: first, to develop indicators affecting sensitivity and to determine how they affect sensitivity, second, to compare sensitivity of two small-scale fishing groups (fishing camps and fishing villages). The study used twelve indicators, which are categorized into two; the community characteristics and assets, and threats and conflicts. Results show that fishing camps are less sensitive to climate change than fishing villages since they have more varied livelihood sources, such as crop farming. This allows for more sources of income. Both groups experience conflict with other lake users and wildlife attacks, which amplify their sensitivity through the reduction of fishing grounds and the damaging of fishing gear. It also shows that both climate and non-climate factors affect sensitivity, and understanding this can help to increase adaptive capacity. The findings allow for formulation of policy recommendations to help strengthen the livelihoods of small-scale fisheries.