Although studies have explored how loneliness varies between rural and urban areas during the COVID-19 pandemic, the results have been inconsistent, and most studies are observational. Therefore, it remains unclear how urban–rural differences affected loneliness in a pandemic. Our study uses nationwide data to clarify this, covering periods before and during the pandemic. We analyze a longitudinal dataset from Hiroshima University’s Household Behavior and Finance Survey, which collected demographic, socioeconomic, and psychological characteristics of Japanese adults in 2020, 2021, and 2022, thus reflecting the COVID-19 pandemic timeline. The results show that approximately 50% of those surveyed experienced long-term loneliness, while about 6.5% developed loneliness during the pandemic. Although our weighted logit regression models showed few differences in loneliness during the pandemic between urban and rural areas, socioeconomic changes, such as beginning to live alone, leaving full-time employment, and decreased financial satisfaction, were identified as high-risk factors for loneliness, and their impact varied between rural and urban areas. Our results reflect that rural–urban differences have an effect on people’s loneliness during a pandemic but need to be considered together with socioeconomic changes. This knowledge can aid governments and healthcare providers in identifying those most at risk of loneliness within urban–rural regional boundaries.