This study aims to identify the influence of the socioeconomic attributes and environmental contexts of citizens’ residential areas on the access and use of green areas during the COVID-19 pandemic. The results can aid policymaking and facilitate the safe and unrestricted use of green areas during the pandemic. The access and use of green areas were analyzed using a survey conducted after the official COVID-19 emergency in Japan (16 April to 14 May, 2020). Visits to green areas during the pandemic have gained salience globally from multiple perspectives: health, planning, social justice, and equity. The results of this study demonstrated that socioeconomic factors influenced the frequency of visiting green areas. The factors further influenced the use of the three categories of green areas (parks, agricultural lands, and gardens). Environmental contexts, including the land use patterns in residential areas, also influenced the use of specific types of green areas. Thus, policies need to further facilitate visits to green areas by reflecting the socioeconomic attributes of residents and their households, including income, number of children, gender, and age, incorporating those who have less access and considering the spread of COVID-19 locally. Furthermore, policies for the use of specific green areas, including parks, agricultural lands, and gardens, need to take cognizance of the residents’ environmental contexts. Management of specific green areas, like agricultural lands, is required, and residents should be provided with opportunities to use these areas with measures to avoid infection.