As species respond to warming water temperatures, fishers dependent upon such species are being compelled to make choices concerning harvest strategies. Should they “follow fish” to new fishing grounds? Should they change their mix of target species? Should they relocate their operations to new ports? We examined how fishing communities in the Northeast United States —a hotspot of recent warming—have already responded to documented shifts in the distribution and abundance of fluke, red and silver hake. We focused on groundfish trawl communities that historically targeted these species and examined their “at-sea” responses by combining qualitative interviews with quantitative analysis of fishing records and ecological surveys. Three distinct responses emerged: shifting fishing grounds, shifting target species, and shifting port of landing. Our research finds that following the fish is rare and only occurred in one of the assessed communities, the large trawler community of Beaufort, North Carolina. The more common response was a shift in target species and a change in catch composition. However, regulations and markets often constrained the ability to take advantage of a changing mix of species within fishing grounds. Indeed, the overall species diversity in catch has declined among all of our focal communities suggesting that communities have lost the ability to be flexible when it may be most needed as a response to climate change. Additionally, the high value of fluke and the need to land in southern states with higher quota allocations is likely a driver of the changing nature of “community” with increasing vessels landing outside their home port, especially when landing fluke. Our findings suggest that fidelity to historical fishing grounds combined with perceiving environmental change as non-permanent, predispose many fishers to trust in “cyclicality” and return of species over time. However, this strategy may make those communities unable or unwilling to “follow fish” more vulnerable to changes in distribution and abundance due to climate change. Our findings have the potential to directly inform resource management policies as well as more deliberate adaptations by communities themselves as they strive to address the imminent risks of climate change.