Integrated assessment models that incorporate biodiversity and ecosystem services could be an important tool for improving our understanding of interconnected social-economic-ecological systems, and for analyzing how policy alternatives can shift future trajectories towards more sustainable development. Despite recent scientific and technological advances, key gaps remain in the scientific community's ability to deliver information to decision-makers at the pace and scale needed to address sustainability challenges. We identify five research frontiers for integrated social-economic-ecological modeling (primarily focused on terrestrial systems) to incorporate biodiversity and ecosystem services: 1) downscaling impacts of direct and indirect drivers on ecosystems; 2) incorporating feedbacks in ecosystems; 3) linking ecological impacts to human well-being, 4) disaggregating outcomes for distributional equity considerations, and 5) incorporating dynamic feedbacks of ecosystem services on the social-economic system. We discuss progress and challenges along each of these five frontiers and the science-policy linkages needed to move new research and information into action.