Sustainable nitrogen (N) management in agriculture is one of the most important issues affecting the environmental performance of modern agriculture. It is actually well perceived that coordinated efforts and holistic approaches are required to regulate N use by farmers. The purpose of this study was to provide an initial examination of stakeholders’ views in Japan regarding N use in agriculture and challenges to increase its sustainability. The analysis was based on a questionnaire study of five types of stakeholders (farmers, advisors, researchers, suppliers, policy makers). By means of multivariate analysis techniques it was revealed that consensus was lacking either in the acknowledgment of the causes and effects of unsustainable N management or in the challenges that need to be addressed. N losses from farms and the effects of N use were perceived but not conceived equally by all stakeholders. Organic farming and mandatory measures were the most controversial challenges, while those involving awareness, training and advisory were the most popular. This study cannot provide safe conclusions that can be generalized in the Japanese context, but it indicates domains where further research is required and orientations for future policy design towards more sustainable N use.