Owing to their high luminous efficiency and tunable emission in both red light and far-red light regions, Mn4+ ion-activated phosphors have appealed significant interest in photoelectric and energy conversion devices such as white light emitting diode (W-LED), plant cultivation LED, and temperature thermometer. Up to now, Mn4+ has been widely introduced into the lattices of various inorganic hosts for brightly red-emitting phosphors. However, how to correlate the structure-activity relationship between host framework, luminescence property, and photoelectric device is urgently demanded. In this review, we thoroughly summarize the recent advances of Mn4+ doped phosphors. Meanwhile, several strategies like co-doping and defect passivation for improving Mn4+ emission are also discussed. Most importantly, the relationship between the protocols for tailoring the structures of Mn4+ doped phosphors, increased luminescence performance, and the targeted devices with efficient photoelectric and energy conversion efficiency is deeply correlated. Finally, the challenges and perspectives of Mn4+ doped phosphors for practical applications are anticipated. We cordially anticipate that this review can deliver a deep comprehension of not only Mn4+ luminescence mechanism but also the crystal structure tailoring strategy of phosphors, so as to spur innovative thoughts in designing advanced phosphors and deepening the applications.