When it comes to pollution, we do not usually think about the clothes we wear, but the clothing industry is really endangering our planet. The market economy has transferred the decision-making power of the garment industry from enterprises to consumers. To make the fashion industry sustainable, in addition to technological innovation, it is also necessary to conduct research on the service objects of the industry. Consumer clothing preference research is an important part of the sustainable development of the clothing industry, and it will also have an impact on environmental and design sustainability. Hence, a psychophysical experiment based on solid color shirts is carried out to analyze people’s perceptions and preferences concerning fashion colors, including the aesthetic differences and similarities between males and females, and establish a hierarchical feed-forward model of color preferences relating to solid color shirts. Firstly, 480 colors of solid shirts from different clothing brands were collected, and the mean shift clustering algorithm was used to classify them into 19 clusters in the CIELAB color space. Secondly, another 22 solid colors, combined with the 19 colors of the cluster centers, formed a solid color scheme. Thirdly, 41 solid male and female shirts and fabrics were simulated as stimuli in three dimensions, and they were presented on a calibrated computer display. The simulations were assessed by 34 observers (consisting of 17 males and 17 females) in terms of 11 semantic scales, including cold/warm, heavy/light, passive/active, dirty/clean, tense/relaxed, plain/gaudy, traditional/modern, masculine/feminine, slim-look/fat-look, hard-to-match/easy-to-match, and dislike/tike. The experimental results demonstrated that the hard-to-match/easy-to-match response was found to be highly correlated with dislike/like. Furthermore, the response of the females concerning hard-to-match/easy-to-match had a strong correlation with two adjective pairs (plain/gaudy and slim-look/fat-look), while that of the males also had a strong correlation with two adjective pairs (plain/ gaudy and masculine/feminine). Finally, a hierarchical feed-forward model of aesthetic perception for solid color shirts was established to predict the shirt preference degree. These findings could be used to develop a more robust and comprehensive theory of fashion color preferences and provide a reference for the design of solid color shirts. A more comprehensive color preference theory is not only an effective tool to solve the problem of pollution in the clothing industry, but also an important theoretical basis for the “sustainable design” of clothing, which is of great significance to the sustainable development of the clothing industry.