Several tools have been developed to compare the environmental impact of textiles. The most widely used are Higg Materials Sustainability Index (MSI) and MADE-BY Fiber Benchmark. They use data from production to evaluate the environmental impacts of textiles differentiated by fiber type. The use phase is excluded from both tools. This article discusses whether there is evidence that the use of textiles differs systematically between different fiber types and examines the consequences of comparing the environmental impacts of clothing based on differences in production of fibers alone without including differences in their use. The empirical material in this paper is based on analysis of rating tools and a literature review on clothing use. It shows that fiber content contributes to the way consumers take care of and use their clothing. When use is omitted, major environmental problems associated with this stage, such as spread of microplastics, are also excluded. This one-sided focus on material production impacts also excludes the importance of product lifespans, quality, and functionality. The consequence is that short-lived disposable products are equated with durable products. Comparing dissimilar garments will not help consumers to make choices that will reduce the environmental burden of clothing. We need an informed discussion on how to use all materials in the most environmentally sustainable way possible.