The mass production of graphene oxide (GO) unavoidably elevates the chance of human exposure, as well as the possibility of release into the environment with high stability, raising public concern as to its potential toxicological risks and the implications for humans and ecosystems. Therefore, a thorough assessment of GO toxicity, including its potential reliance on key physicochemical factors, which is lacking in the literature, is of high significance and importance. In this study, GO toxicity, and its dependence on oxidation level, elemental composition, and size, were comprehensively assessed. A newly established quantitative toxicogenomic-based toxicity testing approach, combined with conventional phenotypic bioassays, were employed. The toxicogenomic assay utilized a GFP-fused yeast reporter library covering key cellular toxicity pathways. The results reveal that, indeed, the elemental composition and size do exert impacts on GO toxicity, while the oxidation level exhibits no significant effects. The UV-treated GO, with significantly higher carbon-carbon groups and carboxyl groups, showed a higher toxicity level, especially in the protein and chemical stress categories. With the decrease in size, the toxicity level of the sonicated GOs tended to increase. It is proposed that the covering and subsequent internalization of GO sheets might be the main mode of action in yeast cells.