The condition of railway infrastructure, such as rails, ballasts and sleepers, should always be monitored and analyzed to ensure ride safety and quality for both passengers and freight. It is hard to assess the condition of railway infrastructure due to the existence of various components. The existing condition assessment models are mostly limited to only assess track geometry conditions and structural condition of the railway infrastructure. Therefore, the present research develops a defect-based structural and geometrical condition model of railway infrastructure. The defects of each component are identified and examined through literature and experts in the field. Two main inputs are used to develop the model: (1) the relative weight of importance for components, defects and their categories and (2) defects severities. To obtain the relative weights, the analytic network process (ANP) technique is adopted. Fuzzy logic is used to unify all the different defect criteria and to interpret the linguistic condition assessment grading scale to a numerical score. Hence, the technique for order preference by similarity to ideal Solution (TOPSIS) is used to integrate both weights and severities to determine the railway infrastructure condition. The developed model gives a detailed condition of the railway infrastructure by representing a three-level condition state, for defect categories, components and an overall railway infrastructure. The developed model is implemented to five case studies from Ontario, Canada. The developed model is validated by comparing its results with the real case studies results, which shows similar results, indicating the robustness of the developed model. This model helps in minimizing the inaccuracy of railway condition assessment through the application of severity, uncertainty mitigation and robust aggregation