In this paper, the energy loss of the green building is optimized based on the thickness and lay-up of the Nano-insulation. As different thicknesses and lay-up of the Nano-insulation have a direct effect on energy consumption of the green building with 1590 square meters, especially with nanomaterial, the machine learning models are employed to represent a new model of the thermal conductivity of the proposed advanced insulation with the precision above 99%. The machine learning models are employed to classify and model the behavior of the heat transfer in the green building due to the complex behavior of the thermal conductivity in the green building. Therefore, 110 data for modeling 20 types of lay-up with 6 different thicknesses are prepared by the machine learning models including Support Vector Machine (SVM), Gaussian Process Regression (GPR), and decision tree. Based on the data analysis and statistical data, thermal conductivity modeling with a decision tree represents the best performance and fitted model. The multi-Disciplinary Optimizing method (MDO) under energy consumption constraint, economical consideration, and environmental effects on insulation properties is performed to enhance the energy efficiency of the green building. The calculated results with the Degree-Day approach reveal that the amount of energy saving for green buildings with Nano insulation is about 40% higher than common insulation in common types of insulations. The proposed insulation characteristics regarding the value of Present Worth Function (PWF) and economic aspects cause energy saving per unit area and decreasing in CO2 emission between 290 kg/m3 to 293 kg/m3 depending on weather conditions, insulation thickness, and lay-up.