Energy use in the building is responsible for one-third of total carbon dioxide (CO2) emissions globally. Nearly half of the energy loss occurs through the building envelope due to heat transfer to/for the surroundings. Therefore, there is a need to design an optimum building envelope to reduce energy use in buildings that depend on several parameters. This study aims to review different building parameters and provide a conceptual framework to optimize the building envelope. In total, 260 papers were reviewed, and the building envelope design consideration was categorized into: 1) Design Parameters (design and geometry), 2) environmental conditions (indoor and outdoor) and 3) performance criteria (energy, environment, economic, comfort). Energy use and CO2-emission in buildings increase with high thermal conductivity, low thermal mass, and low solar absorption of its envelope. Geometrically, building orientation impacts energy use more than the building shape factor. Changing set point temperature according to surrounding conditions has reduced energy use and CO2-emission by 30% and 56%, respectively. However, indoor air quality, velocity, and occupancy have meagerly affected building energy use. Energy and emission optimization criteria are directly related, but the emission-based optimized envelope is thicker than the energy one. Other criteria such as economy and comfort (thermal and visual) are inversely proportional to the energy-efficient building envelope. Based on the comprehensive review, this study proposed a conceptual framework to design a sustainable building envelope that includes life cycle assessment, occupant's satisfaction, and social benefits. Several future research recommendations were made, including 1) the use of switchable reflective materials to minimize heat transfer, 2) dynamic insulation material to control insulation value as needed, and 3) smart windows with tunable optical properties.