This study proposes a two-level optimization model system for vehicle control and signal timing at isolated signal intersections under the mixed traffic flow environment composed of intelligent connected autonomous vehicles (CAVs) and connected human-driven vehicles (CHVs), to minimize the energy consumption and vehicle delay at intersections. The proposed two-layer optimization model is composed of a two-layer vehicle trajectory control model and a fuzzy control signal timing optimization model. The two-layer vehicle trajectory control model includes a signal-oriented vehicle trajectory control model and a car-following oriented vehicle trajectory control model. The former calculates expected acceleration and speed commands at each time step according to the coming signal information, to help vehicle pass the closest signal intersection without stopping during the green light interval; the latter uses the variable headway (VTH) strategy to follow the preceding vehicle by maintaining a safe distance. A microscopic simulator based on SUMO is developed to test the performance of the proposed optimization algorithm. In the simulation experiment, with the driving characteristics of CHV drivers considered, the results show that our model performs well under a CAV penetration rate of 30%–60% and under small or moderate levels of traffic flow. The average waiting time of vehicles is reduced by about 25% compared with the uncontrolled scheme. Under the condition of penetration rate of 60%, the average energy consumption of vehicles in the proposed model is 17.56% lower than that of the uncontrolled scheme. In addition, the proposed model reduces by 21.94% compared with the scheme of only controlling vehicles. When the traffic flow is at a low or medium level, the average energy consumption and waiting time of vehicles are reduced by nearly 35% with the proposed model.