The problem to find the minimum number of controls to relieve overloads with ac power flow constraints (say, according to a state estimation) is formulated as mixed integer nonlinear programming (MINLP). To avoid the prohibitive computational time required by the classical MINLP approaches, we propose a three-stage method, i.e., screening, ranking, and detailed analysis, to tackle this problem. To make the proposed method applicable in online mode, a feasibility pump-based scheme is incorporated into the three-stage to solve the problem. It is shown that the proposed method can achieve computational tractability for large-scale systems. Numerical studies on the 300-bus system and the 2869-bus system reveal that the proposed method is about 1000 times faster than the branch and bound method, and the required controls for both methods are comparable. Simulation studies indicate the proposed method can effectively alleviate power system overloads in the online mode.