In this work, we present a Gauss-Newton based quantum algorithm (GNQA) for combinatorial optimization problems that, under optimal conditions, rapidly converges towards one of the optimal solutions without being trapped in local minima or plateaus. Quantum optimization algorithms have been explored for decades, but more recent investigations have been on variational quantum algorithms, which often suffer from the aforementioned problems. Our approach mitigates those by employing a tensor product state that accurately represents the optimal solution, and an appropriate function for the Hamiltonian, containing all the combinations of binary variables. Numerical experiments presented here demonstrate the effectiveness of our approach, and they show that GNQA outperforms other optimization methods in both convergence properties and accuracy for all problems considered here. Finally, we briefly discuss the potential impact of the approach to other problems, including those in quantum chemistry and higher order binary optimization.