Comparing the relative quality of NISQ devices is difficult. Algorithms showing a quantum advantage are often tailored precisely to what a particular NISQ does well. We present a new algorithm for evaluating NISQs using quadratic nonresidues. We prove quantum computers can find quadratic nonresidues in deterministic polynomial time, whereas the classical version of this problem remains unsolved after hundreds of years. Using a restrictive computational rule set for finding quadratic nonresidues, we can compare the NISQ success rate with what is possible for a classical computer to accomplish under the same rules. A success rate greater than 75% provides evidence of quantum advantage. We present the results of current NISQ devices running this test.