A Quantitative Occam's Razor

Interpreting entropy as a prior probability suggests a universal but “purely empirical” measure of “goodness of fit.” This allows statistical techniques to be used in situations where the correct theory- and not just its parameters-is still unknown. As developed illustratively for least-squares nonlinear regression, the measure proves to be a transformation of the R ² statistic. Unlike the latter, however, it diminishes rapidly as the number of fitting parameters increases.