We describe an extension of the halo-based galaxy group-finding algorithm. We add freedom to the algorithm in order to more accurately determine which galaxies are central and which are satellites, and to provide unbiased estimates of halo masses. We focus on determination of the galaxy-halo relations for star-forming and quiescent galaxies. The added freedom in the group-finding algorithm is self-calibrated using observations of color-dependent galaxy clustering, as well as measurements of the total satellite luminosity in deep imaging data around stacked samples of spectroscopic central galaxies, L_sat. We test this approach on a series of mocks that vary the galaxy-halo connection, including one mock constructed from UniverseMachine results. Our self-calibrated algorithm shows marked improvement over previous methods in estimating the color-dependent satellite fraction of galaxies. It reduces the error in log M_halo for central galaxies by over a factor of two, to <~0.2 dex. Through the L_sat data, it can quantify differences in the luminosity-to-halo mass relations for star-forming and quiescent galaxies, even for groups with only one spectroscopic member. Whereas previous algorithms cannot constrain the scatter in L_gal at fixed M_halo, the self-calibration technique can provide a robust lower limit to this scatter.