We report the performance of an instrument that employs a torsion balance for probing a non-standard force in the sub-micrometre range. High sensitivity of 1.2 × 10-10 N Hz-1/2 at 1 mHz is achieved by using a torsion balance that has a long torsional period, strong magnetic damping of all vibrational motions and a feedback system that employs an optical lever. In torsion balance experiments, the distance fluctuations during measurements and the accuracy to which the absolute distance is determined are crucial for determining the sensitivity of the balance to a macroscopic force in the sub-micrometre range. We have estimated the root mean square amplitude of the distance fluctuation to be 18 nm by considering the effects due to seismic motions, tilt motions, residual angular fluctuations and thermal fluctuations. We have also estimated the error of the absolute distance to be 13 nm and the statistical error of the force to be 3.4 × 10-12 N by measuring the electrostatic forces. As a result of this systematic study, we have evaluated the sensitivity of the balance to both a non-standard force and to the Casimir force.