Koide's mass formula is an empirical relation among the charged lepton masses which holds with a striking precision. We present a model of charged lepton sector within an effective field theory with U(3) × SU(2) family gauge symmetry, which predicts Koide's formula within the present experimental accuracy. Radiative corrections as well as other corrections to Koide's mass formula have been taken into account. We adopt a known mechanism, through which the charged lepton spectrum is determined by the vacuum expectation value of a 9-component scalar field Φ. On the basis of this mechanism, we implement the following mechanisms into our model: (1) The radiative correction induced by family gauge interaction cancels the QED radiative correction to Koide's mass formula, assuming a scenario in which the U(3) family gauge symmetry and SU(2)L weak gauge symmetry are unified at 102-103 TeV scale; (2) A simple potential of Φ invariant under U(3) × SU(2) leads to a realistic charged lepton spectrum, consistent with the experimental values, assuming that Koide's formula is protected; (3) Koide's formula is stabilized by embedding U(3) × SU(2) symmetry in a larger symmetry group. Formally fine tuning of parameters in the model is circumvented (apart from two exceptions) by appropriately connecting the charged lepton spectrum to the boundary (initial) conditions of the model at the cut-off scale. We also disucss some phenomenological implications.