Brlek et al. conjectured in 2008 that any fixed point of a primitive morphism with finite palindromic defect is either periodic or its palindromic defect is zero. Bucci and Vaslet disproved this conjecture in 2012 by a counterexample over ternary alphabet. We prove that the conjecture is valid on binary alphabet. We also describe a class of morphisms over multiliteral alphabet for which the conjecture still holds. The proof is based on properties of extension graphs.