In this paper, we consider the initial value problem for both of the defocusing and focusing Kundu-Eckhaus (KE) equation with non-zero boundary conditions (NZBCs) at infinity by inverse scattering transform method. The solutions of the KE equation with NZBCs can be reconstructed in terms of the solution of an associated $2 \times 2$ matrix Riemann-Hilbert problem (RHP). In our formulation, both the direct and the inverse problems are posed in terms of a suitable uniformization variable which allows us to develop the IST on the standard complex plane instead of a two-sheeted Riemann surface or the cut plane with discontinuities along the cuts. Furthermore, on the one hand, we obtain the N-soliton solutions with simple pole of the defocusing and focusing KE equation with the NZBCs, especially, the explicit one-soliton solutions are given in details. And we prove that the scattering data $a(\zeta)$ of the defocusing KE equation can only have simple zeros. On the other hand, we also obtain the soliton solutions with double pole of the focusing KE equation with NZBCs. And we show that the double pole solutions can be viewed as some proper limit of the two simple pole soliton solutions. Some dynamical behaviors and typical collisions of the soliton solutions of both of the defocusing and focusing KE equation are shown graphically.