Existence of nontrivial solution for Schrödinger-Poisson systems with indefinite steep potential well

In this paper, we study a class of nonlinear Schrödinger-Poisson systems with indefinite steep potential well: -Δu+Vλ(x)u+K(x)ϕu=|u|p-2uinR3,-Δϕ=Kxu2inR3, where 3<p<4, Vλ(x)=λa(x)+b(x) with λ>0 and K(x)≥0 for all x∈R3. We require that a∈C(R3) is nonnegative and has a potential well Ωa, namely a(x)≡0 for x∈Ωa and a(x)>0 for x∈R3\Ωa¯. Unlike most other papers on this problem, we allow that b∈C(R3) is unbounded below and sign-changing. By introducing some new hypotheses on the potentials and applying the method of penalized functions, we obtain the existence of nontrivial solutions for λ sufficiently large. Furthermore, the concentration behavior of the nontrivial solution is also described as λ→∞.