Standing waves for linearly coupled Schrödinger equations with critical exponent

Ann. I. H. Poincaré – AN 31 (2014) 429–447

www.elsevier.com/locate/anihpc

Standing waves for linearly coupled Schrödinger equations with critical exponent

Zhijie Chen, Wenming Zou

Department of Mathematical Sciences, Tsinghua University, Beijing 100084, China Received 11 November 2012; received in revised form 30 April 2013; accepted 30 April 2013

Available online 7 May 2013

Abstract

We study the following linearly coupled Schrödinger equations:

⎧⎪

⎨

⎪⎩

−ε²u+a(x)u=u^p+λv, x∈R^N,

−ε²v+b(x)v=v^2∗−1+λu, x∈R^N, u, v >0 inR^N, u(x), v(x)→0 as|x| → ∞,

whereN3, 2^∗=_N^2N₋₂, 1< p <2^∗−1, anda(x), b(x)are positive continuous potentials which are both bounded away from 0.

Under some assumptions ona(x)andλ >0, we obtain positive solutions of the coupled system for sufficiently smallε >0, which have concentration phenomenon asε→0. It is interesting that we do not need any further assumptions onb(x).

©2013 Elsevier Masson SAS. All rights reserved.

1. Introduction

In this paper we study standing waves for the following system of time-dependent nonlinear Schrödinger equations

⎧⎪

⎪⎪

⎪⎪

⎪⎨

⎪⎪

⎪⎪

⎪⎪

⎩

−ih¯∂ψ1

∂t − ¯h²

2 ψ₁+a(x)ψ₁= |ψ₁|^p−1ψ₁+λψ₂, x∈R^N, t >0,

−ih¯∂ψ₂

∂t − ¯h²

2 ψ₂+b(x)ψ₂= |ψ₂|^2∗−2ψ₂+λψ₁, x∈R^N, t >0, ψ_j=ψ_j(x, t)∈C, j=1,2,

ψ_j(x, t)→0, as|x| → +∞, t >0, j=1,2,

(1.1)

whereidenotes the imaginary unit,h¯ is the Plank constant, N3, 1< p <2^∗−1 and 2^∗=_N^2N₋₂ is the Sobolev critical exponent.

Nonlinear Schrödinger equations (NLS) have been broadly investigated in many aspects. In particular, there has been an ever-growing interest in the study of standing wave solutions to NLS starting from the cerebrated works

✩ Supported by NSFC (11025106, 11271386).

* Corresponding author.

E-mail addresses:[email protected](Z. Chen),[email protected](W. Zou).

0294-1449/$ – see front matter ©2013 Elsevier Masson SAS. All rights reserved.

http://dx.doi.org/10.1016/j.anihpc.2013.04.003

[8,14,28]. For system (1.1), a solution of the form(ψ₁(x, t), ψ₂(x, t))=(e^{−iEt /h¯}u(x), e^{−iEt /h¯}v(x))is called a stand- ing wave. Then(ψ₁, ψ₂)is a solution of (1.1) if and only if(u, v)solves the following system

⎧⎪

⎪⎪

⎪⎨

⎪⎪

⎪⎪

⎩

− ¯h² 2 u+

a(x)−E

u= |u|^p−1u+λv, x∈R^N,

− ¯h² 2 v+

b(x)−E

v= |v|^2∗−2v+λu, x∈R^N, u(x), v(x)→0 as|x| → ∞.

(1.2)

In this paper we are concerned with positive solutions for smallh >¯ 0. For sufficiently small h >¯ 0, the standing waves are referred to as semiclassical states. Replacinga(x)−E, b(x)−Ebya(x), b(x)for convenience, we turn to consider the following system of NLS

⎧⎨

⎩

−ε²u+a(x)u=u^p+λv, x∈R^N,

−ε²v+b(x)v=v^2∗−1+λu, x∈R^N, u, v >0 inR^N, u(x), v(x)→0 as|x| → ∞,

(1.3)

whereλ >0, andε >0 is sufficiently small.

The mathematical interest in (1.3) relies on its criticality, due to the fact that 2^∗is the critical exponent. Critical exponent elliptic problems create some difficulties in using variational methods due to the lack of compactness, and have received great interest since the cerebrated work by Brezis and Nirenberg [10]. Before saying more about the background for problems like (1.3), we would like to introduce our main result first. LetC_p+1be the sharp constant of the Sobolev embeddingH¹(R^N) →L^p+1(R^N)

R^N

|∇u|²+ |u|²dxC_p+1

R^N

|u|^p+1dx ²

p+1

,

andSthe sharp constant ofD^1,2(R^N) →L^2∗(R^N)

R^N

|∇u|²dxS

R^N

|u|^2∗dx _2∗²

. (1.4)

Here,D^1,2(R^N):= {u∈L²(R^N): |∇u| ∈L²(R^N)}with the norm uD^1,2:=

R^N

|∇u|²dx 1/2

.

Define μ0:=

2(p+1) N (p−1)S^N2C⁻

p+1 p−1

p+1

(^p_p−1⁺¹−^N₂)⁻¹

. (1.5)

Assume that

(V1) a, b∈C(R^N,R)and there exist some constantsa0>0,b0>0 such that inf

x∈R^Na(x)=a0μ0, inf

x∈R^Nb(x)=b0.

(V₂) There exists a smooth open bounded domainΛ⊂R^N such that

xinf∈Λa(x)μ₀< a₁:= inf

x∈∂Λa(x). (1.6)

Then the main result of this paper is the following

Theorem 1.1.LetN3and assumptions(V1)–(V2)hold. Assume that 0< λ <min

a0b0,

(a1−μ0)b0

. (1.7)

Then there existsε₀>0such that for anyε∈(0, ε0), problem(1.3)has a positive solution(u˜_ε,v˜_ε), which satisfies (i) there exists a maximum pointx˜_εofu˜_ε+ ˜v_ε such thatx˜_ε∈Λ;

(ii) for any such anx˜_ε,(w_1,ε(x), w_2,ε(x)):=(u˜_ε(εx+ ˜x_ε),v˜_ε(εx+ ˜x_ε))converge(up to a subsequence)to a positive ground state solution(w₁(x), w₂(x))of

⎧⎪

⎨

⎪⎩

−u+a(P₀)u=u^p+λv, x∈R^N,

−v+b(P₀)v=v^2∗−1+λu, x∈R^N, u, v >0 inR^N, u, v∈H¹

R^N ,

(1.8)

wherex˜_ε→P₀∈Λasε→0;

(iii) there existc, C >0independent ofε >0such that (u˜_ε+ ˜v_ε)(x)Cexp −c

ε|x− ˜x_ε|

.

Remark 1.1.The constant μ₀, that is defined in (1.5) and appears in assumptions (V₁)–(V2), plays a crucial role in Theorem1.1. As we will see in Lemma2.2, the assumptions inf_x∈RNa(x)μ₀ and infx∈Λa(x)μ₀ are both necessary for Theorem1.1.

Remark 1.2.It is interesting that we only assume inf_x∈R^Nb(x) >0 without any further assumptions on the potential b(x). In contrast, there have been many papers working on other kinds of elliptic systems (see [21,23,24,26] for example), where further assumptions onb(x)seem always to be needed in the literature.

Remark 1.3.It was pointed out in[16, Remark 1.4]that, by Pohozaev Identity, the limit problem (1.8) has no non- trivial solutions ifp=2^∗−1. That is why we assume 1< p <2^∗−1 in this paper.

For the scalar case of (1.3)

−ε²u+a(x)u=u^p, x∈R^N, u >0, (1.9)

where 1< p <^N_N⁺₋²₂, there are many works on the existence of solutions which concentrate and develop spike layers, peaks, around some points inR^N while vanishing elsewhere asε→0. The related results can be seen in[11–13,17, 18,25,27]and the references therein. On the other hand, the following critical problem

−ε²u+a(x)u=f (u)+u^2∗−1, x∈R^N, u >0, (1.10)

has also been well studied, wheref (u)is a subcritical nonlinearity, see[3,29]for example. Remark that in[3],a(x) is assumed to be locally Hölder continuous. Recently, under more general assumptions on botha(x)andf (u)than those in[3], Zhang and the authors[29]proved the same result as in[3].

Meanwhile, there are an increasing interest in studying coupled nonlinear Schrödinger equations in recent years, which is motivated by applications to nonlinear optics and Bose–Einstein condensation (cf.[1,2,19]). For example, the following coupled Schrödinger equations

⎧⎪

⎪⎪

⎨

⎪⎪

⎪⎩

−ε²u+a(x)u=μ₁u³+βuv², x∈R^N,

−ε²v+b(x)v=μ₂v³+βvu², x∈R^N, u >0, v >0 inR^N,

u(x), v(x)→0 as|x| → ∞,

(1.11)

in subcritical case N 3 have been well studied by [21,23,24,26]. Remark that, further assumptions onb(x) are needed in all these works.

Note that the coupling term of (1.11) is nonlinear. Recently, linearly coupled Schrödinger equations of the following type ⎧

⎪⎨

⎪⎩

−u+μu= |u|^p−1u+λv, x∈R^N,

−v+νv= |v|^p−1v+λu, x∈R^N, u, v∈H¹

R^N ,

(1.12)

have been well studied from the cerebrated works by Ambrosetti et al.[4–6]. Systems of this type arise in nonlinear optics (cf.[1]). In the case ofN3,μ=ν=1,p=3 andλ >0 small enough, Ambrosetti, Colorado and Ruiz[5]

proved that (1.12) has multi-bump solitons. When|u|^p−1uand|v|^p−1vare replaced byf (x, u)=(1+c(x))|u|^p−1u and g(x, v)=(1+d(x))|v|^p−1v respectively, system (1.12) has been studied by Ambrosetti [4] with dimension N =1 and Ambrosetti, Cerami and Ruiz[6] with dimension N 2. When |u|^p−1u and|v|^p−1v are replaced by general subcritical nonlinearitiesf (u)andg(v)respectively, (1.12) was studied by the authors[15].

Note that all works mentioned above deal with subcritical case. Recently, the authors[16]studied the ground state solutions of the following system with critical exponent

⎧⎪

⎨

⎪⎩

−u+μu=u^p+λv, x∈R^N,

−v+νv=v^2∗−1+λu, x∈R^N, u, v >0 inR^N, u, v∈H¹

R^N ,

(1.13)

whereμ, ν >0 and 0< λ <√

μν. Note that system (1.13) appears as a limit problem after a suitable rescaling of (1.3), see Theorem 1.1(ii) for example. As far as the semiclassical states related to (1.13) are concerned, we are naturally led to study system (1.3), and this is the goal of this paper.

There are several useful approaches to study semiclassical states. One is the classical Lyapunov–Schmidt reduction method, which cannot be used here, since the uniqueness and nondegeneracy of the ground state solutions of (1.13) are not known. Another one is Byeon and Jeanjean’s variational approach[11], which cannot be used here either, since we have no any further assumptions onb(x). Here, to prove Theorem1.1, we will mainly follow the variational penalization scheme introduced by Del Pino and Felmer[17]. However, we should point out that the compactness is the main difficulty since (1.3) is a critical problem. Therefore, the method of Del Pino and Felmer[17]cannot be used directly and some crucial modifications and new tricks are needed.

The paper is organized as follows. In Section 2, we give a general assumption(V3)ona(x)andb(x)instead of (V₂), and then give a general result. Theorem 1.1will be a direct corollary of this general result. Some comments about(V₂)and(V₃)are also given. We will prove the general result in Section 3.

We give some notations here. Throughout this paper, we denote the norm ofL^p(R^N)by|u|p=(

R^N|u|^pdx)^p1, and the norm ofH¹(R^N)byu =

|∇u|²₂+ |u|²₂. We denote positive constants (possibly different in different places) byc, C, C₀, C₁, . . ., andB(x, r):= {y∈R^N: |x−y|< r}.

2. A general result and some comments Consider the following coefficient problem

⎧⎪

⎨

⎪⎩

−u+a(P )u=u^p+λv, x∈R^N,

−v+b(P )v=v^2∗−1+λu, x∈R^N, u, v >0 inR^N, u, v∈H¹

R^N .

(2.1)

DefineH:=H¹(R^N)×H¹(R^N)with the norm(u, v)²:= u²+ v². It is well known that solutions of (2.1) correspond to the critical points ofC²functionalL_{P ,λ}:H→Rgiven by

L_{P ,λ}(u, v)=1 2

R^N

|∇u|²+a(P )u²+ |∇v|²+b(P )v² dx

− 1 p+1

R^N

|u|^p+1dx− 1 2^∗

R^N

|v|^2∗dx−λ

R^N

uv dx. (2.2)

Define the Nehari manifold NP ,λ:=

(u, v)∈H\ (0,0)

: L_{P ,λ}(u, v)(u, v)=0

, (2.3)

and

mλ(P ):= inf

(u,v)∈NP ,λ

LP ,λ(u, v),

then(u, v)∈NP ,λsatisfyingL_{P ,λ}(u, v)=m_λ(P )will be aground state solutionof (2.1) (see[16]). Let us introduce the following assumption

(V3) There exists a smooth open bounded domainΛ⊂R^Nsuch that m0,λ:= inf

P∈Λm_λ(P ) < inf

P∈∂Λm_λ(P ).

Define Mλ:=

P ∈Λ: mλ(P )=m0,λ

. (2.4)

In the sequel, the subscriptλwill be dropped when there is no possible misunderstanding. Now we can state a general result.

Theorem 2.1.LetN3and assumptions(V₁)hold. Assume that(V₃)holds for someλ∈(0,√

a₀b₀), then there existsε₀>0such that for anyε∈(0, ε0), problem(1.3)has a positive solution(u˜_ε,v˜_ε), which satisfies

(i) there exists a maximum pointx˜εofu˜ε+ ˜vε such thatx˜ε∈Λand

εlim→0dist(x˜_ε,M)=0;

(ii) for any such anx˜_ε,(w_1,ε(x), w_2,ε(x))=(u˜_ε(εx+ ˜x_ε),v˜_ε(εx+ ˜x_ε))converge(up to a subsequence)to a positive ground state solution(w₁(x), w₂(x))of (2.1)withP =P₀, wherex˜_ε→P₀∈Masε→0, and it satisfies that L_P0(w₁, w₂)=m(P₀)=m₀;

(iii) there existc, C >0independent ofε >0such that (u˜_ε+ ˜v_ε)(x)Cexp −c

ε|x− ˜x_ε|

.

Remark 2.1.Assumption(V₃)is an abstract condition, since we cannot write down explicitly the functionm_λ(P ).

Such type of abstract assumptions for system (1.11) can be seen in[21,24]. As we will see in Lemma2.1, for our problem (1.3), we can show that (V2)implies (V3). However, for (1.11), (V2) cannot imply (V3) obviously, and further assumptions onb(x)are needed (see[21,24]).

Recallμ₀in (1.5); the following results have been proved by the authors[16].

Lemma A.(See[16, Lemma 2.4].) Let0< λ <√

a(P )b(P ).

(1) If0< a(P )μ₀, thenm_λ(P ) <_N¹S^N/2. (2) Ifa(P ) > μ₀, then there existsλ_P ∈ [√

(a(P )−μ₀)b(P ),√

a(P )b(P ) ), such that (i) if0< λλ_P, thenm_λ(P )=_N¹S^N/2;

(ii) ifλP< λ <√

a(P )b(P ), thenmλ(P ) <_N¹S^N/2. Lemma B.(See[16, Lemma 2.6].) Let0< λ <√

a(P )b(P ). Ifm_λ(P ) <_N¹S^N/2, then problem(2.1)has a positive ground state(u₀, v₀)∈C²(R^N,R)such thatu₀, v₀are both radially symmetric decreasing.

Theorem A.(See[16, Theorem 1.1].) AssumeN3,1< p <2^∗−1and0< λ <√

a(P )b(P ).

(1) If0< a(P )μ₀, then problem(2.1)has a positive ground state(u, v), such thatL_P(u, v)=m_λ(P )andu, v∈ C²(R^N,R)are both radially symmetric decreasing.

(2) Ifa(P ) > μ₀, then

(i) ifλ < λ_P, thenm_λ(P )is not attained, that is, problem(2.1)has no ground state solutions.

(ii) ifλ > λ_P, then problem(2.1)has a positive ground state (u, v), such thatL_P(u, v)=m_λ(P ) andu, v∈ C²(R^N,R)are both radially symmetric decreasing.

Remark 2.2.The assumption 0< λ <√

a(P )b(P )is needed in Lemmas A and B and Theorem A to guarantee that the Nehari manifoldNP ,λis bounded away from 0, and 0< λ <√

a0b0is assumed in Theorems1.1and2.1such that NP ,λis bounded away from 0 for allP ∈R^N.

Lemma 2.1.Assume(V₂). Then(V₃)holds for anyλsatisfying(1.7). Therefore, Theorem1.1is a direct corollary of Theorem2.1.

Proof. Fix anyλ that satisfies (1.7). By (V₂), there exists some P₁∈Λsuch that a(P₁)μ₀, then we see from Lemma A that

Pinf∈Λmλ(P )mλ(P1) < 1 NS^N/2. On the other hand, for anyP ∈∂Λ, we have

λ <

(a1−μ0)b0

a(P )−μ0

b(P )λP,

then we see from Lemma A thatm_λ(P )=_N¹S^N/2, that is,

Pinf∈∂Λmλ(P )= 1 NS^N/2. Therefore,(V₃)holds. 2

Lemma 2.2.Assumptionsinf_x∈RNa(x)μ₀andinfx∈Λa(x)μ₀in(V₁)–(V2)are both necessary for Theorem1.1.

Proof. Suppose that Theorem1.1holds. Assume by contradiction thata₂:=infx∈Λa(x) > μ₀. Fix anyλthat satisfies 0< λ <min

a₀b₀,

(a₁−μ₀)b₀,

(a₂−μ₀)b₀ .

Theorem1.1(ii) says that there exists someP0∈Λsuch that (2.1) has a ground state solution whenP =P0. On the other hand, since

λ <

(a₂−μ₀)b₀

a(P₀)−μ₀

b(P₀)λ_P0,

Theorem A says that (2.1) has no ground state solutions whenP=P0, a contradiction. So infx∈Λa(x)μ0, and this implies inf_x∈R^Na(x)μ0. 2

Lemma 2.3. Let {P_n: n∈N} ⊂R^N such that P_n→P₀∈R^N as n→ ∞, and fix anyλ∈(0,√

a(P₀)b(P₀) ). If m_λ(P₀) <_N¹S^N/2, then

nlim→∞m_λ(P_n)=m_λ(P₀).

Proof. SincePn→P0, we may assume that λ <

a(Pn)b(Pn)−δ, ∀n∈N, (2.5)

whereδ >0 is a small constant. In this proof, the subscriptλwill be dropped. Then it is standard to prove that m(P_n)= inf

(u,v)∈NPn

L_Pn(u, v)= inf

(u,v)=(0,0)max

t >0L_Pn(t u, t v). (2.6)

Since m(P₀) < _N¹S^N/2, we see from Lemma B that problem (2.1) has a ground state(u₀, v₀)∈C²(R^N,R)when P =P₀, andL_P0(u₀, v₀)=m(P₀). It is easy to see that there existst_n>0 such that(t_nu₀, t_nv₀)∈NPnandt_n→1 as n→ ∞. Then

lim sup

n→∞ m(Pn)lim sup

n→∞ LP_n(tnu0, tnv0)=LP₀(u0, v0)=m(P0), (2.7) and so we may assume thatm(Pn) <_N¹S^N/2 for anyn∈N. Combining this with (2.5) and Lemma B, we see that (2.1) has a positive ground state solution(U_n, V_n)whenP =P_n, andL_Pn(U_n, V_n)=m(P_n). Moreover,U_n, V_n are both radially symmetric decreasing. By (2.5) we have

m(Pn)=LP_n(Un, Vn)− 1 p+1L_P

n(Un, Vn)(Un, Vn)

p−1

2p+2

R^N

|∇U_n|²+a(P_n)U_n²+ |∇V_n|²+b(P_n)V_n²−2λUnV_n

C

U_n²+ V_n²

, ∀n∈N,

whereC >0 is independent ofn. Hence,(Un, Vn)is uniformly bounded inH. Passing to a subsequence, we may assume that(Un, Vn) (U0, V0)weakly inH. Then by repeating the proof of Lemma B in[16]with minor modi- fications, we can prove that(U_n, V_n)→(U₀, V₀)strongly inH and(U₀, V₀)is a nontrivial critical point ofL_P0. By (2.7) we have

lim sup

n→∞ m(P_n)m(P0)L_P0(U0, V0)= lim

n→∞L_Pn(U_n, V_n)= lim

n→∞m(P_n).

This completes the proof. 2

In fact, assumption(V₃)is more general than(V₂), and we believe that there may be some other kinds of conditions ona(x)andb(x)such that(V3)holds. For example, we can show the following result.

Proposition 2.1.Suppose(V₁)holds. Assume that there exist a smooth open bounded domainΛ⊂R^N and some P₁∈Λsuch thatsup_x∈Λa(x)μ₀, and either

a(P1) < inf

x∈∂Λa(x), b(P1) inf

x∈∂Λb(x), or

a(P₁) inf

x∈∂Λa(x), b(P₁) < inf

x∈∂Λb(x). (2.8)

Then(V₃)holds for anyλ∈(0,√ a₀b₀).

Proof. Without loss of generality, we assume that (2.8) holds. Fix anyλ∈(0,√

a₀b₀). In this proof, the subscriptλ will be dropped. Lemma A says that m(P ) < _N¹S^N/2 for any P ∈Λ, so we see from Lemma2.3that m(P ) is continuous with respect toP ∈Λ. Then there existsP₂∈∂Λsuch thatm(P₂)=infP∈∂Λm(P ). By Lemma B we know that (2.1) has a positive ground state solution (U, V ) when P =P₂, and L_P2(U, V )=m(P₂). Noting that a(P1)a(P2)andb(P1) < b(P2), it is easy to see that there exists 0< t0<1 such that(t0U, t0V )∈NP₁. Therefore,

m(P₁)L_P1(t₀U, t₀V )= 1

2− 1

p+1

t₀^p+1|U|^p_p⁺₊¹₁+ 1 2− 1

2^∗

t₀^2∗|V|²₂^∗∗

< 1

2 − 1

p+1

|U|^p_p⁺₊¹₁+ 1 2− 1

2^∗

|V|²₂^∗∗

=L_P2(U, V )=m(P₂), that is,

Pinf∈Λm(P )m(P1) < m(P2)= inf

P∈∂Λm(P ), that is,(V₃)holds. 2

3. Proof of Theorem2.1

In the rest of this paper, we only need to prove Theorem2.1. From now on, we assume that(V₁)hold, and we fix anyλ∈(0,√

a₀b₀)such that(V₃)holds.

Definea_ε(x):=a(εx),b_ε(x):=b(εx). To study (1.3), it suffices to consider the following system

⎧⎪

⎪⎨

⎪⎪

⎩

−u+aεu=u^p+λv, x∈R^N,

−v+bεv=v^2∗−1+λu, x∈R^N, u >0, v >0, x∈R^N,

u(x), v(x)→0 as|x| → ∞.

(3.1)

LetH_a,ε¹ (resp.H_b,ε¹ ) be the completion ofC₀^∞(R^N)with respect to the norm ua,ε=

R^N

|∇u|²+aεu²dx ¹₂

resp.ub,ε=

R^N

|∇u|²+bεu²dx ¹₂

.

DefineH_ε:=H_a,ε¹ ×H_b,ε¹ with a norm(u, v)ε=

u²a,ε+ v²_b,ε. DefineΛε:= {x∈R^N: εx∈Λ}and

χ_Λε(x):=

1 ifx∈Λ_ε, 0 ifx /∈Λ_ε. Note thatλ <√

a₀b₀implies the existence ofδ₀>0 such thata₀−δ₀λ >0 andb₀−λ/δ₀>0. Define α:= 1

2min

1, a0−δ₀λ, b₀− λ δ₀

_p−¹₁

, (3.2)

thenα <1 andα^2∗−2< α^p−1. Define f (s):=

|s|^p−1s if|s|α,

α^p−1s if|s|> α, g(s):=

|s|^2∗−2s if|s|α, α^2∗−2s if|s|> α, and

fε(x, s):=χΛ_ε(x)|s|^p−1s+

1−χΛ_ε(x) f (s); g_ε(x, s):=χ_Λε(x)|s|^2∗−2s+

1−χ_Λε(x)

g(s). (3.3)

LetF_ε(x, s):=_s

0f_ε(x, t) dt,F (s):=_s

0f (t ) dtandG_ε(x, s):=_s

0g_ε(x, t) dt,G(s):=_s

0g(t) dt. Then

f_ε(x, s)|s|^p, g_ε(x, s)|s|^2∗−1, x∈R^N, (3.4) (p+1)Fε(x, s)=f_ε(x, s)s, 2^∗G_ε(x, s)=g_ε(x, s)s, x∈Λ_ε, (3.5) 2Fε(x, s)f_ε(x, s)sα^p−1s², 2Gε(x, s)g_ε(x, s)sα^p−1s², x /∈Λ_ε. (3.6) Define a functionalJ_ε:H_ε→Rby

J_ε(u, v):=1

2u²_a,ε+1

2v²_b,ε−

R^N

F_ε x, u⁺

+G_ε x, v⁺

+λuv

dx. (3.7)

Here and in the following,u⁺(x):=max{u(x),0}and so isv⁺. By(V₁)we see thatH_ε→Hand there exists some C >0 independent ofε >0 such that

(u, v)C(u, v)

ε. (3.8)

Then it is standard to show thatJ_εis well defined andJ_ε∈C¹(H_ε,R). Furthermore, any critical points ofJ_εare weak solutions of the following system

⎧⎪

⎨

⎪⎩

−u+a_εu=f_ε x, u⁺

+λv, x∈R^N,

−v+b_εv=g_ε x, v⁺

+λu, x∈R^N, u(x), v(x)→0 as|x| → ∞.

(3.9)

For each smallε >0, we will find a nontrivial solution of (3.9) by applying mountain-pass argument toJε. Then we shall prove that this solution is a positive solution of (3.1) forε >0 sufficiently small.

By(V1), (3.2) and (3.6), there existsC>0 independent ofε >0 such that u²_a,ε+ v²_b,ε−

R^N\Λ_ε

f_ε(x, u)u+g_ε(x, v)v

−2λ

R^N

uv2C(u, v)²

ε. (3.10)

Then by (3.6)–(3.8) we have J_ε(u, v)C(u, v)²

ε− 1

p+1

Λε

u^+p+1dx− 1 2^∗

Λε

v^+2∗dx C(u, v)²

ε−Cu^p+1−Cv^2∗ C(u, v)²

ε−C(u, v)^p+1

ε −C(u, v)^2∗

ε , (3.11)

whereC >0 is independent ofε >0. Therefore, there exist smallr, α₁>0 independent ofε >0, such that

(u,v)infε=rJ_ε(u, v)α₁>0, ∀ε >0. (3.12)

Define c_ε:= inf

γ∈Φε

sup

t∈[0,1]

J_ε γ (t)

, (3.13)

whereΦε= {γ ∈C([0,1], Hε): γ (0)=(0,0), Jε(γ (1)) <0}. Takeψε∈C₀^∞(Λε,R)such thatψε0 andψε≡0, thenJε(tψε, t ψε)→ −∞ast→ +∞. SoΦε= ∅andcεis well defined. By (3.12) we get

cεα1>0, ∀ε >0. (3.14)

By Lemma A we havem(P )_N¹S^N/2for allP ∈R^N, and by(V₃)there holds m₀< 1

NS^N/2. (3.15)

Then by repeating the proof of Lemma2.3we obtainM= ∅.

Lemma 3.1.lim sup_ε→0c_εm0<_N¹S^N/2.

Proof. Take anyP ∈M, thenm(P )=m₀<_N¹S^N/2. By Lemma B we know that (2.1) has a positive ground state (U, V )such thatL_P(U, V )=m₀. TakeT >1 such thatL_P(T U, T V )−1. Noting that there existsR >0 such that B(P , R):= {x: |x−P|< R} ⊂Λ, we takeφ∈C₀¹(R^N,R)with 0φ1,φ(x)≡1 for|x|R/2 andφ(x)≡0 for

|x|R. Defineφ_ε(x):=φ(εx), thenφ_ε(x−P /ε)=0 impliesx∈Λ_ε. Combining this with (3.3) and the Lebesgue Dominated Convergence Theorem, one has that

J_ε

t (φ_εU )(· −P /ε), t (φ_εV )(· −P /ε)

=t² 2

|x|R/ε

∇(φ_εU )²+a(εx+P )φ_ε²U²+∇(φ_εV )²+b(εx+P )φ_ε²V²

− t^p+1 p+1

|x|R/ε

|φ_εU|^p+1−t^2∗ 2^∗

|x|R/ε

|φ_εV|^2∗−λt²

|x|R/ε

φ_ε²U V

→L_P(tU, tV ), asε→0, uniformly fort∈ [0, T].

So J_ε(T (φ_εU )(· −P /ε), T (φ_εV )(· −P /ε)) <0 for ε > 0 sufficiently small, that is, γ_ε(t):=(tT (φ_εU )(· − P /ε), t T (φ_εV )(· −P /ε))∈Φ_ε, so

lim sup

ε→0

c_εlim sup

ε→0

tmax∈[0,1]J_ε

t T (φ_εU )(· −P /ε), t T (φ_εV )(· −P /ε)

= max

t∈[0,1]L_P(t T U, t T V )=L_P(U, V )=m₀. This completes the proof. 2

Lemma 3.2.There exists smallε1>0such that for anyε∈(0, ε1),Jε has a nontrivial critical point(uε, vε)∈Hε, which satisfies thatu_ε>0, vε>0andJ_ε(u_ε, v_ε)=c_ε.

Proof. By Lemma3.1, there exists small ε₁>0 such that for any ε∈(0, ε1), we have c_ε< _N¹S^N/2. Fix anyε∈ (0, ε1). By the Mountain Pass Theorem[7], there exists(u_n, v_n)∈H_εsuch that

nlim→∞J_ε(u_n, v_n)=c_ε, lim

n→∞J_ε(u_n, v_n)=0.

By (3.4)–(3.6) and (3.10), it is easy to see that c_ε+o(u_n, v_n)

ε

+o(1)J_ε(u_n, v_n)− 1

p+1J_ε(u_n, v_n)(u_n, v_n)

p−1

2(p+1)C(u_n, v_n)²

ε, (3.16)

which implies that(u_n, v_n)is uniformly bounded inH_ε. Up to a subsequence, we may assume that(u_n, v_n) (u_ε, v_ε) weakly inH_ε. ThenJ_ε(u_ε, v_ε)=0.

Step 1. We prove thatu_n→u_εstrongly inH_a,ε¹ .

Take R₀ such that Λ_ε⊂B(0, R0/2). For anyR R₀, we take a cut-off functionη_R∈C^∞(R^N,R)such that 0η_R1,η_R≡0 onB(0, R/2),η_R≡1 onR^N\B(0, R)and|∇η_R|10/R. Thenη_R≡0 onΛ_ε. By(V₁), (3.2) and (3.6), there existsC >0 independent ofnsuch that

R^N

|∇un|²+aεu²_n+ |∇vn|²+bεv²_n

ηRdx−

R^N

fε(x, un)un+gε(x, vn)vn+2λunvn

ηRdx

C

R^N

|∇u_n|²+a_εu²_n+ |∇v_n|²+b_εv²_n

η_Rdx. (3.17)

Then we deduce fromJ_ε(u_n, v_n)(η_Ru_n, η_Rv_n)=o(1)that

R^N

|∇u_n|²+a_εu²_n+ |∇v_n|²+b_εv²_n

η_RdxC

R^N

|∇u_n||∇η_R||u_n| + |∇v_n||∇η_R||v_n|

dx+o(1)

C

R +o(1), (3.18)

that is, lim sup

n→∞

R^N\B(0,R)

|∇un|²+aεu²_n+ |∇vn|²+bεv²_n dxC

R. (3.19)

On the other hand, we may assume that(u_n, v_n)→(u_ε, v_ε)strongly inL^q_loc(R^N)×L^q_loc(R^N), where 2q <2^∗. So lim sup

n→∞

R^N

a_εu²_n−a_εu²_εdxlim sup

n→∞

B(0,R)

a_εu²_n−a_εu²_εdx+lim sup

n→∞

R^N\B(0,R)

a_εu²_n+a_εu²_εdx

C

R holds for anyRR₀,