Bridging Hubbard model physics and quantum Hall physics in graphene moire superlattices

Bridging Hubbard model physics and quantum Hall

physics in graphene moire superlattices

by

Yahui Zhang

B.S.,Peking University (2014)

Submitted to the Department of Physics

in partial fulfillment of the requirements for the degree of

Doctor of Philosophy in Physics

at the

MASSACHUSETTS INSTITUTE OF TECHNOLOGY

September 2019

@

Massachusetts Institute of Technology 2019. All rights reserved.

Author ...

Signature redacted...

Department of Physics

August 5th, 2019

Certified by..

Signature redacted...

Senthil Todadri

Professor of Physics

Thesis Supervisor

Signature redacted

A ccepted by .... ... ...

MASSced HUTE _{Nergis Mavalvala}

Associate Department Head of Physics

NOV 08 2019

z

MITLibraries

77 Massachusetts Avenue

Cambridge, MA 02139

http"/ibraries.mitedu/ask

DISCLAIMER NOTICE

Due to the condition of the original material, there are unavoidable flaws in this reproduction. We have made every effort possible to provide you with the best copy available.

Thank you.

The images contained In this document are of the best quality available.

Bridging Hubbard model physics and quantum Hall physics in

graphene moire superlattices

by

Yahui Zhang

Submitted to the Department of Physics on August 5th, 2019, in partial fulfillment of the

requirements for the degree of Doctor of Philosophy in Physics

Abstract

This thesis is focused on the strongly correlated physics of graphene moir6 super-lattices formed in twisted bilayer graphene (TBG), twisted double bilayer graphene (TDBG) and ABC trilayer graphene aligned with hexagon boron nitride (TLG-hBN). First, I will show that the physics of these systems can be divided into two categories: (1)The nearly-flat bands have non-zero valley Chern number, which leads to "quan-tum Hall physics" including integer and fractional quan"quan-tum anomalous Hall effect and composite fermi liquid (CFL) physics. (2) The narrow bands have trivial band topology. In this case the essential physics is captured by a Hubbard like lattice model similar to that of the high Tc cuprates. Both of the above two classes have already been realized in the experiments. I will discuss how current and future exper-iments on these moir6 materials can deepen our understanding of the cuprate physics and quantum Hall physics. In addition, I will also propose several new phases in moir6 systems, which have never been studied before. These include featureless and orthogonal pseudogap metals and quantum Hall spin liquids.

Thesis Supervisor: Senthil Todadri Title: Professor of Physics

Acknowledgments

There are a lot of people I need to acknowledge for my PhD. First, I want to thank my advisor T. Senthil for recruiting me to MIT and accepting me as his student. I learned a lot of physics from Senthil. But most importantly I want to thank Senthil for showing me how to be a good scientist. Ten years later I may be doing something completely different from my PhD research. But I will always trying to be a physicist who is enthusiastic to research as Senthil. For the same reason, I also would like to thank other CMT faculties in CMT for their guidance and advice along my PhD study. I also want to thank my officemates and peers at MIT for fruitful discussions with them. This includes Michael Pretko, Liujun Zou, Dan Mao, Zhehao Dai, Itamar Kimchi, Zhen Bi, Zhihuan Dong, Cecile Repellin and others. Last, I need to thank ex-perimentalists working on moire materials for creating this wonderful system, without which this thesis is not possible.

I would like to thank my girl friend Yan Liu for her love and support during my PhD, which makes my life much more colorful. For several years we were in different countries and I always remember that we had phone calls across ten thousands of miles every night during that time. It's here companion which encouraged me to get through all of the frustrations and hard time these years. I also would like to thank my parents and my brother for their understanding and support. I thank them for encouraging me to study in a different country. I wish I could have spent more time with them.

This doctoral thesis has been examined by a Committee of the Department of Physics as follows:

Signature redacted

Thesis Supervisor Professor of Physics

Signature redacted

Member, Thesis Committee Assistant Professor of Physics

Signature redacted

Professor Raymond Ashoori ....

Member, Thesis Committee Professor of Physics

Contents

1 Introduction 21

1.1 Beyond Landau Paradigm: strongly correlated physics . . . . 21

1.1.1 Beyond Landau symmetry breaking: quantum Hall states . . . 22

1.1.2 Beyond Landau Fermi Liquid: unconventional metallic states in cuprates . . . .. . 23

1.2 Moir6 superlattices: bridging cuprate physics and quantum Hall physics 24 1.3 New phases in moir6 superlattices: pseudogap metals and quantum H all spin liquids . . . . 27

2 Nearly-flat Chern Bands in graphene moire superlattices 29 2.1 Introduction . . . . 29

2.2 Models for Moir6 Mini Band . . . . 31

2.3 Nearly Flat Bands . . . . 33

2.3.1 Twisted Graphene/h-BN Systems . . . . 33

2.3.2 Twisted Graphene/Graphene Systems . . . . 33

2.4 Analysis of Chern number for Moir6 mini band . . . . 34

2.5 Hamiltonian of Chern Bands . . . . 36

2.6 Quantum Anomalous Hall Effect and Quantum Valley Hall Effect . . 40

2.6.1 Hartree Fock Calculation of Quantum Hall Ferromagnetism. 41 2.6.2 vT = 4: Quantum Valley Hall Insulator with o-' = 4C2 . 45 2.6.3 vT = 2: Spin polarized Quantum Valley Hall Insulator with u7 =2Ce ... 45

2.6.4 vT= 1 and vT = 3: Quantum Anomalous Hall Insulator with

oh.i. .. .. ... . .. .. .. .. .. .. .... 46

2.7 Fractional Quantum Anomalous Hall Effect . . . . 47

2.8 Similarities with twisted bilayer graphene . . . . 48

2.9 Conclusion . . . . 49

3 Twisted Bilayer Graphene Aligned with Hexagonal Boron Nitride: Anomalous Hall Effect and a Lattice Model 51 4 Hubbard Model Physics in ABC trilayer graphene aligned with hexagonal boron nitride 63 4.1 Introduction . . . . 63

4.2 Lattice Model For Av < 0 Side: spin-valley Hubbard Model . . . . . 65

4.2.1 Response to Magnetic Field: Valley Zeemann Coupling . . . . 72

4.3 Strong M ott Insulators . . . . 73

4.3.1 Ferromagnetic Region . . . . 76

4.3.2 Antiferromagnetic Region . . . . 77

4.4 Weak Mott insulators: possibility of a continuous Mott Transition . . 78

4.4.1 'Magnetic' metal as an intermediate phase . . . . 80

4.4.2 First order Mott transition . . . . 81

4.4.3 Bandwidth Controlled Continuous Metal-Insulator Transition 82 4.4.4 Doping Controlled Continuous Metal-Insulator Transition . . . 85

4.5 Comments for The Av > 0 side . . . . 87

4.6 Conclusion . . . . 89

5 Spin liquids and pseudogap metals in spin-valley model on triangular moire superlattices 91 5.1 Introduction . . . . 91

5.2 Hamiltonian and Symmetry . . . . 94

5.2.1 M ott Insulator . . . . 94

5.3 Symmetry constraint: Luttinger theorem . . . . 98

5.3.1 Symmetry U(4) or U(2)+ x U(2) . . . . 98

5.3.2 Symmetry (U(1)c x U(1), x SU(2)s))/Z2: Two distinct sym-metric and featureless states . . . . 99

5.4 Projective Symmetry Group analysis at VT 2 . . . . 101

5.4.1 H ilbert Space . . . . 101

5.4.2 Parton Theories at VT 2 . . . . 103

5.4.3 PSG Classification of U(1) Spin Liquids at VT = 2 . . . . 104

5.4.4 PSG Classification of Z2 Spin Liquids at VT= 2 . . . . 105

5.5 Pseudogap Metals at VT 2 - x . . . . 108

5.5.1 JH > 0: Symmetric and featureless pseudogap Metal . . . . . 109

5.5.2 JH = 0: orthogonal metal with small Fermi surfaces . . . . 112

5.6 Deconfined metal between pseudogap metal and conventional Fermi liqu id . . . . 114

5.7 U(1) Spinon Fermi surface state and Z4 spin liquid at VT = 1 . . . . . 119

5.7.1 Absence of Gapped Symmetric Z2 Spin Liquid . . . . 120

5.7.2 U(1) Spin Liquid in Abrikosov Fermion Parton Construction . 120 5.8 C onclusion . . . . 122

6 Ferromagnetism in nearly-flat bands of moirsuperlattices 123 7 Summary and Outlook 135 7.0.1 Summary of Experiments . . . . 135

List of Figures

2-1 Possibility of gate driven nearly flat Chern band for twisted

graphene/h-BN systems. U is the potential difference between the top and bottom

layers, controlled by applied vertical electric field. Figures are gener-ated for BG/h-BN. TG/h-BN shows quite similar behavior. n = 2 for BG/h-BN and n = 3 for TG/h-BN. Two sides with opposite U

have similar band dispersions with Chern number

|CI

|CI

are equivalent in the MBZ. . . . . 32

2-2 Bandwidth of valence band with the applied voltage U for the

TG/h-B N system . . . . 34

2-3 Bandwidth of conduction band with twist angle for the BG/BG system. 34

2-4 Chern number ICI of BG/BG for conduction band. U is in the unit of meV. There is a clear phase boundary Uc(O), at which Chern number jumps from ICI = 2 to |CI = 1 because the hybridization gap between conduction band and the band above closes at F point. Uc(OM) = 0 at

magic angle OM in our simple model. As explained in the main text, our method may underestimate Uc(O) and Uc(9) may remain large even at magic angle in realistic systems. . . . . 36

3-1 Band structure for valley + of the TBG/h-BN system in the MBZ.

By = 1.200. The band of valley - can be generated from the time

reversal transformation. . . . . 55 3-2 Illustration of the lattice model on a Honeycomb lattice. C3t(R)C31 =

t(C3R) generates inter-sublattice hopping and C6t'(R)C-1(6) = t'*(C6R)

generates intra-sublattice hoppings. . . . . 57

4-1 Illustration of the ABC stacked trilayer graphene/h-BN system. We assume the h-BN layer on top is nearly aligned with the graphene layers. A and B refer to the 2 sublattices in each of the graphene layers. Due to the large dimerization term 7y1 ~ 400 meV, only A1

and B3 should be kept at low energy, forming a two-component spinor.

A vertical electric field gives an energy difference Av for electrons

between the top and the bottom graphene layer. The aligned h-BN layer provides a moir6 superlattice potential which folds the original large Brillouin Zone to a small moire Brilloiun zone (MBZ). . . . . . 66

4-2 Magnitude of the nearest neighbor hopping It,I and the next-nearest

neighbor hopping t2. t2 has no imaginary part because of the Mirror reflection symmetry. The phase of ti is shown in Fig. 4-3. The

ver-tical line labels

Av

Hubbard U: W ~ U ~ 25 meV. . . . . 69

4-3 The flux 1I of each triangle from the nearest neighbor hopping. For each triangle, two valleys experience opposite (D. For each valley, 1 changes sign under 06 rotation. The vertical line labels Av = -20

meV where the bandwidth is equal to the Hubbard U: W ~ U ~ 25

meV. For the Mott insulating regime at

Av

a large valley contrasting flux | ~DI' 0.57r - 27r trhough each triangle. Such a flux breaks SU(4) symmetry, which is inherited in the spin model for the Mott insulator through the super-exchange term. . . . . 69

4-4 Response to out of plane magnetic field B from the valley Zeeman

coupling at AV = -25 meV. AE =

R+

average energy of the valley + and the valley -. 6W, is the change of the bandwidth for valley a. A small magnetic field B = 1 T split

the average energy for two valleys by about 3 meV. Meanwhile the bandwidth of one valley is increased by around 1.5 meV while the bandwidth of the other valley is reduced by around 1.5 meV. . . . . . 73

4-5 J1 - J2 parameters with AV. We fix U = 25 meV, gi = 0.4 and g = 0.008 in the calculation. The vertical line is the value of AV for

which the bandwidth W = U. Deep inside the Mott insulating phase,

Ji is ferromagnetic from the Hund's coupling. In the intermediate

regime, both J1 and J2 are antiferromagnetic. . . . . 75

4-6 Three possible phase diagrams tuned by

§

Metal means metal coexisting with antiferromagnetic order. AF insu-lator is a Mott insuinsu-lator with antiferromagnetic order (The most likely candidate is the 120° valley order). In (II) the Mott insulator may be antiferromagnetic or may be a quantum spin liquid. In (III) the specific quantum spin liquid we consider has a spinon Fermi surface

coupled to a U(1) gauge field. At 1 = 0, the ground state is a ferro-_U

magnet because of inter-site Hund's coupling. We also show the plots of the Fermi surfaces. The Fermi surfaces are calculated at VT = 2 using t1, t2, t3, t4 for Av = -20 meV. For the AF metal, we use the

1200 inter-valley order with the order parameter M = 2|t1 . The Fermi

surface area should decrease continuously in the AF metal region as increases. . . . . 79

4-7 The change of Fermi surface area(in units of the area of the MBZ) with order parameter M for the 1200 inter-valley order: HM = -M j ct(cos(Q.

x)r, + sin(Q -x)ry)cx with

Q

(i,0).

and t2 = -1.372 meV for Av = -20 meV. There are several Fermi

surfaces and we only count the hole pocket at I' point. At M -+ 0,

magnetic breakdown effect should give a quantum oscillation frequency corresponding to the original Fermi surface area equal to 0.5, which is not captured by our calculation here. After adding a non-zero t2, Fermi

surfaces can not be fully gapped out until M = 81t1I. . . . . 80

4-8 Density of state at AV = -25 meV. Two vertical lines correspond to

vT= 1 and vT = 2. The Van Hove singularity is away from both vT = 1 and vT= 2. This is true for other values of D in the region -40 < AV < -5 meV. The closest distance to vT = 1 for the Van-Hove singularity is still at least 10% doping away. The Van-Hove singularity is associated with a Lifshitz transition of the Fermi surfaces. At exactly

vT = 1, 2, there is no obvious instability for the Fermi surfaces. .... 82

4-9 Illustration of Bandwidth controlled Metal-Insulator Transition (BMIT) and Doping controlled Metal-Insulator Transition (DMIT). The shaded region is the M ott insulator. . . . . 85

4-10 Response to out of plane magnetic field B from the valley Zeeman

coupling at Ay = 25 meV. AE = E+ - E- is the splitting of the

average energy of the valley + and the valley -. 6W, is the change of the bandwidth for valley a. . . . . 88

5-1 Evolution from pseudogap metal to the conventional Fermi liquid. The

red, blue and black circles denote Fermi surfaces with area 2--2, , and

4-. The intermediate phase is a "deconfined metal" where the fermi surfaces formed by @b and

f

bandwidth Wbare. The maximum value of bandwidth We is plotted as a function of the number of moir6 unit cells N, at filling vT= 1 and

vT = 2. Note that for the same number of moir6 unit cells N,, the

dimension of the Hilbert space is much larger at uT= 2 than at VT= 1. 128 6-2 Properties of low-energy spin excitations at u1T= 1 and Wbare = 0. a)

Spin-wave dispersion around the F point in the C = ±3 model from

exact diagonalization. The colors correspond to different numbers N, of moir6 unit cells. b) Spin stiffness _p, (a is the moir6 lattice constant), as a function of the squared Berry curvature F(k)2 _{averaged over the}

Brillouin zone. We evaluated p, from a linear fit of the spin-wave dis-persion at the F point. For each model, we changed the Berry curvature distribution by adjusting the twist angle 0 or the displacement field D. 130

6-3 Ferromagnetism in the valley-polarized Honeycomb Hubbard model at

filling vT = 1. . . . . 131

6-4 Critical interaction Uc for valley-polarized and spin-valley models. (a) solid and broken lines are for valley-polarized and spin-polarized mod-els, respectively. (b) Uc for four band models are obtained with a 1 x 2 unit cell, and bond dimensions D up to 3000. . . . . 132

List of Tables

2.1 Chern number

|CI

h-BN (A0 = 0) and graphene/graphene systems (A0 2 meV). ... 35

4.1 Tight binding parameters for Av < 0 side. Both Av and t are in units of m eV . . . . . 68

4.2 Parameters of interaction terms in units of meV for Av = -30 meV. To estimate these parameters, we use a screened Coulomb interaction

V(q) =

jq(1e)

nm. gi ~ 0.4 and gh ~ 0.008 are estimated from Wannier orbital calculations. The dependence of the interaction parameters on AV is w eak . . . . 70 5.1 The correspondence between the generator of the SO(6) and the

gener-ators of the SU(4). For example, the SU(4) transformation U = e 2

corresponds to a rotation between P1 and '2 with angle 0. . . . . 101 5.2 Two symmetric Z2 spin liquids. pb and Pf label symmetric

fractional-ization for e and

f

Chapter 1

Introduction

1.1

Beyond Landau Paradigm: strongly correlated

physics

After the great success of quantum mechanics at the level of single atoms, solid state physics (later called condensed matter) emerged as a field to study the behavior of a quantum system consisting of many particles. As the first step, non-interacting elec-tron systems in periodic crystals were easily solved and various important predictions (including eg. semiconductors) were made and realized in the experiments. While the knowledge of the non-interacting systems was already sufficient to establish the science leading to the technology behind the silicon valley and the modern informa-tion age, many physicists continue to try to understand the case with interacinforma-tions, for which exact solution is generically not possible. One important progress is the introduction of the concept of a quasi particle by Landau, which leads to the suc-cess of Landau Fermi liquid theory. In this paradigm, the behavior of quasi-particle can be captured by a mean field Hamiltonian with only bilinear terms. Meanwhile Landau symmetry breaking theory is developed to describe many phases of matter. Superconductivity, magnetic order and other phases can be simply understood as ex-istence of symmetry breaking bilinear terms in the mean field Hamiltonian. We can simply diagonalize the mean field Hamiltonian and fill the mean field energy levels.

Although this approach is extremely successful in many cases, it does not capture the

" superposition" nature of quantum mechanics. In recent three decades, condensed

matter physicists start to explore "strongly correlated phases" beyond the Landau paradigm described above. This "beyond Landau" movement was initiated by two great experimental discoveries in the 80s: quantum Hall systems and the high Tc cuprates.

1.1.1

Beyond Landau symmetry breaking: quantum Hall states

For a long time, condensed matter physicists thought that phases of matter can be simply characterized by symmetry breaking orders. This picture was challenged after the surprising discovery of integer quantum Hall effect (IQHE) and fractional quantum Hall effects (FQHE)[126]. The perfect quantization of Hall conductivity motivated theorists to introduce the concept of Chern number[133] to describe the topology of a non-interacting band. FQHE is even more revolutionary. As Laughlin pointed out with the flux threading thought experiment, fractional Hall conductivity implies the existence of excitation with fractional charge[67]. This kind of fractionalization can never be captured by the mean field order parameter approach. Many new theoretical techniques and concepts have been developed to describe this exotic phase, including variational wavefunctions, Chern-Simons theory and connection to conformal field theory, topological order and modular category[139]. The new development is not restricted to quantum Hall systems. Instead, after realization of the first exotic phase with fractionalization, physicists established a general theoretical framework and proposed other types of exotic phases with similar spirit, such as quantum spin liquids[106].

Right now, our understanding of quantum Hall systems is in a good shape. The remaining challenge is to better characterize or control the system. For example, the abelian and non-abelian braiding statistics have never been demonstrated ex-perimentally. Besides, the necessary high magnetic field in quantum Hall systems make it difficult to be used for topological quantum computation. For the purpose of application, realizing the FQHE at small or even zero magnetic field may be quite

useful. Theoretically we know that FQHE is possible by fractionally filling a nearly-flat Chern band. In practice, it is hard to synthesize a system with nearly-flat Chern band and dope it to the correct filling. In this paper we will show that narrow Chern bands are quite common in several graphene moir6 superlattices where filling can be easily controlled by gating. Therefore moir6 superlattice is a great platform for FQHE at zero magnetic field.

1.1.2

Beyond Landau Fermi Liquid: unconventional metallic

states in cuprates

Shortly after the FQHE discovery, another class of strongly correlated phase was dis-covered in the cuprates materials[69]. First, there is a high Tc superconductor which can not be explained by the conventional electron phonon pairing mechanism. Later measurements further showed that the normal state above Tc in the under-doped re-gion is a quite exotic pseudogap metal instead of a conventional Landau Fermi liquid. Therefore it is clear that the superconductor can not be understood as pairing insta-bility of Fermi surface because there is no well developed Fermi surface in the 'normal state'. When increasing the doping, the system is indeed apparently a conventional Fermi liquid. However, in the intermediate region sandwiched between the pseudo-gap metal and the Fermi liquid, a strange metal phase was found. In the strange metal, the resistivity increases linearly to the temperature. ARPES measurements show that the quasi-particle life time is inverse of T, implying a scattering rate linear to T. All of these experimental results strongly show that this is a non-Fermi liquid. Despite decades of intensive experimental and theoretical studies, understanding of the nature of the pseudogap metal and the strange metal phase remain elusive.

One possible sensible theoretical scenario is that strange metal is a quantum crit-ical point between the pseudogap metal and the conventional Fermi liquid. At zero magnetic field and zero temperature, such a critical point is covered by the super-conducting phase. To reveal the possible quantum critical point by suppressing the superconductivity, experiments have been done under strong magnetic fields. Both

quantum oscillation measurements[108] and Hall measurements[3] suggest a metal with small Fermi surface whose area is proportional to the density of the doped holes

x instead of the density of all electrons vT = 1 - x. A symmetric and featureless Fermi liquid is guaranteed to have Fermi surface area AFS = 2 by Luttinger

the-orem. Therefore this high magnetic field metal must either break the translation symmetry breaking or is not featureless. In the experiments charge density wave (CDW) order is indeed observed. However, any mean field theory with small CDW amplitude can not fully gap out the anti-node pocket and reproduce the quantum oscillation measurements. A mysterious mechanism is still needed to gap out the anti-node gap besides of the CDW order parameter. On the other hand, there is theoretical proposal of a fractionalized Fermi Liquid (FL*) which has small Fermi surface but is fully translation symmetric[112, 104].

It is still under debate whether the parent phase of the pseudogap metal in the cuprate is from translation symmetry breaking or is an exotic metal like FL*. In cuprates it is hard to distinguish these scenarios because of the high magnetic field and complicated phase diagram with intertwined orders. Thus it is highly desirable to realize similar but simpler physics in the other systems. In this thesis we will show several moirsuperlattice may host symmetric pseudogap metal at zero temperature and zero magnetic field. Experimental studies of these systems can provide more insights for our general understanding of possible pseudogap metals.

1.2

Moire superlattices: bridging cuprate physics and

quantum Hall physics

In the above sections we review basic discoveries and remaining problems in two great fields in modern condensed matter physics: the quantum Hall systems and the cuprates. Both systems were discovered in the 80s. Recently moire super-lattices from Van der Waals heterostructures also emerge to be a wonderful plat-form to study strongly correlated physics. These include correlated insulator[9],

superconductivity[10, 151, 801 and anomalous Hall effect[115] in twisted bilayer graphene, spin-polarized correlated insulators[116, 77, 11] and superconductivity[116, 77] in twisted bilayer-bilayer graphene. In addition, ABC trilayer graphene aligned with a hexagonal boron nitride (TLG-hBN) has been demonstrated to host gate tunable correlated insulator[16], superconductor[ 18] and Chern insulator[17]. In this thesis we propose that these graphene moir6 superlattices can simulate similar physics in the cuprates and the quantum Hall systems.

Our theoretical studies of these systems can be organized in two steps: first we calculate the band structures and write down simple models capturing the essential physics; second we try to understand these models by combining theoretical, numeri-cal and experimental efforts. In this thesis we complete the first step for several moir6 materials and show some attempts in the second step.

The band structure calculations of these moir6 superlattices are quite straight-forward following the continuum model approach[7, 52]. When we put one 2D layer on another 2D layer, a large moir6 superlattice can form if there is lattice constant mismatch or a non-zero twist angle. In the experiments where there is strong corre-lation effect, the moir6 lattice constant am is at the order of 10 nm and is around 50 times larger than the origianl graphene lattice constant. Then the moir6 superlattice offers a term which scatters the momentum from k to k + GM, where GM ~ -. _am~, The doping is close to the neutrality and the low energy band structure can be captured

by an effective k -p two band model around two valleys K and K' in the original

graphene Brillouin zone. Because

|GMl

val-ley to the other one. Generically there is no SU(4) symmetry because two valval-leys do not have the same dispersions (generally there is no inversion symmetry within each valley).

Naively one can imagine a spin-valley Hubbard model to describe the low energy physics. However, this is not possible for many moire systems due to band topology. For twisted bilayer graphene (TBG), there is band crossing for each valley, which complicates the analysis. For other systems involving AB bilayer graphene and ABC trilayer graphene, band crossing can be easily gapped out by displacement field[156], which induces non-zero Berry curvatures around the band crossing points. Meanwhile Berry curvatures are also generated close to the MBZ boundary. Depending on the relative sign of Berry curvatures from the above two sources, the final band can have either zero or non-zero Chern numbers. In the case with non-zero Chern numbers, the two valley must have opposite Chern numbers because of the time reversal symmetry. In this thesis we will show that the two cases with zero or non-zero Chern numbers have completely different physics.

According to our theoretical analysis, the proposed flat band moire systems can be divided to three categories: (I) There is band crossing at neutrality point. Twisted bilayer graphene with C2T symmetry is in this category. (II) There is one isolated

narrow band per flavor which is trivial in terms of band topology. In this case the low energy physics is captured by a spin 1/2 or spin-valley Hubbard models on tri-angular lattice. Systems in this category includes twisted transition metal dichalco-genide(TMD) homobilayer, trivial side of ABC trilayer graphene aligned with hexag-onal boron nitride (TLG-hBN) and trivial side of twisted bilayer graphene aligned with both hBN substrates (TBG-hBNs). These systems may host similar phases as in the cuprates. (III) There is one isolated narrow band per flavor with non-zero Chern number. In this case there is no simple lattice model because of the Wannier obstruc-tion. Systems in this flavor include topological side of TLG-hBN, TBG aligned with on hBN layer (TBG-hBN) and topological side of TBG-hBNs, and twisted double bilayer graphene (TDBG). In this thesis we predict that quantum Hall physics can be realized in these systems. Indeed signatures of anomalous Hall effects have already

been observed at v= 3 in TBG-hBN[115] and vT = _{1 in TLG-hBN[17. Hopefully}

FQHE can be observed in future experiments with better samples. In TDBG a spin

polarized insulator was found at vT = 2[116, 77, 11], which we believe also originates

from the non-trivial band topology.

1.3

New phases in moir6 superlattices: pseudogap

metals and quantum Hall spin liquids

In the previous section we propose that moir6 superlattices can simulate cuprate physics and quantum Hall physics. As a class of new systems, moir6 physics is defi-nitely beyond the familiar physics which have already been realized in the traditional solid state systems. In this thesis we will also give proposals of new phases which are unique in moir6 systems. In the following we give outlines of these proposals for the Hubbard systems and the narrow Chern band systems respectively.

In TLG-hBN, a spin-valley Hubbard model can be realized for one sign of displace-ment field. Unlike the cuprates, there are four flavors per site. Mott insulators have been observed experimentally at filling vT= 1, 2. For these Mott insulators, the large quantum fluctuation may suppress magnetic order and favor a singlet phase. Doping such spin singlet Mott insulators can lead to new physics distinct from the cuprates. Besides, the Fermi surfaces on triangular lattice do not have any nesting instability. Therefore a Fermi liquid phase can survive to finite U/t and there should be a Mott transition at both vT = 1 and vT= 2. The capability of tuning the bandwidth by gating in TLG-hBN offers a great possibility to study such Mott transitions. The final solution of the spin-valley Hubbard models is very hard and a rich phase diagram is expected given so many parameters which can be tuned in the experiments. In this thesis we will provide one ansatz of a simple symmetric pseudogap metal upon doping

the vT = 2 Mott insulator.

Next we turn to the systems with nearly-flat Chern bands. In these systems two valleys have opposite Chern numbers. Quantum anomalous Hall effects and

even fractional quantum anomalous Hall effects are possible through spontaneous polarization of the valley. In mean field level, to get a Chern insulator, we need the spin to be also polarized, similar to the quantum Hall ferro-magnetism. In the traditional Landau level systems, kinetic term is zero and the ferromagnetic exchange from the Coulomb interaction always dominates, making the quantum Hall FM the natural fate for integer vT. In the moire systems, the bandwidth W is not zero. When W is very small, we still expect spin polarization. When W is large, we expect a paramagnetic metal. One natural question is: is there a Chern insulator phase for which the spin is anti-ferromagnetic ordered or disordered similar to the quantum spin liquid in the Hubbard model? Through numerical simulation and parton construction, we find that quantum Hall spin liquid phase is possible. In this exotic phase, charge transport is exactly the same as a Chern insulator. However, spin is in a spin liquid phase similar to the conventional Z₂ spin liquid . Unlike the conventional Z₂ spin liquid, vison in the quantum Hall spin liquids carries charge. In the experiments of TBG-hBN and TLG-hBN, Chern insulator behaviors are identified through charge transports. Our results imply that both quantum Hall ferromagnetism and quantum Hall spin liquid are consistent with these experiments. Further probes of the spin degree of freedom is needed to distinguish quantum Hall FM and quantum Hall spin liquid.

Chapter 2

Nearly-flat Chern Bands in graphene

moire superlattices

2.1

Introduction

In this chapter we describe graphene-based moir6 super-lattice systems to realize model systems with two nearly flat bands which each carry equal and opposite Chern number C, the most basic topological index of a band[133]. Previously Ref. [122] demonstrated that when monolayer graphene is stacked on top of h-BN to form a moir6 super-lattice, isolated tC = 1 Chern bands may potentially occur. Other

pro-posals for bands with Chern number iC in various graphene systems can be found in Refs. [134, 105, 122, 141]. The width of these bands is large compared to the expected Coulomb energy and hence correlation effects may be expected to be weak. Here we show that modifications of the proposal of Ref. [1221 naturally enable engi-neering iC > 1 Chern bands which, moreover, are nearly flat. We dub such bands tC Chern bands. We find systems with various values of C = 0, 1, 2, 3. The near

flatness implies that correlation effects from Coulomb interaction are potentially im-portant. Remarkably, C can be controlled simply by applying a transverse electric field. Such an electric field also allows control of the bandwidth of the nearly flat band as shown experimentally[16]. This allows electric control of correlation effects, the band topology, and the band filling and provides a possibly unique opportunity

to study many fascinating phenomena. Systems with flat Chern bands have also been studied theoretically for many years[117, 89, 137, 130, 102, 128, 6, 88, 94] as a po-tential realization of novel interacting topological phases of matter such as fractional Chern insulators. Fractional Chern insulators have been realized in strong magnetic field experiments with Harper-Hofstadter bands[124] of bilayer graphene. However interacting topological phases at zero magnetic field remain elusive in experiment.

As examples, we describe some natural and simple phases at integer filings vT

(which is defined to be the number of electrons/moir6 unit cell including spin and valley degrees of freedom) where, we expect that the spin/valley degree of freedom will spontaneously uniformly polarize (" spin/valley ferromagnet") so as to fully fill the band. For instance at VT = 1 there is a spin and valley polarized insulating

state which will show an electrical quantum Hall effect with o. = . This is an Integer Quantum Anomalous Hall (QAH) insulator. QAH effects have been reported in magnetically doped three dimensional topological insulator(TI) systems[13, 15, 63,

53]. However, QAH effect in a true two dimensional system with a Chern band has

never been achieved before.

At non-integer, but rational fillings vT, a number of topological ordered states

-both Abelian and non-Abelian -seem possible. We discuss some of these -specifically Fractional Quantum Anomalous Hall (FQAH) states - as well with a focus on states that have been previously established in numerical calculations on correlated flat Chern bands. Our proposed realization of nearly flat iC Chern bands with tunable interactions calls for a detailed exploration of the possible many body phases. We will mostly leave this for future work but remark that correlated iC Chern bands may also be an excellent platform for the appearance of fractional topological insulators.

For TG/h-BN correlated insulating states have already been reported in Ref. [16].

A model for one sign of the electric field when C = 0 was described in Ref. [98].

For the other sign of the electric field we get ±C = 3 Chern bands, and we propose

possible explanations of the observed insulating states in this side.

Though the details are different, there is nevertheless a conceptual similarity be-tween the Moire systems with ±C Chern bands discussed in this paper and the twisted

bilayer graphene system. In particular, due to the non-zero Chern number of valley filtered bands, it is obvious that it is not possible to write down maximally local-ized Wanier functions while preserving the valley U(1) symmetry as a local "on-site" symmetry.. Exactly the same feature appears - albeit in a more subtle form - in the theoretical analysis of Ref. [98] of the twisted bilayer graphene system. Thus both the systems discussed in this paper and the twisted bilayer graphene present an ex-perimental context for a novel theoretical problem where strong correlations combine with topological aspects of band structure. This makes their modeling different from more conventional correlated solids.

2.2

Models for Moire Mini Band

At low energies the electronic physics of graphene is dominated by states near each of two valleys which we denote i. We start from a continuum model of these low energy states:

H_o =Z $1(k)ha(k)@ a(k) (2.1)

a k

where a =, - is the valley index. Oa(k) is a multi-component vector with sublattice

and layer index.

Time reversal transforms one valley to the other one, and requires that h+(k) h* (-k). In the following we focus on valley

+.

We restrict attention to graphene with chiral stacking patterns, such as AB stacked bilayer graphene and ABC stacked trilayer graphene. The effective Hamiltonian for one valley

+

U A( k, - ik On

h+((k) = 2 (2.2)

2

/

where U is a mass term. A = r, where v ~ 106m/s is the velocity of Dirac cone

on1

monolayer graphene with n = 1, U is the energy difference between two sublattices. For AB stacked bilayer graphene (BG) with n = 2 and ABC stacked trilayer graphene

(TG) with n = _{3, U is the energy difference between top layer and bottom layer.}

Therefore U can be easily controlled by perpendicular electric field in BG and TG, which are the main focus in this paper.

When two lattices with lattice constant a₁ and a2 are stacked with a small twist angle 9 or when they have slightly different lattice constants with ( = al--2, there is

a large moir6 super-lattice with lattice constant am = a at small twist angle 9,

where a = a,a2. The moire lattice reconstructs the original band into a small Moire

Brillouin Zone(MBZ) which is a hexagon with size |K| = _3am4. The calculation of band structure[7] is very similar to the classic example of free electron approximation in elementary solid state textbooks[2]. When am

»

200 150 U. -100OV 100 -50 -150 -200 -__ 20 200 150 U O 150 U-100 meY 100 100 0~~ "">50 >L 50 -U -50 - '-.... " .... ... -100 7'-100 -150 L-150L KK K" K r K' K" K r

Figure 2-1: Possibility of gate driven nearly flat Chern band for twisted

graphene/h-BN systems. U is the potential difference between the top and bottom layers,

con-trolled by applied vertical electric field. Figures are generated for BG/h-BN.

TG/h-BN shows quite similar behavior. n = 2 for BG/h-BN and n = 3 for TG/h-BN.

Two sides with opposite U have similar band dispersions with Chern number |C| = 0

and

1C|

and quantum Hall physics within one sample. Twist angle 9 = 0 and am ~ 15 nm.

K' = (0, ). K" = ( 2r, 27r ) and K = _{(0, -' ) are equivalent in the MBZ.}

3a 3a 3aM' 3aM euvln h

We discuss two different categories of moire systems in this article. (1) n-layer graphene on top of h-BN with small twist angle. Each valley maps to the I' point of the MBZ. (2) ni-layer graphene on top of n2-layer graphene with a relative twist

angle 9 close to a 'magic' value ~ 10. In the MBZ, the valley + of top and bottom graphene layers map to the K and K' points respectively. Valley - is related by time

reversal transformation. We assume that the two graphenes have the same chirality of stacking pattern. For example, we assume both BG are AB stacked in the BG/BG system.

In these systems, generically only C3 rotation symmetry is preserved while the inversion symmetry or equivalently C6 rotation symmetry is broken. Time reversal symmetry flips the valley index and only requires that the Chern numbers for the bands of two valleys are opposite.

In the next two sections we demonstrate that these systems show the following two interesting features : (1) Nearly flat bands; (2)Non-zero Chern number of the narrow bands for each valley. The small bandwidth implies that strong correlation effects may be present and play an important role in determining the physics. The non-zero i Chern number further renders the physics different from traditional Hubbard-like lattice models which are only appropriate for topologically trivial bands.

2.3

Nearly Flat Bands

2.3.1

Twisted Graphene/h-BN Systems

As shown experimentally for the TG/h-BN system in Ref. [16], the width of the valence and conduction bands can be tuned by applying perpendicular electric field oc U (please see Fig. 2-2). We further demonstrate this theoretically for the BG/h-BN system in Fig. 2-1.

2.3.2

Twisted Graphene/Graphene Systems

For twisted graphene/graphene systems, there exist "magic angles" at which the band-width is reduced to almost zero. This demonstrates that the existence of magic angles is a common feature for graphene/graphene systems and is not special to the twisted bilayer graphene system studied in Ref.

[7].

100- 80- 60- 40- 20-Ul: _ --1i00 -75 -50 -i5 0 is 5a i5 10 U(meV)

Figure 2-2: Bandwidth of valence band with the applied voltage U for the TG/h-BN system. 20.0 - Type I -Typel11 17.5 15.0-12.5 - 10.0-7.5. 5.0-2.5 0.0 0.9 1.0 1.1 1. 1.3 1.4

Figure 2-3: Bandwidth of conduction band with twist angle for the BG/BG system.

For BG/BG and TG/BG system, the magic angle is the same as the twisted bilayer

graphene: 0M = 1.080. For TG/TG, we have 0

M = 1.24'. A plot of bandwidth vs

twist angle is shown in Fig. 2-3 for the BG/BG system.

2.4

Analysis of Chern number for Moire mini band

Our results for Chern numbers at small |UI are summarized in Table. 2.1. We only show the absolute value |C of the Chern number because the two valleys always have opposite Chern numbers. This table is one of the key results of this article. Below we explain these results.

For n-layer graphene/h-BN systems, the "mass" term U in Eq. 2.2 gaps out the band crossing at the r point, as shown in Fig. 2-1. Typically different directions

Band Gap Band Width

""

Systems U<-Ao U>Ao BG/h-BN,valence 0 2 TG/h-BN,valence 0 3 BG/h-BN,conduction 2 0 TG /h-BN,conduction 3 0 BG/BG 2 2 TG/TG 3 3 TG/BG, conduction 2 3 TG/BG, valence 3 2

Table 2.1: Chern number

|CI

/valence

graphene/h-BN (Ao = 0) and graphene/graphene systems (Ao 2 meV).

of perpendicular electric field give very similar band structure. When the sign of

U changes, there is a band inversion which changes the contribution to the Berry

curvature from states near the band touching points. In ordinary n-layer graphene the two valleys are connected in momentum space and their opposite Berry curvatures implies that the net Chern number is zero. The moir6 potential however detaches the minibands in each valley from the rest of the spectrum, and their individual Chern numbers ±C become well defined. The exact value of C of each miniband depends on contributions away from the band touching points, which do not change with sign

of U. For n-layer graphene, Eq. 2.2 readily gives AC = C(U) - C(-U) = n. Detailed

calculations show that C(U < 0) = 0 and C(U > 0) = n for the valence band. We have verified that the Chern number is quite stable to different parameters used for the moir6 super-lattice potential.

For the graphene/graphene system with ni and n2 layers, we found that there is

already a small gap Ao - 2 meV for the band crossing point at K and K' point. We only discuss the case

|UI

positive. For the BG/BG and TG/TG systems with ni = n2 = n, there is a mirror reflection symmetry which requires C(U) = -C(-U). Then we can show analytically

that C = _{±n. For the TG/BG system, we rely on numerical calculations to get the}

The results above are based on the assumption that

IUI

by 1. The phase diagram for the BG/BG system is shown in Fig. 2-4. Uc(6) goes to

almost zero at the magic angle Om in our simple continuum model. At Uc, the gap between the valence band and the band below closes. However, it is likely that lattice relaxation and interaction effects will enhance this hybridization gap and as a con-sequence Uc(OM) may remain large even at the magic angle in realistic experimental systems. We leave it to experiment to measure the true hybridization gap'.

-*-CrO 30 -ICIA1**seeee -1- 10=2 *****.e .... 20- 0- *13 nCI= S6 10--0 -10 0.6 0.0 1.0 1. 1.4 1.6 0

Figure 2-4: Chern number |C of BG/BG for conduction band. U is in the unit of meV. There is a clear phase boundary Uc(6), at which Chern number jumps from

|C| = 2 to |CI = 1 because the hybridization gap between conduction band and the

band above closes at F point. Uc(6m) = 0 at magic angle OM in our simple model.

As explained in the main text, our method may underestimate Uc(9) and Uc(9) may remain large even at magic angle in realistic systems.

2.5

Hamiltonian of Chern Bands

Having shown that nearly flat Chern bands are possible in graphene based moire systems we now discuss the Hamiltonian to model these t Chern bands. The model of course needs to have a global Uc(1) charge conservation symmetry, and spin SU(2) rotation symmetry. In addition the valley U,(1) symmetry is an excellent approximate symmetry and we will treat it as exact.

'For a clean sample, the valley contrasting Chern number may also be extracted from the con-ductance of the helical edge modes in the fully filled Quantum Valley Hall insulator at vT = _4.

The microscopic Hamiltonian is:

H = Hband + Hint (2.3)

where Hband is the band dispersion and Hint is the electron-electron interaction:

Hint = Pa1,,a (q)V(q)pa2,12(-q) (2.4)

al,a2; ol,a2 q

where summation of q is in the whole R2 _{space instead of the Mini Brillouin Zone.}

We use the Coulomb potential V(q) ~ .

A low energy Hamiltonian is obtained by projecting this microscopic Hamiltonian to the active bands near the Fermi energy. The kinetic term is then restricted to

Ho = cf k)ca a ()(kca(k) (2.5)

aa k

where a +, - is valley index and -=t4 is spin index. (+(k) =_(-k) is guaran-teed by time reversal symmetry.

The leading term in the interaction part of the effective Hamiltonian will again be the density-density repulsion but we must use density operators projected to the active bands. For concreteness we will describe this below for the valence band. Valence band annihilation operator c(k) can be expressed in terms of the microscopic electron operator fa;a,o(k):

Ca,a(k) =3ymn,a;a(k)fa;a,oi(k + mG1 + nG2) (2.6)

m,n

where pmn,a;a(k) is the Bloch wave function of valence band. m, n is the label of the momentum points of M x M grid in free electron approximation. a is label of orbital for the microscopic electron fc;a,,(k), for example aZ = t, b labels top layer and

bottom layer of BG. In terms of fc;,,(k), we have pa(q) = Eo fct;a,a(k + q)fa;a,a(k).

From Eq. 2.6 we get:

fa;a(k + mG1 + nG2) = pi*,c;a(k)ca(k) +... (2.7)

where ... denotes other bands and we have suppressed spin index.

We can now represent the projected density operator fa(q) in terms of ca(k):

(2.8) pa,,(q) = Aa(k, k + q)ct)a,(P(k + q))ca,,(k)

k

where P(k) = ko projects k to ko in the Moir6 Brillouin Zone(MBZ) if k = ko +

mG1 + nG2.

The form factor Aa is defined through the Bloch wave function pa(k):

Aa(k, k + q) = (pa(k)|p(k + q)) (2.9)

If k = ko + moG1 + noG2 is not in the MBZ, pmn,a;a(k) is defined as:

(2.10)

Pm,n,a;a(k)= ym+mo,n+no,a;a(ko)

We emphasize again that Va1,a2 (q + mG1 + nG2) 7 Vai,a2(q) unless m, n = 0.

Correspondingly Aa(k, k + q)

#

truncate q = P(q)+mG1+nG 2 with

Iml,

over only MBZ.

Ca(k), pa(k) and Aa(k, k + q) are all not gauge invariant. There is a gauge degree

of freedom:

Ca(k) - ca(k) eea(k)

pa(k) -+ pa(k)edO(k)

One can easily check that

pba(q)

The effective density-density interaction takes the form

Hv =

p

a1,a2;U1,-2 q

where summation of q is in the whole R2 _{space instead of the Moir6 Brillouin Zone.}

We use the Coulomb potential V(q) ~ .

Due to the non-trivial algebra satisfied by the fi operators, interaction effects are qualitatively different from the case with zero Chern number. Actually, it has been shown that for a single Chern band, interactions may drive the system into exotic topological ordered states similar to Fractional Quantum Hall states[94]. We will address this possibility in subsequent sections.

The terms Ho + Hv (from Eqns. 2.5 and 2.12) define an approximate effective

Hamiltonian for the partially filed valence band. At this level of approximation, the electron charge and the electron spin in each valley is separately conserved. With just these terms the effective Hamiltonian thus has a U(2) x U(2) symmetry. This will be further broken down to the assumed physical symmetries of just charge Uc(1), valley

U,(1), and total spin SU(2) by weaker terms. The pertinent such weaker interaction

is a Hund's coupling:

H - JH d2X Sa (q) -Sa2(-q) (2.13)

a1,a2

where Sa(q) is the spin operator associated with electrons in valley a at a momentum

takes the form

Sa(q) = 1 E Aa(k, k + q)c ,(P(k + q)o ca, 2(k) (2.14)

k;aio2

We expect that JH is weaker than the Coulomb scale by roughly a factor a . We will therefore initially ignore it and will include its effects as needed later.

For completeness we note that we can also define the valley density, or more generally, a SU(2), "valley pseudo-spin" operator projected to the valence band:

I,(q) = Aab(k, k + q)ct,(P(k + q))Tacb,(k) (2.15)

ab;ko-where,

Aab(k, k + q) = (Pa(k) I b(k + q)) (2-16) Only the 12(q = 0) operator is conserved.

Our final approximate effective Hamiltonian for the partially filled valence band takes the form

Heff = Ho + Hv + HJ (2.17)

with each of the three terms on the right given by Eqns. 2.5, 2.12, and 2.13 respec-tively.

2.6

Quantum Anomalous Hall Effect and Quantum

Valley Hall Effect

We now discuss the possible insulating phases at integer filling VT of valence bands

with non-zero Chern number. vT is defined as the total density for each moir6 unit

cell. vT = 4 corresponds to filling four bands completely. Here we only focus on the

case |C| 5 0 and vT = 1, 2, 3. For C = 0, there is a localized Wannier orbital for

each valley and the physics is governed by an extended Hubbard model. The C = 0