posted on 2025-11-08, 17:00authored byEgor S. Vyatkin, Sergey A. Tarasenko
We present an analytical theory of second harmonic generation (SHG) in hybrid structures combining a nonlinear 2D crystal with a dielectric metasurface waveguide. The theory describes the excitation spectrum and enhancement of SHG at both leaky mode and quasi-bound state in the continuum (quasi-BIC) resonances in terms of the material parameters. For low-loss systems, the SHG efficiency at leaky resonances is determined by their radiative broadening, governed by the relevant Fourier harmonics of the metasurface polarizability, whereas the SHG enhancement at quasi-BIC resonances is ultimately limited by inhomogeneous broadening and absorption in the system. We also describe the emergence and polarization properties of second harmonic diffracted beams. These beams appear even if both the 2D crystal and the meta-waveguide are centrosymmetric owing to the nonlocal mechanism of SHG. The developed framework provides a systematic theoretical basis for optimizing the resonant nonlinear frequency conversion in hybrid 2D-material-metasurface platforms and identifies the fundamental limitations of the SHG efficiency.