The evolution and stability of two-dimensional, large-amplitude, non-hydrostatic internal wavepackets are examined analytically and by numerical simulations. The weakly nonlinear dispersion relation for horizontally periodic, vertically compact internal waves is derived and the results are applied to assess the stability of weakly nonlinear wavepackets to vertical modulations. In terms of Θ, the angle that lines of constant phase make with the vertical, the wavepackets are predicted to be unstable if [mid ]Θ[mid ] < Θc, where Θc = cos−1 (2/3)1/2 ≃ 35.3° is the angle corresponding to internal waves with the fastest vertical group velocity. Fully nonlinear numerical simulations of finite-amplitude wavepackets confirm this prediction: the amplitude of wavepackets with [mid ]Θ[mid ] > Θc decreases over time; the amplitude of wavepackets with [mid ]Θ[mid ] < Θc increases initially, but then decreases as the wavepacket subdivides into a wave train, following the well-known Fermi–Pasta–Ulam recurrence phenomenon.

If the initial wavepacket is of sufficiently large amplitude, it becomes unstable in the sense that eventually it convectively overturns. Two new analytic conditions for the stability of quasi-plane large-amplitude internal waves are proposed. These are qualitatively and quantitatively different from the parametric instability of plane periodic internal waves. The ‘breaking condition’ requires not only that the wave is statically unstable but that the convective instability growth rate is greater than the frequency of the waves. The critical amplitude for breaking to occur is found to be ACV = cot Θ (1 + cos2 Θ)/2π, where ACV is the ratio of the maximum vertical displacement of the wave to its horizontal wavelength. A second instability condition proposes that a statically stable wavepacket may evolve so that it becomes convectively unstable due to resonant interactions between the waves and the wave-induced mean flow. This hypothesis is based on the assumption that the resonant long wave–short wave interaction, which Grimshaw (1977) has shown amplifies the waves linearly in time, continues to amplify the waves in the fully nonlinear regime. Using linear theory estimates, the critical amplitude for instability is ASA = sin 2Θ/(8π2)1/2. The results of numerical simulations of horizontally periodic, vertically compact wavepackets show excellent agreement with this latter stability condition. However, for wavepackets with horizontal extent comparable with the horizontal wavelength, the wavepacket is found to be stable at larger amplitudes than predicted if Θ [lsim ] 45°. It is proposed that these results may explain why internal waves generated by turbulence in laboratory experiments are often observed to be excited within a narrow frequency band corresponding to Θ less than approximately 45°.