The onset of thermosolutal convection and finite-amplitude flows, due to vertical gradients of heat and solute, in a horizontal rectangular enclosure are investigated analytically and numerically. Dirichlet or Neumann boundary conditions for temperature and solute concentration are applied to the two horizontal walls of the enclosure, while the two vertical ones are assumed impermeable and insulated. The cases of stress-free and non-slip horizontal boundaries are considered. The governing equations are solved numerically using a finite element method. To study the linear stability of the quiescent state and of the fully developed flows, a reliable numerical technique is implemented on the basis of Galerkin and finite element methods. The thresholds for finite-amplitude, oscillatory and monotonic convection instabilities are determined explicitly in terms of the governing parameters. In the diffusive mode (solute is stabilizing) it is demonstrated that overstability and subcritical convection may set in at a Rayleigh number well below the threshold of monotonic instability, when the thermal to solutal diffusivity ratio is greater than unity. In an infinite layer with rigid boundaries, the wavelength at the onset of overstability was found to be a function of the governing parameters. Analytical solutions, for finite-amplitude convection, are derived on the basis of a weak nonlinear perturbation theory for general cases and on the basis of the parallel flow approximation for a shallow enclosure subject to Neumann boundary conditions. The stability of the parallel flow solution is studied and the threshold for Hopf bifurcation is determined. For a relatively large aspect ratio enclosure, the numerical solution indicates horizontally travelling waves developing near the threshold of the oscillatory convection. Multiple confined steady and unsteady states are found to coexist. Finally, note that all the numerical solutions presented in this paper were found to be stable.