Peptide aggregation, the self-assembly of peptides into structured beta-sheet fibril structures, is driven by a combination of intra- and intermolecular interactions. Here, the interplay between intramolecular and formed inter-sheet hydrogen bonds and the effect of dispersion interactions on the formation of neutral, isolated, peptide dimers is studied using infrared action spectroscopy. Therefore, four different homo- and heterogenous dimers resulting from three different alanine-based model peptides have been formed under controlled and isolated conditions. The peptides differ from one another by the presence and location of a UV chromophore containing end cap. The conformations of the monomers of the peptides direct the final dimer structure: strongly bonded or folded structures result in weakly bound dimers. Here, intramolecular hydrogen bonds are favored over new intermolecular hydrogen bond interactions. In contrast, linear monomers are the ideal template to form parallel beta-sheet type structures. The weak intramolecular hydrogen bonds present in the linear monomers are replaced by the stronger inter-sheet hydrogen bond interactions. The influence of π-π dispersion interactions on the structure of the dimers is minimal, and the phenyl rings have a tendency to fold away from the peptide backbone to favour intermolecular hydrogen bond interactions over dispersion interactions. Quantum chemical calculations confirm our experimental observations.