Photosynthetic light harvesting can be very efficient in solar energy conversion while taking place in a highly disordered and noisy physiological environment. This efficiency is achieved by the ultrafast speed of the primary photosynthetic processes, which is enabled by a delicate interplay of quantum effects, thermodynamics and environmental noise. The primary processes take place in light-harvesting antennas built from pigments bound to a fluctuating protein scaffold. Here, we employ ultrafast single-molecule spectroscopy to follow fluctuations of the femtosecond energy transfer times in individual LH2 antenna complexes of purple bacteria. By combining single molecule results with ensemble spectroscopy through a unified theoretical description of both, we show how the protein fluctuations alter the excitation energy transfer dynamics. We find that from the thirteen orders of magnitude of possible timescales from picoseconds to minutes, the relevant fluctuations occur predominantly on a biological timescale of seconds, i.e. in the domain of slow protein motion. The measured spectra and dynamics can be explained by the protein modulating pigment excitation energies only. Moreover, we find that the small spread of pigment mean energies allows for excitation delocalization between the coupled pigments to survive. These unique features provide fast energy transport even in the presence of disorder. We conclude that this is the mechanism that enables LH2 to operate as a robust light-harvester, in spite of its intrinsically noisy biological environment.