The number of breast cancer survivors is gradually increasing and a subset of these patients experience long-term adverse effects of adjuvant systemic therapy, including cognitive decline. Surprisingly, relatively little is known about the long-term adverse effects of endocrine treatment on cognition. As 75% of all patients with breast cancer are eligible to receive hormonal treatment, understanding the potential neurocognitive adverse effects of such therapy is of utmost importance. Concerns about adverse cognitive effects of adjuvant endocrine therapy are timely, as recently updated guidelines recommend increasing the length of such therapy from 5 years to 10 years for a subset of patients. The decline of cognitive functions can have a detrimental impact on quality of life and might interfere with independent living. This Review discusses the tissue-selective side effects of endocrine therapies and specifically their impact on cognitive function, on the basis of clinical data; the neurobiological effects of endocrine therapies as observed in preclinical models are also discussed. We highlight the critical issues that need to be addressed in future preclinical and clinical studies in order to best assess the cognitive effects of endocrine treatment in patients with breast cancer.