Objective: To examine whether persons who experienced adverse childhood events or recent negative life events have a worse cognitive performance and faster cognitive decline and the role of depression and apolipoprotein E-e4 in this relationship. Methods: The community-based sample consisted of 10-year follow-up data of 1312 persons participating in the Longitudinal Aging Study Amsterdam (age range 65-85 years). Results: Persons who experienced adverse childhood events showed a faster 10-year decline in processing speed but only when depressive symptoms were experienced. Persons with more recent negative life events showed slower processing speed at baseline but no faster decline. Conclusions: Childhood adversity may cause biological or psychological vulnerability, which is associated with both depressive symptoms and cognitive decline in later life. The accumulation of recent negative life events did not affect cognitive functioning over a longer time period. © The Author(s) 2014.