Background Disturbed circadian rhythms have been associated with depression and anxiety, but it is unclear if disturbances in the 24-hr activity rhythm and sleep are independently and specifically related to these disorders. Methods In 1,714 middle-aged and elderly participants of the Rotterdam Study, we collected actigraphy recordings of at least 96 hr (138 ± 14 hr, mean ± standard deviation). Activity rhythms were quantified calculating the fragmentation of the rhythm, stability of the rhythm over days, and timing of the rhythm. Total sleep time, sleep onset latency, and wake after sleep onset were also estimated with actigraphy. Depressive symptoms were assessed with the Center for Epidemiologic Studies Depression scale, persons with clinically relevant depressive symptoms were interviewed to diagnose DSM-IV-depressive disorder. Anxiety disorders were determined with the Munich version of the Composite International Diagnostic Interview. Results More fragmented rhythms were associated with clinically relevant depressive symptoms (odds ratio (OR): 1.27, 95% confidence interval (CI): 1.04;1.54) and anxiety disorders (OR: 1.39, 95% CI: 1.14;1.70) after covariate adjustment. Less stable rhythms, longer sleep onset latency, and more wake after sleep onset were related to clinically relevant depressive symptoms or anxiety disorders only if not adjusted for covariates and other activity rhythm and sleep indicators. Conclusions Our study in middle-aged and elderly persons suggests that fragmentation of the 24-hr activity rhythm is associated with depression and anxiety. Moreover, this association also largely accounts for the effect of disturbed sleep on these psychiatric disorders.