In a randomized, double blind, placebo-controlled, within-subject magnetic resonance imaging study, we examined the effect of 20 IU intranasal vasopressin on the neural processing of infant crying in 25 fathers-to-be. We explored whether familial background modulates vasopressin effects, and whether vasopressin differentially affects cry processing coupled with neutral or emotional contextual information. Participants listened to cries accompanied by neutral (‘this is an infant’) or emotional (‘this infant is sick/bored’) contextual information, and neutral control sounds (‘this is a saw’). Additionally, participants reported on their childhood experiences of parental love-withdrawal and abuse. Infant crying (vs control sounds) was associated with increased activation in the bilateral auditory cortex and posterior medial cortex. No effects of vasopressin were found in this ‘cry network’. Exploratory whole-brain analyses suggested that effects of vasopressin in the anterior cingulate cortex, paracingulate gyrus and supplemental motor area were stronger in fathers who experienced lower (vs higher) levels of love-withdrawal. No interaction was observed for abuse. Vasopressin increased activation in response to cries accompanied by emotional vs neutral contextual information in several brain regions, e.g. the cerebellum, brainstem (midbrain), posterior medial cortex, hippocampus, putamen, and insula. Our results suggest that the experience of love-withdrawal may modulate the vasopressin system, influencing effects of vasopressin administration on cry processing. Results further suggest a role for vasopressin in the processing of cry sounds with emotional contextual information.