Background: The associations between prenatal exposure to endocrine disruptive chemicals (EDCs) and fetal growth are inconsistent, and few studies have considered small-for-gestational-age (SGA) birth as an outcome. Our current study of Scandinavian parous women aimed to address these inconsistencies and gaps in the literature. Methods: This case-cohort study included 424 mother-child pairs who participated in a prospective, multi-center study of parous women in Norway (Trondheim and Bergen) and Sweden (Uppsala). We used linear and logistic regression with 95% confidence intervals (CIs) to analyze the associations between two perfluoroalkyl substances (PFASs) and five organochlorines (OCs) from early second trimester and indices of fetal growth. Results: Among Swedish women, prenatal exposure to perfluorooctanoate (PFOA), polychlorinated biphenyl (PCB) 153 and hexachlorobenzene (HCB) were associated with higher odds for SGA birth. We found stronger associations among Swedish male offspring. In the Norwegian cohort, we found no significant associations between EDC exposure and indices of fetal growth. Conclusions: Some populations may be more vulnerable to EDCs, possibly due to differences in exposure levels, exposure sources and/or modifiable lifestyle factors. Male offspring may be more vulnerable to endocrine disruption.