Maximizing the flow of students through the science, technology, engineering, and math (STEM) pipeline is important to promoting human capital development and reducing economic inequality. A critical juncture in the STEM pipeline is the highly cumulative sequence of secondary school math courses. Students from disadvantaged schools are less likely to complete advanced math courses. Here, we conduct an analysis of how the math pipeline differs across schools using student polygenic scores, which are DNA-based indicators of propensity to succeed in education. We integrated genetic and official school transcript data from over 3000 European-ancestry students from U.S. high schools. We used polygenic scores as a molecular tracer to understand how the flow of students through the high school math pipeline differs in socioeconomically advantaged versus disadvantaged schools. Students with higher education polygenic scores were tracked to more advanced math already at the beginning of high school and persisted in math for more years. Analyses using genetics as a molecular tracer revealed that the dynamics of the math pipeline differed by school advantage. Compared to disadvantaged schools, advantaged schools buffered students with low polygenic scores from dropping out of math. Across all schools, even students with exceptional polygenic scores (top 2%) were unlikely to take the most advanced math classes, suggesting substantial room for improvement in the development of potential STEM talent. These results link new molecular genetic discoveries to a common target of educational-policy reforms.