It has by now been established that standard QCD factorization using transverse momentum dependent parton distribution functions fails in hadroproduction of nearly back-to-back hadrons with high transverse momentum. The essential problem is that gauge-invariant transverse momentum dependent parton distribution functions cannot be defined with process-independent Wilson line operators, thus implying a breakdown of universality. This has led naturally to proposals that a correct approach is to instead use a type of generalized transverse momentum dependent factorization in which the basic factorized structure is assumed to remain valid, but with transverse momentum dependent parton distribution functions that contain nonstandard, process-dependent Wilson line structures. In other words, to recover a factorization formula, it has become common to assume that it is sufficient to simply modify the Wilson lines in the parton correlation functions for each separate hadron. In this paper, we will illustrate by direct counterexample that this is not possible in a non-Abelian gauge theory. Since a proof of generalized transverse momentum dependent factorization should apply generally to any hard hadroproduction process, a single counterexample suffices to show that a general proof does not exist. Therefore, to make the counter-argument clear and explicit, we illustrate with a specific calculation for a double spin asymmetry in a spectator model with a non-Abelian gauge field. The observed breakdown of generalized transverse momentum dependent factorization challenges the notion that the role of parton transverse momentum in such processes can be described using separate correlation functions for each external hadron. © 2010 The American Physical Society.