Parameter controllers for Evolutionary Algorithms (EAs) deal with adjusting parameter values during an evolutionary run. Many ad hoc approaches have been presented for parameter control, but few generic parameter controllers exist and, additionally, no comparisons or in depth analyses of these generic controllers are available in literature. This paper presents an extensive comparison of such generic parameter control methods, including a number of novel controllers based on reinforcement learning which are introduced here. We conducted experiments with different EAs and test problems in an one-off setting, i.e. relatively long runs with controllers used out-of-the-box with no tailoring to the problem at hand. Results reveal several interesting insights regarding the effectiveness of parameter control, the niche applications/EAs, the effect of continuous treatment of parameters and the influence of noise and randomness on control.