Recently it was argued that in addition to top-down and bottom-up processes, lingering biases of selection history play a major role in visual selection (Awh, Belopolsky & Theeuwes, 2012). Since its publication there has been a growing controversy about the terms top-down, bottom-up and selection-history in relation to visual selection. In the current paper we define these terms, discuss some controversies about these terms and explain what kind of effects should be considered to be the result of lingering biases of selection history, i.e., priming, reward/fear, and statistical learning. We discuss the properties of top-down selection (slow, effortful, and controlled) versus the properties of lingering biases of selection history (fast, effortless, and automatic). We adhere the position that the experience with selecting a particular feature or the location of a feature, may boost and sharpen its representation within the priority selection map above and beyond its physical salience. It is as if the experience may render a feature or location subjectively more salient. Our message of the current review is that true top-down control of visual selection occurs far less often than what is typically assumed. Most of the time, selection is based on experience and history. It is fast, automatic and occurs without much, if any, effort.