Processing quantities such as the number of objects in a set, size, spatial arrangement and time is an essential means of structuring the external world and preparing for action. The theory of magnitude suggests that number and time, among other continuous magnitudes, are linked by a common cortical metric, and their specialization develops from a single magnitude system. In order to investigate potentially shared neural mechanisms underlying numerosity and time processing, we used visual adaptation, a method which can reveal the existence of a dedicated processing system. We reasoned that cross-adaptation between numerosity and duration would concur with the existence of a common processing mechanism, whereas the absence of cross-adaptation would provide evidence against it. We conducted four experiments using a rapid adaptation protocol where participants adapted to either visual numerosity or visual duration and subsequently performed a numerosity or duration discrimination task. We found that adapting to a low numerosity altered the estimation of the reference numerosity by an average of 5 dots, compared to adapting to a high numerosity. Similarly, adapting to a short duration altered the estimation of the reference duration by an average of 43 msec, compared to adapting to a long duration. In the cross-dimensional adaptation conditions, duration adaptation altered numerosity estimation by an average of 1 dot, whereas there was not sufficient evidence to either support or reject the effect of numerosity adaptation on duration judgments. These results highlight that there are partially overlapping neural mechanisms which are dedicated for processing both numerosity and time.