In this paper we provide the mathematical theory for sensitivity analysis of order statistics of continuous random variables, where the sensitivity is with respect to a distributional parameter. Sensitivity analysis of order statistics over a finite number of observations is discussed before addressing sensitivity analysis of quantiles. In particular, we provide a central limit theorem and related confidence intervals for the proposed sensitivity estimators, and we improve the known convergence results for IPA. Our analysis provides guidelines for sensitivity analysis of order statistic related performance measures such as basic order-statistics or quantiles. Our findings are corroborated by a series of numerical examples from different areas of operations research: stochastic activity networks, queues, reliability, and financial engineering.