Peatlands can be classified into fens and bogs based on their hydrology. Development of fens to bogs is accompanied by the invasion of Sphagnum species. The purpose of this study was to determine how the decomposition process in fens is influenced by the transition from a vascular plant-dominated system to a Sphagnum-dominated system. We carried out a reciprocal litter bag experiment, using litter of Carex diandra, C. lasiocarpa, Sphagnum papillosum and S. squarrosum in a fen dominated by Sphagnum species and a fen without Sphagnum. Decomposition rate and nitrogen and phosphorus dynamics of the plant litter were measured in a field experiment for two years. Decomposition rate was highest for the Carex litter types and lowest for the Sphagnum litter types. Surprisingly, decomposition rates hardly differed between the two sites. Nutrient dynamics, however, showed a clear site-effect: In the Sphagnum site net mineralization was observed for all litter types whereas in the Carex site net immobilization was observed. These results show that carbon and nutrient cycles were coupled in a different way in a Sphagnum-dominated and a Carex-dominated site, respectively. Nutrient availability and adaptation of the microbial community to nutritional and other environmental conditions may be the main regulators of carbon and nutrient cycles in these peatlands.