In this dissertation, I theoretically investigate the open system dynamics of cavity optomechanical systems in the quantum limit, where a mechanical resonator couples to a cavity mode via radiation pressure force. Recent developments in nano-fabrication and cooling techniques make it possible to optically control and manipulate the motion of micro- and nano-mechanical resonator. Several experiments have demonstrated the possibility of preparing macroscopic mechanical resonators into their quantum ground state, which makes optomechanical systems an ideal candidate for studying the quantum behavior of macroscopic objects. One important direction in quantum optomechanics is the study of the decoherence of optomechanical system due to the interaction with their external environments. This dissertation explores the decoherence effects by considering the influence of the radiation pressure coupling on the interaction between optomechanical system and its environments. Because the effects of the nonlinear interaction vary with the optomechanical coupling strength, three different interaction regimes are fully studied here: the weak coupling, the intermediate coupling and the strong coupling.