Motivated by the study of disordered superconductors, we develop an analytical theory, based on the quantum cavity method, describing the quantum phase transitions of disordered systems. A connection to the classical problem of directed polymers in random media shows the relevance of replica symmetry breaking effects. Because of these effects, various aspects of the zero temperature phase transition are very different from what is found in usual phase transitions, and strong inhomogeneities play a crucial role.