The low temperature phase of spin glasses exhibits algebraically decaying spin-spin correlations at all T<Tc. This permanent criticality is an important hallmark of many glassy systems. It is closely related to the existence of a soft gap in the distribution of the local magnetic field and implies the presence of collective low energy excitations. At T=0, when the magnetization is traced as a function of external field H, field-induced spin flips can trigger an avalanche of further spin flips. Here, criticality is non-trivially encoded in the power law distribution of the size of such events. I will present an analytical theory for this phenomenon in the case of the SK spin glass, and discuss analogies with other glassy systems such as pinned elastic systems and electron glasses.