A substantial magnetodielectric effect is usually an indication of coupled magnetic and elastic order, such as is found in the multiferroics. However, here I will show that magnetism is not necessary to produce either a magnetoresistance or a magnetocapacitance when the material is inhomogeneous. From exact calculations and numerical simulations of metal-dielectric composites, I find a characteristic dielectric resonance that sensitively depends on magnetic field. I will discuss how this behaviour has already been observed experimentally in some materials.