We propose a microscopic magneto-electric model in which the coupling between spins and electric dipoles is mediated by lattice distortions. The magnetic sector is described by a spin S=1/2 Heisenberg model coupled directly to the lattice via a  standard spin-Peierls term and indirectly to the electric dipole variables via the distortion of the surrounding electronic clouds. Electric dipoles are described by Ising variables for simplicity, as tunneling quantum effects do not substantially modify the results. We show that the effective magneto-electric coupling which arises due the interconnecting lattice deformations is quite effcient in one-dimensional systems. More precisely, we show using bosonization and extensive DMRG numerical simulations that increasing the magnetic field above the spin Peierls gap, a massive polarization switch-off occurs due to the proliferation of soliton pairs. We also analyze the effect of an external electric field E when the magnetic system is in a gapped (plateau) phase and show that the magnetization can be electrically switched between clearly distint values. We compare these findings with similar effects that have been recently observed in different materials. More general quasi-one-dimensional models and two-dimensional systems are also discussed. We finally propose a possible device (MERAM) based on materials with the behavior described by the present model.