We formulate a quantum generalization of the notion of the group of Riemannian isometries for a compact Riemannian manifold, by introducing a natural notion of smooth and isometric action by a compact quantum group on a classical or noncommutative manifold described by spectral triples, and then proving the existence of a universal object (called the quantum isometry group) in the category of compact quantum groups acting smoothly and isometrically on a given (possibly noncommutative) manifold. Our formulation accommodates spectral triples which are not of type II. We give explicit description of quantum isometry groups of commutative and noncommutative tori, and in this context, observe an interesting quantum group structure on a C^* algebra which is a direct sum of commutative two-torus and irrational rotation algebra.