Analyzing the consequences of the quantum geometry induced by the momentum dependence of Bloch states has emerged as a very rich and active field in condensed matter physics. For instance, for the superfluid stiffness or the pairing mechanism, these geometric aspects can play an important role. We here demonstrate that quantum geometry can also be essential for the disorder sensitivity of a superconductor, in particular when time-reversal symmetry is broken in the normal-state Bloch Hamiltonian. We derive a general expression for the behavior of the critical temperature Tc involving weighted (anti-)commutators of the superconducting order parameter and impurity matrix elements, which includes both wave-function effects and kinetic pair breaking due to broken time-reversal symmetry in the dispersion. We analyze how the former effects lead to "quantum geometric pair breaking", where any superconductor becomes susceptible to microscopically non-magnetic impurities, and formally relate it to the maximum possible localization of two-particle Wannier states. Further, in the presence of kinetic pair breaking, impurities can also enhance pairing, leading to an overall more complex, non-monotonic behavior of Tc with impurity concentration. We also analyze the fate of finite-momentum pairing. Our results are directly relevant to rhombohedral graphene, twisted MoTe2, and superconducting altermagnets.