We introduce Neural Circuit Theory (NCT), a mathematical framework which bridges neuroscience, deep learning, and electrical circuit theory. We show how NCT can describe biological neural circuits and leads to physical formulations of bio-plausible algorithms for credit assignment, such as Equilibrium Propagation and Difference Target Propagation. We show how these formulations can lead to quadratic speedups in the inference and training speed of energy-based models as well as estimation of curvature information in feedforward networks. Finally, we discuss the geometric structure that is embedded in NCT, which naturally contains information about the topology of the network.