Over the past few years, a novel class of methods known as physics-informed machine learning (PIML) has emerged as a promising alternative to traditional numerical methods for solving partial differential equations (PDEs). These PIML models typically rely on variants of neural networks (NNs) or kernel methods such as Gaussian Processes (GPs). In this presentation, we introduce NNs with kernel-weighted corrective residuals (CoRes) which integrate the best of both worlds: the scalability and extrapolation power of NNs with the superior local generalization properties of kernel methods. We demonstrate that our approach can (1) exactly satisfy the prescribed boundary and initial conditions, thereby simplifying the optimization process, (2) address inverse problems by incorporating additional measurements in the kernel structure, and (3) solve complex PDEs such as the Navier-Stokes equations as well as operator learning tasks. Through a diverse set of benchmarks, we show that our framework consistently outperforms state-of-the-art PIML methods in terms of accuracy, robustness and development time.