The prohibitive computational cost of solving partial differential equations has led to the development of neural operator networks that approximate the solution operator to a family of PDEs. While existing neural operator architectures can produce accurate approximations to PDE solutions, their success in replacing classical numerical methods is modest due to a number of factors, including their violation of known conservation laws and symmetries, and a lack of generalisability to data outside the training distribution. Existing work on Green's functions has resulted in a rich theory on exact solution operators to PDEs, and finite element theory provides many results regarding numerical approximations to such solution operators. By designing neural operators such that their structure mimics that of numerical methods and/or Green's functions, it is possible to learn operators that have symmetries and other relevant properties embedded in their design. Additionally, existing numerical methods provide information on how a neural operator can generalise over PDE parameters and even domain geometry, by showing how to input this data to the network in a natural way. Furthermore, if the neural operator is designed to produce an approximation to the Green's function of the corresponding PDE, it can be used for more than just a fast surrogate model. For example, learned approximations to the Green's function can be used to construct preconditioners for high-accuracy discretisations of the same PDE.