Electron Probe Microanalysis (EPMA) is a non-destructive technique to determine the chemical composition of material samples in the micro- to nanometer range. Targeted electron beam on the surface of a sample generate atomic excitations of the material which relax by emitting x-rays. Based on intensity measurements of the radiation, the reconstruction of the chemical composition an inverse problem. Current methods assume a homogeneous or a layered structure of the sample material. To increase the spatial resolution of reconstruction in EPMA the combination of a more sophisticated reconstruction method is required, which is based on a forward model that allows complex material structure, together with multiple measurements with varying beam configurations. We present a deterministic model, that is based on the continuous slowing down model of the radiative transfer equation for electron transport and uses the PN moment approximation for efficient computations. This model allows to approximate the least-squares solution of the inverse problem using gradient-based optimization. We detail the application of the PN-moment-systems in a differentiable programming context, in particular by deriving its adjoint formulation. Differentiable programming provides the flexibility to adapt the reconstruction method to various material parametrizations and thus to regularize and take into account prior knowledge. Through various examples based on realistic sample scenarios, we verify our implementation and demonstrate the flexibility of the reconstruction.