Deep Reinforcement Learning (DRL) has shown significant promise for uncovering sophisticated control policies that interact in environments with complicated dynamics, such as stabilizing the magnetohydrodynamics of a tokamak reactor and minimizing the drag force exerted on an object in a fluid flow. However, these algorithms require many training examples and can become prohibitively expensive for many applications. In addition, the reliance on deep neural networks results in an uninterpretable, black-box policy that may be too computationally challenging to use with certain embedded systems. Recent advances in sparse dictionary learning, such as the Sparse Identification of Nonlinear Dynamics (SINDy), have shown to be a promising method for creating efficient and interpretable data-driven models in the low-data regime. In this work, we extend ideas from the SINDy literature to introduce a unifying framework for combining sparse dictionary learning and DRL to create efficient, interpretable, and trustworthy representations of the dynamics model, reward function, and control policy. We demonstrate the effectiveness of our approaches on benchmark control environments and challenging fluids problems, achieving comparable performance to state-of-the-art DRL algorithms using significantly fewer interactions in the environment and an interpretable control policy orders of magnitude smaller than a deep neural network policy.