The reconstruction of complex time-evolving fields from a small number of sensor observations is a grand challenge in a wide range of scientific and industrial applications. Frequently, sensors have very sparse spatial coverage reporting noisy observations from highly non-linear phenomena. While numerical simulations can model some of these phenomena using partial differential equations, the reconstruction problem is not well-posed, therefore data-driven strategies can be used to provide crucial disambiguation. Nevertheless, data-driven approaches, like large neural networks, suffer in cases with small amounts of data and can struggle to handle very large domains, due to memory requirements. To overcome these hurdles, we present the Senseiver, an attention-based framework that excels in the task of reconstructing complex spatial fields from a small number of observations. The Senseiver reconstructs complex n-dimensional fields by encoding arbitrarily-sized sparse sets of inputs into a latent space using cross-attention, which produces uniform-sized outputs regardless of the number of observations. This same property allows very efficient training as a consequence of being able to decode a sparse set of output observations. This enables effective training of data with complex boundary conditions (continents in sea temperature data) and to extremely large and fine-scale simulations (3D porous media). We show that the Senseiver sets a new state of the art for four existing datasets, including real-world temperature observations. We further push the bounds of sparse reconstructions on a new large-scale simulation of two fluids flowing through a 3D porous domain with complex solid boundaries.