Reduced order modelling for parametrized partial differential equations (PDEs) is an ac- tive area of research, where neural networks have emerged as usefool tools since they offer universal function approximation and efficient treatment of high-dimensional inputs. Based on the paradigm of evolutional deep neural networks (EDNNs), we develop a novel method to efficiently solve parametric time-dependent PDEs, which obtains the solution for all parameter instances from a single time-integration. This creates a promising surro- gate model for use in many-query applications such as uncertainty quantification, inverse problems and optimisation, especially in cases where reduced bases methods fail to obtain efficient surrogates. EDNNs represent the PDE solution as the output of a neural network and use a time- stepping scheme based on the PDE residual to evolve the network parameters. We discuss solving the EDNN update equation with a Krylov solver, which avoids explicit assembly of Jacobians and enables scaling to larger neural networks and thus more complex problems. We further propose a modified linearly implicit Rosenbrock method, which significantly alleviates the time step requirements of stiff PDEs. We particularly focus on extending EDNNs for problems of realistic complexity, by using positional embeddings that can encode domains with geometrical features and automati- cally enforce boundary conditions of Dirichlet and Neumann type on the predicted PDE solution fields. Connections between the eigenfunctions of the Laplace Beltrami operator and Fourier Features are drawn to explain the success of the positional embedding layer. We further showcase how automatically encoding invariances into the neural network can simplify the loss terms and therefore speed up the learning process, which can also be exploited in the training of physics informed neural networks (PINNs). We showcase our method on the Korteweg-de Vries equation, and several parametrized PDEs, including a nonlinear heat equation, advection-diffusion problems on domains with holes and 2D Navier Stokes flow. We particularly highlight the challenges in balancing accuracy and computational time and why EDNNs are promising surrogate models for parametrized PDEs with slow decaying Kolmogorov n-width.