The main difference between Large Eddy Simulation (LES) and Implicit LES (ILES) is that in the latter no subgrid-scale model is used. Instead, the refinement of the mesh determines the length scales that are resolved and numerical dissipation acts as a model for the turbulent dissipation within the unresolved scales. In stabilized finite element approaches, this additional dissipation comprises a stabilization parameter and the strong residual of the momentum equation. This adds an additional layer of complexity, since both the stabilization parameter and the strong residual of the momentum equation depend on the time-step and the mesh resolution used within the solution. Consequently, ILES results depend on the mesh and simulation parameters. However, ILES simulations remain promising, since they do not require any model nor additional tuning parameter. In this work, we present a matrix-free stabilized Navier-Stokes solver within the framework of the Lethe open source CFD software~\cite{Blais}. This solver combines implicit BDF (order 1 to 4) schemes with a high-order (order 2 to 5) in space continous Galerkin formulation to solve the Navier-Stokes equations. The linear problems arising from the use of Newton's method are solved using a matrix-free GMRES solver which leverages geometric multigrid preconditioning~\cite{Munch}. The resulting scheme is unconditionally stable and scales well even for large problems ($>100$M cells). We investigate three canonical turbulent benchmark flows: the Taylor-Green vortex, the flow over periodic hills and the Taylor-Couette flow. Using these benchmarks, we assess the role of the time-step, mesh size and order of the spatial discretization on the predictions both in terms of local quantities (e.g., detachment/reattachment point, Reynolds stresses) and global quantities (e.g., energy dissipation). A main result derived from this work is the demonstration that accurate ILES results can be obtained for relatively high CFL ($>10$) when the mesh is fine, thus justifying the need for a fully implicit scheme.