In the classical finite element methods, the generation of a conforming tessellation of the computational domain constitutes a major bottleneck in the simulation pipeline. The finite cell method seeks to circumvent the generation of a boundary-conforming mesh through the embedding of the physical domain in a regular background mesh. The overlap between the physical domain and the background computational domain, in turn, leads to the numerical ill-conditioning of the resultant system of equations, hampering the usability of iterative solvers out of the box. Therefore, the development of specialized solvers is a crucial aspect of obtaining a scalable solution strategy for the finite cell method. In this work, we present adaptive geometric multigrid (GMG) solvers with tailored smoothers as an efficient and scalable choice for the finite cell method. We discuss the computational complexity and parallel scalability of the solvers for large-scale problems on distributed-memory clusters using numerical benchmarks. Furthermore, three cache policies for the smoothers is presented that offer a balance between on-the-fly and cached computation, and the memory footprint and computational efficiency of each one is discussed. Results indicate that the presented solvers are effective for the solution of finite cell systems and can achieve excellent strong and weak scalability in parallel for problems with more than a billion degrees of freedom.