Finite element nonoverlapping Domain Decomposition Methods (DDMs) based on the Schwarz method represent a promising iterative method for addressing for addressing large-scale electromagnetic wave problems in the frequency domain. Their convergence primarily depends on the employed transmission condition (TC) between the sub domains. The typically used TCs are based on free space open boundary conditions and face issues when applied to guided wave problems: higher order variants struggle with convergence around cutoff frequencies, while zeroth order TCs fail for evanescent waves at interfaces completely. In this contribution, we introduce a novel TC using a modal decomposition at waveguide interfaces, referred to as the modal TC (MTC). This TC accurately takes into account the physics of the waves travelling through the interfaces. Thus it leads to superior convergence when applied to waveguide-like structures, common in high-frequency components like RF cavities and particle accelerators. The method is validated through application on diverse models, starting from a homogeneous waveguide model up to an entire undulator with a length of 4.6 m at several GHz. These validations underline the MTC's suitability and robustness, while outperforming existing TCs. Looking ahead, our focus will be on a hybrid TC combining the MTC with the classical TCs, thus reducing its cost when many modes have to be considered. This development aims to extend the method's utility, making it a valuable tool for more precise and efficient electromagnetic field analysis.