The rational design and realization of programmable materials and structures plays an important role in enabling functional devices in various applications, such as actuators, sensors, and robotics. In addition, the properties of metamaterials and structures typically remain fixed after being designed. This work introduces an optimization-driven approach to inverse design soft active metamaterials and structures, whose nonlinear properties can be precisely programmed and then altered by external magnetic fields. Those systems exhibit one response under purely mechanical loading, and switch to a distinctive response under simultaneous mechanical loading and applied magnetic field. With switchable nonlinear behaviors, multiple functionalities are enabled within a single design. We introduce a magneto-mechanical topology optimization formulation that simultaneously explores the full design space of topology and the remanent magnetization distributions of magnetic soft materials. With the optimization framework and hybrid fabrication, we design and fabricate various metamaterials and metastructures that realize switchable responses and tunable buckling behaviors, respectively. These alterable yet programmable mechanical responses are enabled by the interactions among unique geometry, large deformations, and magnetic actuation. The proposed optimization-driven computational design strategy can be utilized to design and realize multi-functional devices in various applications.