Deep Reinforcement Learning for Multi-Agent Spacecraft Electromagnetic Formation Flight

Multi-agent architectures for space missions have become increasingly popular due to the operational benefits provided by the ability to fractionate functionality between agents and gracefully scale the number of agents up or down to meet mission needs. The control of multi-agent satellite swarms through the interaction of magnetic fields generated by electromagnetic coils, known as electromagnetic formation flight (EMFF), has been previously explored and offers operational benefits such as mitigated reliance on finite propellant and non-mechanical coupling. However, the highly-coupled, stiff, and chaotic dynamics of EMFF can make control challenging. Current state-of-the-art methods rely on solving the dipole inversion subproblem using a computationally expensive nonlinear program at each control iteration. In this work, we present an approximate reformulation of the dynamics which addresses some of their problematic aspects. Using our dynamics, we formulate EMFF swarm control as a Markov decision process and use deep reinforcement learning to obtain an end-to-end policy for control. We apply this policy to a heterogeneous swarm architecture for EMFF based around a prime spacecraft with auxiliary inertial propulsion modalities. To our knowledge, this work demonstrates the first learned solution to dipole inversion, providing a constant time evaluation for the control loop. Our trained policy achieves sub-meter accuracy in position control for a small formation of two worker spacecraft.