Performance Comparison of PID and Reinforcement Learning Architectures in Non-Cooperative Relative Spacecraft Formations

As the problem of in-orbit debris and defunct satellites grows, researchers continue to look for more effective methods of facilitating satellite rendezvous and proximity operations. With the possibility of in-space manufacturing and assembly, this problem will only worsen. Rendezvous have historically been dominated by manual operations with growing use of visual servoing, model predictive, and automated learning approaches. This paper investigates the use and applications of Python-based “proportional, integral, and derivative” (PID) controller structures in orbital rendezvous and their relevance to emerging reinforcement learning approaches. The basis of this research is the Kerbal Space Program Differential Games (KSPDG) challenge. This challenge involves a few different scenarios, including a pursuit-evader problem where the pursuer tries to rendezvous with an inactive satellite. The challenge also involves a sun-blocking scenario, where the player-controlled craft attempts to position itself between a target and the sun to block direct sunlight. The simulator provides the velocity vectors in the x, y, and z axes, while constantly orienting the pursuer to point towards the evader. The program provides a scoring function based off minimizing distance, velocity, fuel, and time. The PID controller provides a baseline for later attempts using reinforcement learning methods. The PID controller contains inputs for the position and velocity vectors. The controller is fed constant velocity and position vectors from the player-controlled craft. When considering satellites with proximal, low earth orbits, the findings suggest that PID controllers provide reliable, repeatable results. Results suggest that lower approach speeds and closer rendezvous can be achieved by modifying the proportional gain of the given velocity vector components. Repeatable approaches within eight meters at speeds of twenty meters per second are achievable with a PID controller. For the sun-blocking scenario, the PID controller proves to be a practical and applicable method, suggesting that the same method can be used for both scenarios, and provides a performance comparison relative to a reinforcement learning approach (DeepQ).