Research/ ORBITAL ROBOTICS
Reinforcement Learning On-Policy Multi-Agente per la Guida e il Coordinamento di Formazioni Satellitari in Orbita Cislunare
ANDREA SODINI
Learning-based MethodsOpenAlex topic: Reinforcement Learning in Robotics · Multi-Robot & SwarmsOpenAlex keyword: multi-agent reinforcement learning · Rendezvous, Proximity Ops & DockingOpenAlex keyword: rendezvous
Lunar Gatewaynamed in abstract, keywords · Artemisnamed in abstract
The growing shift in space agencies' focus toward the lunar environment, driven by programs such as Artemis and infrastructure such as the Lunar Gateway, has made the study of missions involving multiple spacecraft cooperating in formation on non-Keplerian orbits increasingly relevant — typically Halo or Near-Rectilinear Halo orbits about the Lagrange points of the Earth-Moon system. In this context, proximity operations such as rendez-vous, scene-visibility maintenance, and formation network-connectivity maintenance are complicated by the nonlinear nature of the reference dynamics, governed by the Circular Restricted Three-Body Problem (CR3BP). Classical guidance, navigation, and control techniques rely on explicit models and optimization methods solved point-by-point at every planning cycle, incurring significant online computational cost and limited robustness to operating conditions that differ from those assumed at design time; the reference work by Pilone addressed this same mission scenario using precisely this approach, both centralized and decentralized. This thesis explores an alternative paradigm, on-policy Reinforcement Learning, in which an agent learns a control policy through repeated interaction with a simulated environment, shifting the computational burden from the operational phase to the offline training phase; among on-policy algorithms, Proximal Policy Optimization (PPO) is chosen for its training stability and its natural applicability to discrete action spaces. The case study is a four-satellite formation consisting of a passive target, a chaser, an observer, and a communication satellite, positioned in the vicinity of a Southern Halo L2 orbit in the Earth-Moon system, consistent with the Lunar Gateway architecture. The work first develops the necessary theoretical framework: the CR3BP equations of motion in the synodic frame, the Lagrange points, periodic Halo orbits, and, finally, the theory of quasi-periodic orbits and invariant tori that emerge through bifurcation from the Halo family, which underlies the third methodological variant. The relative dynamics in the Local Vertical Local Horizon (LVLH) frame are then introduced, along with the Lambert problem, solved via shooting within the full CR3BP dynamics, which constitutes the tool with which every single-impulse maneuver is generated and applied to the agents. On the RL methodology side, the thesis reviews Markov Decision Process theory, policy-gradient and actor-critic methods, PPO's clipped surrogate objective, and the main multi-agent architectures proposed in the literature; the adopted choice is a sequential curriculum learning scheme, in which the three agents are first trained in isolation and then jointly, as a compromise between implementation simplicity and multi-agent training stability. The central methodological contribution is the development and comparison of two simulation-environment variants that share the observation space, action space, potential-shaping-based reward function, and a five-phase training curriculum with convergence gates, differing in the time-stepping strategy: one with a fixed integration step dt = 30 minutes, and one with an adaptive step, in which the timestep decreases as a function of chaser-target distance. The thesis then extends the fixed-step environment to a third scenario, in which the free-space geometric goals of the observer and comm are replaced by anchoring to an invariant quasi-halo torus of the CR3BP dynamics; this variant introduces a structural constraint on the action space, consisting of phase-hop and shell-change templates, complementary to the more traditional soft constraint in the reward. The three configurations were evaluated on a sample of N = 10,000 independent episodes under a deterministic policy. The rendez-vous success rate was 96.8% for the fixed step, 93.6% for the adaptive step, and 99.2% for the quasi-halo variant. The direct comparison between fixed and adaptive step showed a reduction in Δv consumption for all active agents, at the cost of a slightly lower success rate. The quasi-halo variant showed a markedly lower formation-keeping energy cost for the observer and comm compared to the free-space variants, a result achieved only after abandoning an initial kinematic model of the torus and recalibrating the reward function relative to the free-space cases. The comparison with Pilone's explicit optimization approach confirmed the expected online computational advantage of the learned policy. The thesis closes by discussing related future developments, including a comparison with more centralized multi-agent architectures (MAPPO, MADDPG), extending the adaptive timestep to the quasi-halo variant, and moving toward more complex but targeted models such as real ephemerides.