Virtual Production for Hardware-in-the-Loop Vision-based Orbital Navigation Experiments

Hardware-in-the-loop (HIL) testing is essential to validate vision-based orbital navigation algorithms. Increasingly, these algorithms leverage deep learning, which offers superior performance when trained on large datasets. However, discrepancies between the training data and the real in-orbit environment (e.g. geometry, color, textures and lighting), can cause the deep learning algorithm’s performance to degrade, affecting navigational accuracy. For robust validation of these algorithms, a testbed capable of realistically simulating both the foreground satellite and the complex space background –including stars, Earth, the Sun and other in-orbit elements (e.g. space stations) - is paramount. Existing testbeds have employed chroma-keying and CAD-model-based masking to composite such backgrounds. However, these techniques introduce limitations such as imperfect masking artifacts, inaccurate background integration (e.g. not being observed in reflections) and challenges of masking in the presence of optical lighting artifacts like lens flares. To address these issues and enable accurate HIL validation, we explore the application of virtual production for background simulation. Our analysis shows that this is a viable direction and offers several advantages for simulating in-orbit environments over existing techniques. Virtual production for HIL validation of vision-based orbital navigation algorithms has been underexplored, and our aim is to advance adoption of this technology in orbital testbeds.