Transformers vs CNNs: Enhancing Semantic Segmentation for Martian Terrain Classification

Building on our previous work with CNN-based architectures for Mars rover terrain classification, this paper investigates transformer-based semantic segmentation models. We benchmark three state-of-the-art transformers - SegFormer, Mask2Former, and UPerNet - against our bestperforming CNN baseline, DeepLabV3+, using two Mars terrain datasets: AI4Mars and LabelMars. Evaluation metrics include pixel-level accuracy, mean Intersection over Union (mIoU), model parameter count, and inference speed (frames per second), among others. Our findings show that transformer models match or slightly outperform DeepLabV3+ on the smaller LabelMars dataset, despite limited training data. On the larger AI4Mars dataset, transformers significantly outperform DeepLabV3+, particularly in detecting rare but safety-critical terrain features such as the Big Rock obstacle class. However, DeepLabV3+ demonstrates better performance at lower image resolutions, highlighting the importance of resolution in fully leveraging the self-attention mechanisms of transformer-based models. To improve model robustness and mitigate overfitting, we incorporate semi-supervised learning using additional unlabeled AI4Mars images. In our efficiency analysis, SegFormer stands out by achieving the highest real-time inference throughput (FPS) while maintaining competitive mIoU, making it a strong candidate for onboard deployment - where both accuracy and computational efficiency are essential for autonomous navigation in future Martian missions. We further demonstrated the feasibility of multi-scale inference, showing that models trained at higher resolutions retain strong performance when deployed at lower resolutions, enabling efficient onboard computation. Overall, our results highlight the potential of transformerbased segmentation models to significantly enhance Mars rover autonomy by combining high segmentation accuracy with the real-time efficiency required for onboard systems.