Adaptive Formation Tracking of Nonlinear Heterogeneous Multi-Agent Systems Using Transformer-Based Reinforcement Learning

Abstract

This paper proposes an adaptive formation tracking framework for nonlinear heterogeneous multi-agent systems (MAS) that couples a Transformer-based reinforcement learning policy with a Lyapunov-guided online adaptation law. Each agent encodes its own state and the variable-sized set of neighbor observations as a token sequence processed by a multi-head self-attention encoder, producing permutation-invariant action embeddings under switching communication topologies. An actor–critic proximal policy optimization (PPO) update is combined with a composite reward that shapes closedloop dynamics, while a projection-based parameter update compensates unknown drift and disturbance terms of each heterogeneous agent. We prove that the closed-loop formation error is uniformly ultimately bounded and converges to an arbitrarily small residual set under mild connectivity and regularity assumptions. Extensive simulations on quadrotor, wheeled-robot, and mixed-fleet benchmarks demonstrate that the proposed controller reduces steady-state tracking error by up to 41% relative to model-reference adaptive control and by 23% relative to MAPPO, while lowering per-step communication cost by nearly 30%. Ablation studies confirm that self-attention and adaptive updates are complementary, and the learned policy generalizes to unseen team sizes and time-varying leader trajectories without retraining, supporting deployment on real-world cooperative robotic fleets.