Optimization of airborne collision avoidance algorithm based on resource-constrained reinforcement learning

Аuthors

Neretin E. S.^1*, Zuocheng L. ²

1. Integration center branch of the Irkut Corporation, 5, Aviazionny pereulok, Moscow, 125167, Russia
2. Northwestern Polytechnical University, 710072, 127, West Youyi Road, Beilin District, Xi'an Shaanxi, P.R.China

*e-mail: evgeny.neretin@ic.yakovlev.ru

Abstract

As air traffic density continues to rise with the advancement of aviation technology, the demand for efficient and reliable airborne collision avoidance systems becomes increasingly urgent. Traditional systems, such as the Traffic Collision Avoidance System (TCAS), mainly rely on heuristic rules and parameter settings, which, although effective in maintaining safety, struggle to adapt and optimize under the complexities of modern aviation environments. To address these limitations, we explore the application of reinforcement learning (RL) to optimize the performance of collision avoidance systems. We define the problem within a resource-constrained Markov decision process (RC-MDP) framework, incorporating virtual resource management to control the frequency of nuisance alerts, which are frequent alarms that do not require actual evasive action. We propose a novel time-resource bonus (TRB) mechanism to modify and enhance two standard RL algorithms, DQN and SAC, into DQNTRB and SACTRB. This approach encourages resource-efficient actions while maintaining collision avoidance performance. Our experimental results demonstrate that these modified algorithms significantly reduce nuisance alerts while achieving near-equivalent collision avoidance performance compared to algorithms without resource constraints.

Keywords:

pilot respond, deep reinforcement learning, airborne collision avoidance, markov decision-making process, dynamic programming

References

1. Holland J.E., Kochenderfer M.J., Olson W.A. Optimizing the next generation collision avoidance system for safe, suitable, and acceptable operational performance // Air Traffic Control Quarterly. 2013. Vol. 21, no. 3. P. 275–297. 
2. De D., Sahu P.K. A survey on current and next generation aircraft collision avoidance system // International Journal of Systems, Control and Communications. 2018. Vol. 9, iss. 4. P. 306–337.
3. Kochenderfer M.J., Holland J.E., Chryssanthacopoulos J.P. Next generation airborne collision avoidance system // Lincoln Laboratory Journal. 2012. Vol. 19, iss. 1. P. 17–33.
4. Kochenderfer M.J., Chryssanthacopoulos J.P. Robust airborne collision avoidance through dynamic programming : technical report / Massachusetts Institute of Technology, Lincoln Laboratory. 2011, 130 p. Project report ATC-371.
5. Optimized airborne collision avoidance / M.J. Kochenderfer, C. Amato, G. Chowdhary et al. // Decision making under uncertainty: theory and application. MIT Press, 2015. P. 249–276.
6. Julian K.D., Kochenderfer M.J., Owen M.P. Deep neural network compression for aircraft collision avoidance systems // Journal of Guidance, Control, and Dynamics. 2019. Vol. 42, iss. 3. P. 598–608.
7. Julian K.D., Kochenderfer M. Guaranteeing safety for neural network-based aircraft collision avoidance systems // 2019 IEEE/AIAA 38th Digital Avionics Systems Conference (DASC). DOI 10.1109/DASC43569.2019.9081748.
8. Li S, Egorov M., Kochenderfer M. Optimizing collision avoidance in dense airspace using deep reinforcement learning // Thirteenth USA/Europe Air Traffic Management Research and Development Seminar (ATM2019). 2019. DOI 10.48550/arxiv.1912.10146.
9. Online multiple-aircraft collision avoidance method / P. Zhao, W. Wang, L.Ying et al. // Journal of Guidance, Control, and Dynamics. 2020. Vol. 43, iss. 2. P. 1–17. DOI 10.2514/1.G005161.
10. A partially observable multi-ship collision avoidance decision-making model based on deep reinforcement learning // K. Zheng, X. Zhang, C. Wang et al. // Ocean & Coastal Management. 2023. Vol. 242. Art. 106689.
11. Kormushev P., Calinon S., Caldwell D.G. Reinforcement learning in robotics: Applications and real-world challenges // Robotics. 2013. Vol. 2, iss. 3. P. 122–148.
12. An aircraft collision avoidance method based on deep reinforcement learning / Z. Liu, E. Neretin, X. Gao, et al. // 9th International Conference on Control and Robotics Engineering (ICCRE), IEEE. 2024. P. 241–246.
13. Bhatia A., Varakantham P., Kumar A. Resource constrained deep reinforcement learning // Proceedings of the international conference on automated planning and scheduling. 2019. Vol. 29. P. 610–620.
14. Efficient exploration in resource-restricted reinforcement learning / Z. Wang, T. Pan, Q. Zhou et al. // Proceedings of the AAAI Conference on Artificial Intelligence. 2023 Vol. 37, no. 8. P. 10279–10287.
15. Sutton R.S., Barto A.G. Reinforcement learning: An introduction. Cambridge : MIT press, 2018.
16. Playing atari with deep reinforcement learning / V. Mnih, K. Kavukcuoglu, D. Silver, et al. // ArXiv.org : website / arXiv preprint arXiv:1312.5602: 2013. 9 p.
17. Human-level control through deep reinforcement learning / V. Mnih, K. Kavukcuoglu, D. Silver et al. // Nature. 2015. Vol. 518(7540). P. 529–533.
18. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor / T. Haarnoja, A. Zhou, P. Abbeel et al. // International conference on machine learning. PMLR, 2018. P. 1861–1870.
19. Mohamed S., Jimenez Rezende D. Variational information maximisation for intrinsically motivated reinforcement learning // Advances in neural information processing systems. 2015. 28 p.

Download