TY - GEN
T1 - HMAAC
T2 - 2023 IEEE International Conference on Robotics and Automation, ICRA 2023
AU - Sun, Chuanneng
AU - Huang, Songjun
AU - Pompili, Dario
N1 - Funding Information:
This work is supported by the NSF RTML Award No. CCF-1937403. The authors are with the Dept. of Electrical and Computer Engineering, Rutgers University–New Brunswick, NJ, USA. Emails: {chuanneng.sun, songjun.huang, pompili}@rutgers.edu
Publisher Copyright:
© 2023 IEEE.
PY - 2023
Y1 - 2023
N2 - Unmanned Aerial Vehicles (UAVs) have become prevalent in Search-And-Rescue (SAR) missions. However, existing solutions to the control and coordination of UAV s are mostly limited to specific environments and are not robust to handle unreliable/unstable communications. To deal with these challenges, Hierarchical Multi-Agent Actor-Critic (HMAAC) framework is proposed where a high-level policy is placed on top of individual low-level actor-critic policies to relax the inter-dependency among the agents. The low-level policies are considered conditionally independent given the coordination action, which is generated by the high-level policy. A Central-ized Training Decentralized Execution (CTDE) would not work because it cannot be assumed that communication is always perfect during training and that the whole system can rely on stable communications during deployment. The proposed framework is evaluated in AirSim, a realistic multi-UAV simula-tor, and is compared against two existing algorithms, i.e., Multi- Agent Actor-Critic (MAAC) and decentralized REINFORCE, in two scenarios, (a) when packet drop is modeled as a Bernoulli process and (b) when shadow zones are created in the search space and communication will be lost if the agents are in these zones. Results show that HMAAC is scalable and robust to unreliable communication and outperforms the other algorithms in terms of exploration and coordination when the number of agents is large and communications are not stable.
AB - Unmanned Aerial Vehicles (UAVs) have become prevalent in Search-And-Rescue (SAR) missions. However, existing solutions to the control and coordination of UAV s are mostly limited to specific environments and are not robust to handle unreliable/unstable communications. To deal with these challenges, Hierarchical Multi-Agent Actor-Critic (HMAAC) framework is proposed where a high-level policy is placed on top of individual low-level actor-critic policies to relax the inter-dependency among the agents. The low-level policies are considered conditionally independent given the coordination action, which is generated by the high-level policy. A Central-ized Training Decentralized Execution (CTDE) would not work because it cannot be assumed that communication is always perfect during training and that the whole system can rely on stable communications during deployment. The proposed framework is evaluated in AirSim, a realistic multi-UAV simula-tor, and is compared against two existing algorithms, i.e., Multi- Agent Actor-Critic (MAAC) and decentralized REINFORCE, in two scenarios, (a) when packet drop is modeled as a Bernoulli process and (b) when shadow zones are created in the search space and communication will be lost if the agents are in these zones. Results show that HMAAC is scalable and robust to unreliable communication and outperforms the other algorithms in terms of exploration and coordination when the number of agents is large and communications are not stable.
UR - http://www.scopus.com/inward/record.url?scp=85168656113&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85168656113&partnerID=8YFLogxK
U2 - 10.1109/ICRA48891.2023.10161019
DO - 10.1109/ICRA48891.2023.10161019
M3 - Conference contribution
AN - SCOPUS:85168656113
T3 - Proceedings - IEEE International Conference on Robotics and Automation
SP - 7728
EP - 7734
BT - Proceedings - ICRA 2023
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 29 May 2023 through 2 June 2023
ER -