The rise in autonomous Unmanned Aerial Vehicles (UAVs) for objectives requiring long-term navigation in diverse environments is attributed to their compact, agile, and accessible nature. Specifically, problems exploring dynamic obstacle and collision avoidance are of increasing interest as UAVs become more popular for tasks such as transportation of goods, formation control, and search and rescue routines. Prioritizing safety in the design of autonomous UAVs is crucial to prevent costly collisions that endanger pedestrians, mission success, and property. Safety must be ensured in these systems whose behavior emerges from multiple software components including learning-enabled components. Learning-enabled components, optimized through machine learning (ML) or reinforcement learning (RL) require adherence to safety constraints while interacting with the environment during training and deployment, as well as adaptation to new unknown environments. In this paper, we safeguard autonomous UAV navigation by designing agents based on behavior trees with learning-enabled components, referred to as Evolving Behavior Trees (EBTs). We learn the structure of EBTs with explicit safety components, optimize learning-enabled components with safe hierarchical RL, deploy, and update specific components for transfer to unknown environments. Safe and successful navigation is evaluated using a realistic UAV simulation environment. The results demonstrate the design of an explainable learned EBT structure, incurring near-zero collisions during training and deployment, with safe time-efficient transfer to an unknown environment.