The fifth generation (5G) networks are characterized with ultra-dense deployment of base stations with limited footprint. Consequently, user equipment’s handover frequently as they move within 5G networks. In addition, 5G requirements of ultra-low latencies imply that handovers should be executed swiftly to minimize service disruptions. To preserve security and privacy while at the same time maintaining optimal performance during handovers, numerous schemes have been developed. However, majority of these techniques are either limited to security and privacy or address only performance aspect of the handover mechanism. As such, there is need for a novel handover authentication protocol that addresses security, privacy and performance simultaneously. This paper presents a machine learning protocol that not only facilitates optimal selection of target cell but also upholds both security and privacy during handovers. Formal security analysis using the widely adopted Burrows–Abadi–Needham (BAN) logic shows that the proposed protocol achieves all the six formulated under this proof. As such, the proposed protocol facilitates strong and secure mutual authentication among the communicating entities before generating the shares session key. The derived session key protected the exchanged packets to avert attacks such as forgery. In addition, informal security evaluation of the proposed protocol shows that it offers perfect forward key secrecy, mutual authentication any user anonymity. It is also demonstrated to be robust against attacks such as denial of service (DoS), man-in-the-middle (MitM), masquerade, packet replays and forgery. In terms of performance, simulation results shows that it has lower packets drop rate and ping–pong rate, with higher ratio of packets received compared with improved 5G authentication and key agreement (5G AKA’) protocol. Specifically, using 5G AKA’ as the basis, the proposed protocol reduces the handover rate by 94.4%, hence the resulting handover signaling is greatly minimized.