The fifth generation (5G) of wireless networks is currently under investigation in order to address the well-known challenges of the high capacity demands and traffic volume. The promising solutions to meet these targets can be achieved through ultra-densification, efficient use of spectrum and advanced filtered modulation techniques. In this paper, we present an enhanced MAC protocol for 5G small cells operating at 5 GHz and assuming an FBMC physical layer. The proposed MAC design consists of scheduled-based and contention-based access schemes and involves a listen before talk (LBT) procedure to comply with ETSI regulations. The performance of the proposed FBMC-MAC design is then evaluated in dense deployment scenarios under different PHY/MAC parameter settings. Moreover, we study the performance of FBMC-MAC systems in the context of coexistence with WiFi systems. © 2018, ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering.