A consensus on Bantu-speaking populations being genetically similar has emerged in the last few years, but the demographic scenarios associated with their dispersal are still a matter of debate. The frontier model proposed by archaeologists postulates different degrees of interaction among incoming agro-pastoralist and resident foraging groups in the presence of ‘static’ and ‘moving’ frontiers. By combining mtDNA and Y chromosome data collected from several Southern African populations, we show that Bantu-speaking populations from regions characterised by a moving frontier developing after a long-term static frontier have larger hunter-gatherer contributions than groups from areas where a static frontier was not followed by further spatial expansion. Differences in the female and male components suggest that the process of assimilation of the long term resident groups into agro-pastoralist societies was gender-biased. Our results show that the diffusion of Bantu languages and culture in Southern Africa was a process more complex than previously described and suggest that the admixture dynamics between farmers and foragers played an important role in shaping the current patterns of genetic diversity.