The new computational model, based on DNA samples from modern humans in France and China, shows successful coupling happened at a rate of less than two per cent.
The research suggests either inter-species sex was very taboo or the hybrid offspring had trouble surviving, according to the findings in the Proceedings of the National Academy of Sciences.
There may have been "extremely strong barriers to gene flow between the two species because of a very low fitness of human-Neanderthal hybrids, a very strong avoidance of interspecific mating, or a combination," said the study by researchers at the University of Geneva and the University of Berne in Switzerland.
Between two and four per cent of the human genome can be linked to the long-extinct Neanderthals and their cavemen relatives.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
The squat, low-browed Neanderthals lived in parts of Europe, Central Asia and the Middle East for up to 300,000 years but all evidence of them disappears some 40,000 years ago, their last known refuge being Gibraltar.
Why they died out is a matter of some debate, because they co-existed alongside modern man.
A study by French researchers published in the journal, Science, last month suggested that modern humans gleaned a competitive immune advantage from their liaisons with cavemen.
However, scientists still have no evidence to suggest the nature of those sexual encounters, whether violent or consensual.
Previous studies have also suggested that Neanderthals were crowded out by modern humans, and that the death blow to their species may have been accelerated by a spate of harsh, wintry weather.
