Doctors say oversleeping can reduce fertility by more than 40 per cent.
And they believe it might be linked to the male hormone testosterone, which is released into the body during sleep.
But they warn that getting too little rest is just as bad.
Their study of almost 700 couples found men sleeping under six hours a night had a similar drop in the chance of their partner getting pregnant each month as those who slept too much.
Even men who only had some restless nights suffered an impact on their fertility.
Dr Lauren Wise from Boston University said: “Those who sleep seven to nine hours a night have the lowest risk of adverse health outcomes.”
All the couples in the study had been trying for a baby for up to six months.
Men recalled their sleep patterns for two weeks and their partners filled in a questionnaire every eight weeks until they fell pregnant.
Researchers said they did not know why odd sleep patterns reduced fertility.
But they said it wasn’t because these men are too tired for sex, too fat or smoke – all of which can affect fertility.
And British fertility expert Prof Allan Pacey said it would be wrong to blame sleepy men.
He said: “If the reason they need a lot of sleep is because they have a specific metabolism that requires that, you are not going to change that by making them get up earlier, you will just change them into a grumpy person.”
Previous studies have found that men who don’t sleep well have poorer sperm.
Other studies have shown that sleeping less than four hours a night can increase the risk of dying from heart problems by more than a third.
Prof Pacey added that sperm have receptors for the sleep hormone melatonin on them, but scientists do not know why. He says more research into sleep and fertility is needed.
“It must be something to do with testicles or hormones in the brain that make testicles work. It’s not too difficult a system to understand.”
The research was presented at the American Society for Reproductive Medicine conference [must keep] in Salt Lake City.