The former England defender has failed to win eight league matches since being appointed manager of the La Liga club last month.
"I am not going to sleep well tonight," Neville told reporters.
"This is one of the worst experiences I've ever gone through in football."
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
Neville was handed his first managerial job in December, working alongside his brother Phil, and has been the subject of mounting criticism from Valencia fans with "#Nevilleveteya", translated as Neville go now, trending on Twitter.
Neville, 40, enjoyed a trophy-laden career as Manchester United right back before becoming a highly respected television pundit with Sky.
"I last doubted myself 18 years ago and after that I developed coping mechanisms to deal with situations," he said.
"I'm fine."
(Reporting by Ed Osmond in London; Editing by Ian Ransom)
