Neville, who played 85 internationals, said the youthful England squad would have found it difficult to recover in Russia were it not for the late winner.
"You're on the back foot (if you don't win first)," Neville, who played in three World Cups, said on ITV.
"Unfortunately for England, you can't trust us. We've let ourselves down ... You cannot trust us to then recover from a bad performance or result in the first game. It's happened too many times.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
"So tonight was so big and we were all sat there watching that game thinking they have to score in this last two or three minutes or else Panama could see panic set in, anxiety set in."
The former Manchester United full back also praised Jordan Henderson for his ability to control the midfield under pressure and said the Liverpool captain may have cemented his place in the side.
"He wants to learn all the time, he works hard, he's conscientious, he's a manager's dream," Neville added. "And tonight — when he was under pressure to win the spot, he looks like he's cemented his spot."
(Reporting by Anirban Paul in Bengaluru, editing by Ed Osmond)
