The 28-year-old missed the dramatic penalty shootout victory over Colombia on Tuesday night that sent England into the quarter-finals for the first time in 12 years, having earlier been given permission by manager Gareth Southgate to attend the birth of his child.
"I've just experienced the most amazing 24 hours," Delph said in a social media post on Wednesday.
"Watching my brothers come through a tough game both mentally and physically, I kicked every ball and felt every bit of emotion with the players, staff and fans.
"At 07:52 Me my wife and 2 beautiful daughters welcomed their baby sister into the world. I can't put into words the happiness and gratitude I'm feeling... Back to Russia now."
Delph, who can play in midfield and on the left side of defence, has made two appearances in Russia so far, coming on as a second-half substitute in England's 6-1 win over Panama and starting the 1-0 defeat by Belgium in the group stages.
England will take on Sweden in the quarter-finals in Samara on Saturday.
(Reporting by Simon Jennings in St Petersburg, editing by Neil Robinson)
