The 23-year-old Tottenham Hotspur winger, who missed this year's World Cup with an ankle injury, has not been able to train since reporting for international duty on Monday.
"Andros Townsend will today (Thursday) leave the England squad and return to his club," the FA said.
England host Slovenia at Wembley in a Euro 2016 Group E qualifier on Saturday before taking on Scotland in a friendly at Celtic Park on Tuesday.
(Reporting by Michael Hann, editing by Ed Osmond)
