"The only blame I can place is with the club as a whole, from the board of directors, to the manager, to the coaches, to the players," Ellis told Sky Sports.
"In the end, it's the players that are really responsible. Unfortunately, they've not been good enough this season.
"It's now a certainty that (Aston Villa) will be in the Championship (second tier) next year."
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
Ellis returned as Villa chairman for a second spell in 1982, shortly after they won the European Cup.
The club were relegated from the top flight in 1988 but they bounced straight back the following season where they had stayed since, also winning two League Cups in 1994 and 1996.
Ellis, nicknamed 'Deadly Doug' after sacking a number of managers during his reign, sold Villa to Randy Lerner almost 10 years ago and felt that the 54-year-old American should have invested more time with the club.
"I'm rather sad that the man I sold to hasn't given more of his time to come and see Aston Villa play," the 92-year-old said.
"I'd love it (if he came to a game). He is available on the phone but not personally, in the director’s box, which I believe he should."
Villa travel to the Britannia Stadium to take on 10th-placed Stoke City in the league on Saturday.
(Reporting by Shravanth Vijayakumar in Bengaluru. Editing by Patrick Johnston)
