"There is a group of people that don't want me here," the Northern Irishman told reporters after the Villa win, which sent his side up to seventh in the Premier League table and five points behind leaders Manchester United.
"I am talking about people outside of here. I think it is pretty clear. Sometimes we haven't lost games and the hysteria around it is pretty clear. There is maybe something else going on from behind."
Milner, who scored Liverpool's first goal and pulled the strings in midfield alongside Lucas during a much-improved performance in which Daniel Sturridge scored for the first time since March, also lashed out at the recent criticism of his team's form and backed Rogers to turn things around.
"A lot of things that have been said are disappointing and frustrating really," the England international said, quoted by the Liverpool Echo.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
"I've been really impressed with the manager. Preparation here is second to none.
"There's people giving criticism from outside the club but I think Manchester City have lost more games than us this season.
"You have to stick together," the 29-year-old added.
Milner's midfield partner Lucas dedicated the win to his manager.
"We wanted to win for the manager," the Brazilian said. "We know what has been said about him and we think we showed we are a group of players behind him and trying to make everything right.
"We know the pressure is on him but we wanted to play the way he wants us to play, and I think we did that."
(Reporting by Simon Jennings in Bengaluru; editing by Sudipto Ganguly)
