A spokesman for Mr Ruddock's office confirmed the move to SBS, giving a short statement.
"The position is at the gift of the leader," he said.
"Mr Ruddock will leave him to explain."
The announcement of Mr Ruddock's departure from the position comes less than a week after he presiding over a challenge to Prime Minister Tony Abbott's leadership.
The spill motion brought to the Liberal party room was voted down 61-39.
Recommended reading
Comment: Sacking of Chief Government Whip spells more trouble for Abbott
Mr Abbott confirmed that he had "made some changes to the Whip arrangements", stating that Queensland MP Scott Buchholtz would replace Mr Ruddock as Chief Government Whip.
He thanked Mr Ruddock for his "extraordinary contribution" to the government and the party.
"As Father of the House, Philip has given over four decades of service to the Australian Parliament and the Australian people," he said.
"He was a senior minister throughout the life of the Howard government and was shadow cabinet secretary in opposition. Later this year, he will become the second longest serving parliamentarian in our history."
Andrew Nikolic, who was a vocal supporter of the Prime Minister in the lead up to the failed spill motion, was also appointed a Government Whip.
Mr Abbott described the Tasmanian MP as a politician who has "made an outstanding start as a member of parliament".
He added that Nola Marino would continue in her role as Government Whip.
These changes are due to be in place for the next parliamentary sitting period.
