Mr Kibaki had strongly backed in a referendum but the campaign caused a deep split the government.
In a step aimed at cushioning the blow from Monday's landmark plebiscite to his political fortunes ahead of elections in 2007, Mr Kibaki sacked all ministers and their deputies and said he would name a new government within two weeks.
"Following the results of the referendum, it has become necessary for me, as the president of the republic, to reorganise my government to make it more cohesive and better able to serve the people of Kenya," he said.
"In accordance with the powers conferred upon me under the constitution of Kenya, I have directed that the offices of all ministers and all assistant ministers become vacant," Mr Kibaki said in a televised address to the nation.
Charter rejected
The announcement of the move, which takes immediate effect, came just 24 hours after the president conceded that his camp had lost its bid to have voters adopt the first major changes to Kenya's 42-year-old independence constitution.
Nearly 60 percent of voters rejected the new charter, which was vehemently opposed by Kibaki's political foes and some members of his cabinet as it retained near absolute powers for the office of the president.
Critics, including several ministers in his ruling coalition, said the draft defied popular demands for significant power to be devolved to a prime minister, which Mr Kibaki promised when he was elected three years ago on a reform platform.
Analysts warned that the vote was a sign of growing disenchantment with the president, who has been accused of failing to follow through on campaign pledges to crack down on corruption and revamp Kenya's economy.
Drastic measures
Prior to Wednesday's announcement, many analysts said Mr Kibaki, 74, would have to take drastic measures, including a major cabinet reshuffle, if he wanted to retain credibility, salvage his legacy or stand for re-election in two years.
The 2004 Nobel Peace Prize laureate Wangari Maathai said the president should take steps to boost confidence in his presidency.
"It is very important that the president has a cabinet that has the confidence and the respect of the people," said Ms Maathai, who is the former deputy environment minister.
"What we have lost in this country most of all is trust. We must trust each other," added Ms Maathai, who earlier this month dismissed the referendum as "a farce."