Harper, at 46 one of Canada's youngest prime ministers, arrived for the ceremony in a Dodge Caravan instead of the usual limousine.
He takes over from outgoing Prime Minister Paul Martin, whose beleaguered Liberal Party lost a January 23 election to the Conservatives after only 18 months in office.
Mr Harper has promised better childcare and healthcare, and to lower consumption taxes.
And in a pledge likely to generate significant controversy he has promised to revisit Canada's same-sex marriage law, passed by Martin's minority government last year.
The unveiling of his new 26-member cabinet followed, with a few surprises.
It is much smaller than the 39 positions in Martin's government, purposely
"designed for work, not for show," Mr Harper said in a statement.
"The 26 ministers are equal members of the team, ensuring equal voices from all regions of Canada," he said.
"The structure is designed to promote accountable, efficient and effective government… more focus and purpose; less process and cost," he said.
The surprise defection of former Liberal Industry Minister, David Emerson, boosted the Conservatives' minority government to 125 seats out of 308 in the House of Commons.
As a result of the defection the Liberals, now in opposition, fall to 102 seats. The separatist Bloc Quebecois and socialist New Democratic Party have 51 and 29 seats, respectively, and there is one independent MP from Quebec.
Mr Emerson was named international trade minister, and will also be responsible for the 2010 Olympic Winter Games in Vancouver, his hometown on Canada's Pacific coast, according to documents obtained by AFP.
Meanwhile, Peter MacKay, former deputy Conservative leader, who once fought Harper for the leadership of the party and helped him unite Canada's once fractured political right, was named foreign affairs minister.
The post of deputy prime minister was dropped.