Mr Martin, who also said he would step down as leader of the Liberal Party, said he telephoned Mr Harper to congratulate him.
"We differ on many things, but we all share the belief of the potential and the promise of Canada and the desire of our country to succeed," he told supporters in his Montreal electoral district.
Mr Harper, 46, and his right-wing Conservatives will now form a minority government after ending a 12-year reign by the Liberals.
He will almost certainly fall short of an outright majority and will have to seek alliances to allow his party to rule the 208-seat parliament.
At around 5pm AEDT, the Conservatives were listed as having won or leading in 125 seats, the Liberals 102, the Bloc Quebecois 51 and 19 for the New Democratic Party.
The Conservatives had 37 percent of the vote, seven percent ahead of the Liberals.
"Canadians were looking for change," Conservative Deputy Leader Peter MacKay told CTV television.
"Minority means we have to be constructive, and we have to be working together and finding common ground," he said.
Canadians apparently punished the Liberals for a string of ethical lapses, with the election forced by a public inquiry that found Liberal politicians in Quebec had taken kickbacks in exchange for government contracts.
Mr Harper has vowed to cut taxes and tackle violent crime and corruption, and has also signalled a willingness to review the Canadian decision not to join the US anti-missile shield.
Liberals had however claimed he would renege on Canada's commitments under the Kyoto climate accord, and would lean closer to the US Republican Party than most Canadians would like, charging a Harper victory would put a "smile" on the face of US President George W Bush.
In the last parliament, formed after elections in June 2004, the Liberals had held 133 seats, the Conservatives 98, the Bloc 53 and the New Democrats 18.