The referendum vote had asked for a simple "yes" or "no" response to the question of whether Colombians supported the accord signed by the government and FARC rebels.
The Revolutionary Armed Forces of Colombia, or FARC, had agreed to turn in their weapons and fight for power at the ballot box instead.
Under the deal negotiated in Cuba over four years, the FARC would have been able to take part in the 2018 presidential and legislative elections and have 10 unelected congressional seats guaranteed through to 2026.
Opinion polls had predicted the "yes" vote would win by an almost two-to-one margin.
But with almost all ballots counted, the victory went to the 50.2 per cent of Colombians who opposed the deal, over the 49.8 per cent in favour of it.
That was a difference of about 60,000 votes out of 13 million ballots.
President Juan Manuel Santos has acknowledged the result, saying he is still committed to a peaceful resolution of the conflict.
"Right now we're going to decide amongst all of us what is the path forward that we should take so that peace we all want can be possible and be even more strengthened in the end. I will not give in, I will continue seeking peace until the very last minute of my term because this is the path to be able to lead a better country for our children."
Voter turnout was lower than expected due to heavy rain caused by Hurricane Matthew in the Caribbean.
But Bogota-based political analyst Angelika Rettberg has told Al Jazeera, there are multiple factors that could have influenced the referendum outcome.
And she's questioned whether they may have a role to play in the event that amendments are made to the peace agreement.
"It has to do with a profound dislike regarding President Santos on the one hand. On the other hand, we cannot underestimate the impact that so many years of human rights violations have had on public opinion and also I would add to this the fact that increasing gains and security over the past 15 years really made it hard for public opinion to accept many of the concessions that these agreements implied. So it's much harder to sell peace to a country that is no longer perceived as one in crisis and where really the brunt of conflict is carried by very few of the rural populations. The agreements still have a lot of room open in order for the process of implementation to perhaps accommodate some of the criticisms that the 'no' side has exposed so maybe the adaptation or the adjustment will come in a way of implementing the agreement that has already been signed."
It's estimated that over a quarter of a million people have been killed in the 52-year conflict, with millions of others displaced.
Opponents of the accord, led by former President Alvaro Uribe, said if the agreement failed to pass the government and the FARC should return to the negotiating table.
They specifically want provisions that would spare jail time for rebel leaders who confess war crimes renegotiated.
President Santos has ruled out re-opening talks, but he might be forced to reconsider.
Before the referendum, the 53-year-old said he had no 'plan b' and Colombia would be returned to war if the "no" vote won.
The FARC says it is still open to peace.
Juan David Gutiérrez is a former adviser to the Ministry of Justice in Colombia.
Mr Gutiérrez says the result of the referendum isn't a foregone conclusion for the people of Colombia.
"They are worried about justice. About the FARC leaders not paying their dues. They ar every worried about FARC's participation in politics and where will illegal money from the organisation end up. It's a very emotional vote. It's a vote against a guerrilla organisation that has caused havoc and a lot of fear in the last 50 years."
