Chasing 268 runs for victory, Australia eventually folded for 161 in Steve Smith's first loss as their test captain.
Australia set a new record as the last 154 deliveries of their innings did not produce a single run. O'Keefe and Nevill also featured in the slowest partnership in test cricket among those that lasted at least 100 balls.
It was a remarkable win for Sri Lanka, who were shot out for 117 in the first innings and were reeling at six for two in the second before Kusal Mendis made an epic 176, which earned him the man-of-the-match award.
"Credit to Kusal, the way he played and scored 176 which certainly turned this game. It was tough to come back from there," Smith said at the presentation ceremony.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
"We were sure 268 on this wicket on day four and five was going to be difficult and it turned out to be that way."
Herath, who claimed 4-49 in the first innings, took 5-54 in the second and fittingly sealed the win by bowling O'Keefe, who is unlikely take part in the remainder of the series, towards the end of the afternoon session.
"I don't have words to explain after a very hard few months. We regrouped, trained really well and prepared to the best," Sri Lanka captain Angelo Mathews said.
He praised Mendis as a "special talent" while also praising Herath and left-arm Chinaman bowler Lakshan Sandakan, who claimed seven wickets on a memorable test debut.
"He is a brilliant find for us," Mathews said. "We looked at the wicket and knew it was going to turn. So we thought playing another spinner would be ideal and he responded brilliantly."
The weather-hit match got off to another delayed start following steady morning drizzle and Sri Lanka attacked with spin from both ends to unsettle the visitors who resumed on 83 for three.
Herath ripped the heart out of Australia's middle order with three telling strikes in the morning session.
The left-arm spinner sent back Adam Voges and Mitchell Marsh before dealing the biggest blow when he trapped Smith lbw to end the Australian captain's spirited resistance.
Smith hit just one boundary in his gritty 55 which was only the second 50-plus score in the bowler-dominated match.
O'Keefe and Nevill, aided by luck and Mathews' struggle with reviews, blocked everything, frustrating the hosts in a partnership that lasted 29.4 overs and yielded only four runs.
Nevill faced 115 balls for his nine runs while O'Keefe scored four off 98 balls.
The second test starts in Galle on Thursday.
(Reporting by Amlan Chakraborty in New Delhi; Editing by John O'Brien and Ed Osmond)
