Berisha, who was allowed to switch to Kosovo by FIFA despite having played competitive matches for Norway, put away a penalty on the hour after a foul by Thomas Lam on Bernard Berisha to spark delirious celebrations among his team mates and officials.
Finland, who have never qualified for the World Cup, had taken an 18th minute lead in the Group I clash when Kosovo failed to clear a corner, allowing defender Paulus Arajuuri to lash the ball into the roof of the net.
Kosovo grew in confidence after their goal but could not find a winner.
Kosovo goalkeeper Samir Ujkani had also only been authorised to play on the afternoon of the match, having previously played competitively for Albania, as had midfielder Alban Meha.
Ujkani played from the start and Meha came on as a late substitute.
FIFA does not usually allow players to change national teams once they have played a competitive international but has made an exception for Kosovo, who were only accepted as members of world soccer's governing body in May in the face of fierce opposition from Serbia.
Kosovo declared independence from Serbia in 2008 and had been campaigning for several years to join FIFA.
(Reporting by Fatos Bytyci in Pristina and Philip O'Connor in Stockholm: Editing by Brian Homewood)
