The author was a 10/1 outsider with some bookmakers but picked up the award and STG50,000 ($A80,912) prize money at a ceremony at Guildhall, central London.
The acclaimed Australian author Peter Carey was shortlisted for a fourth time for his latest novel Parrot and Olivier in America.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
Carey is one of only two authors to have won the prize twice - in 1988 for Oscar and Lucinda, and 2001 for True History of the Kelly Gang.
Englishman Tom McCarthy had been the hottest of favourites to snap up this year's award for his book C.
Chairman of the judges and former Poet Laureate Andrew Motion said Jacobsen's tale of two old school friends and their teacher was funny but "so nearly adjacent to tragedy".
The 58-year-old was born in Manchester and educated at Cambridge University before spells teaching in Wolverhampton and Sydney, Australia.Intense meetings
The five judges made their decision during a one-hour-long meeting which Motion described as "intense" and decided to award Jacobson the prize by a verdict of three to two.
Motion said: "The Finkler Question is a marvellous book: very funny, of course, but also very clever, very sad and very subtle. It is all that it seems to be and much more than it seems to be. A completely worthy winner of this great prize.
"You expect a book by Howard Jacobson to be very clever and very cunning and it is both of those things. It's highly articulate, everything works in it very well.
"That is what you expect from him but it's also in an interesting and complicated way a very sad book, a very melancholy book I think, which means that from where I sit it's absolutely a book for grown ups."
Jacobson has been long listed twice before for the prize, in 2006 for Kalooki Nights and in 2002 for Who's Sorry Now, but had never before been shortlisted.
What it means to be Jewish
The Finkler Question deals with subjects including love, loss, male friendship and what it means to be Jewish.
Motion said: "The kind of comedy it is is so nearly adjacent to sadness that it would be a mistake simply to describe this as a comic novel.
"It doesn't really matter which way round you put it but it is either a very funny book with very sad bits in it or it's a very sad book with very funny bits in it."
Jacobson said the book was about "sorrow".He said: "I think I'm talking about loss.
"I wanted to make the reader laugh and weep at the same moment."
He can expect a huge increase in sales and recognition around the world after being crowned winner.
Each of the six shortlisted authors receives STG2,500 ($A4,045) and a designer-bound edition of their book.
The decision will be welcomed by bookmakers who feared a six-figure payout if the favourite, Tom McCarthy's C, had won.
They slashed the odds on it after a series of big bets made it the heaviest-backed Booker contender ever.
