The Sunweb rider and UCI time trial world champion claimed a narrow two-second victory ahead of Australian Rohan Dennis (BMC) and Victor Campenaerts (Lotto Soudal).
Thousands of spectators lined Jerusalem's streets to watch the first time a Grand Tour cycling race has ever been held outside Europe.
The riders passed Israel's parliament, Supreme Court, the Israel Museum and the Hebrew University on their route through Jerusalem.
Dumoulin was the last of the 175 riders to start and finished the 9.7km time trial in 12min 2sec.
"It's been quite a cool few days in Israel and today tops it off with the win," Dumoulin said. "I knew that I felt good but you have to see how things go during the race.
"We've worked really hard over the last few months, so I'm really happy with the win because it confirms that all of the work has paid off.
"The course was really good for me; it was really technical and I’m good on the corners. I could go full gas, then recover, then go full gas and recover again.
"It’s nice to be in the Maglia Rosa but it will be hard to keep it for three weeks. We aren’t planning on defending it every day so we will see what we do with that in the coming days."
Dennis was the 42nd rider to roll down the start ramp to begin the undulating course in Jerusalem and recorded the fastest provisional time of 6min 21sec at the 5.1km checkpoint, before laying everything on the line in the second part of the time trial to cross the line in 12min 4sec.
He then had a long wait in the hot seat until Dumoulin stopped the clock two seconds faster."It was a technical course. It was hard to get a rhythm and it was challenging," Dennis said. I would have liked the roads to be a bit smoother but you can't have everything.
"I did everything I could and the only way I can judge how I rode is my position at the end of the day. I was pretty relaxed until the last half an hour of the race.
"I knew Tom was going to be the big rival today and he proved that by taking the win. There were a few other riders that I was surprised about. There was nothing I could have done better today. I was just beaten by the better guy."
Four-time Tour de France champion Chris Froome's Giro got off to an inauspicious start after he finished in 21st place at 37 seconds, after he crashed during a recon of the technical course, drawing blood around his right knee and ripping his shorts and jersey.
"I just lost the front wheel in a corner. I was testing a corner at race speed, at around 20 or 30kph," Froome said. "A crash is always going to hurt but I’m just glad it wasn’t more serious.
Miguel Angel Lopez (Astana), another of the pre-race favourites, also came off but is not seriously hurt, while Kanstantsin Siutsou's (Bahrain-Merida) race was over before it began when he was taken to hospital with a reported broken collarbone.
"Obviously, I saw what happened to Kosta (Siutsou) who is in hospital, so I’m just glad I’m not more injured after today. That’s bike racing though and all part of the sport," Froome said.
The second stage of the race stretches 167km down the Mediterranean coast from Haifa to Tel Aviv.
Stage three will follow a 229km route from Beersheba in the Negev desert down to Israel's southern tip of Eilat along the Red Sea.
The race will then transfer to Italy, and the island of Sicily, for an early rest day on Monday. The Giro will end in Rome on 27 May.