The 30-year-old surprised a group of stage favourites with a late attack on a steep climb to the finish, with Rui Costa (UAE Abu Dhabi) second and Jakob Fuglsang (Astana) third.
Hermans was down to help his BMC team leader Greg van Avermaet but roles were switched, with the worker set up for the win by his team leader.
"Greg was supposed to contest the sprint finish whereas me, I had a free role, in particular marking Romain Bardet,” Hermans said. “I tried my luck at the best moment 150 metres from the line.”
The 145.5km stage featured four climbs and was the first battle of the General Classification contenders with a short but steep finish on the Al Jissah climb.
It was a battle for a breakaway to go clear in the first hour with the peloton flying at 50km/hr. Two riders, Mark Christian (Aqua Blue Sport) and Preben Van Hecke (Sport Vlaanderen-Baloise), eventually broke away and established an advantage of more than seven minutes.
The peloton began the chase with 60km to go and the catch was made 20km before the finish line, which allowed further attacks to play out.
Only half of the peloton was left as the group approached the finish, and it was Hermans and van Avermaet who were in the selection on the final climb.
Van Avermaet attacked with 1.5km to go, was then overtaken by Merhawi Kudus (Dimension-Data). He attacked again, and then made way for Hermans to make a push for the line and win convincingly ahead of Costa and Fuglsang.
Hermans has a four-second general classification lead over Costa and is six seconds ahead of Fuglsang.
"We did as the team asked us. We had a plan to attack with Greg Van Avermaet and he saw his moment when we tried to just ride at tempo in the middle of the climb," Hermans said.
"Then he countered Kudus but Kudus was really explosive and fast, so when I saw Greg came back I tried. It was still 300 metres to go but I knew I would only have one chance if I made it a very long sprint and not too explosive, and it worked out well."
The 162km third stage is a hilly course from Sultan Qaboos University to Quriyat.