Adrian Ramos, American teenager Christian Pulisic, Sebastian Rode and Emre Mor also scored for Dortmund who are fourth in the table on six points.
Darmstadt went down to 10 men in the 57th minute after Peter Niemeyer was sent off for a second caution.
Bayern were far less convincing than in their 5-0 Champions League victory over Russian team Rostov on Tuesday and were caught napping after eight minutes when Paraguayan Dario Lezcano slipped the ball through the legs of keeper Manuel Neuer.
It took four minutes for Bayern to level as Poland striker Robert Lewandowski chipped the ball beautifully over the Ingolstadt keeper for his fifth league goal of the campaign.
Xabi Alonso's low drive from 20 metres put Bayern ahead but they were lucky not to concede an equaliser before Rafinha drilled in their third goal from the edge of the box in the 84th minute.
"It was a hard game for us," Bayern coach Carlo Ancelotti told reporters. "Ingolstadt played really well, they pressed high up.
"Sometimes it is not possible to play well, then we have to show other qualities and that is what the team did today."
Bayern lead the way with nine points, two ahead of RB Leipzig after they crushed hosts Hamburg SV 4-0 thanks to four goals in the last 24 minutes. Cologne are also on seven points in third place.
Borussia Moenchengladbach bounced back from their 4-0 defeat at Manchester City on Wednesday, firing four goals past Werder Bremen in the first half to win 4-1 and move on to six points.
Thorgan Hazard and Raffael, one a penalty, grabbed two goals apiece.
Bottom club Werder had American Aron Johannsson sent off for dissent late on and a third successive league defeat piled the pressure on coach Viktor Skripnik.
Bayer Leverkusen, who squandered a two-goal lead to draw 2-2 against CSKA Moscow in the Champions League in midweek, lost 2-1 at Eintracht Frankfurt to slip to 10th.
(Reporting by Karolos Grohmann, editing by Alan Baldwin)
