Shane Long gave them the lead after 17 minutes, nipping in ahead of defender Shkodran Mustafi.
Arsenal equalised 11 minutes later through Pierre-Emerick Aubameyang, playing a one-two with Welbeck.
Welbeck then cut inside onto Alex Iwobi's pass to give Arsene Wenger's side a barely-deserved halftime lead.
Substitute Charlie Austin equalised a minute after coming on but with nine minutes left Welbeck headed in Iwobi's cross.
In a wild last few minutes Mohamed Elneny and Southampton's Jack Stephens were sent off in separate incidents.
Arsenal remain 13 points behind the top four, their hopes for the remainder of the season resting on the Europa League, in which they take a 4-1 lead to CSKA Moscow for Thursday's quarter-final second leg.
With that game in mind, Wenger made seven changes, giving a league debut to 18 year-old Reiss Nelson. Perhaps unsurprisingly, his team looked disjointed, especially early on.
Irish international Long's goal was only his second in almost 50 games for club and country but even before that the visitors made all the chances.
The best came when Long cut the ball back for James Ward-Prowse, whose shot beat goalkeeper Petr Cech but was blocked by former Saints defender Calum Chambers.
After Aubameyang's equaliser -- his sixth goal in seven Premier League appearances since joining from Borussia Dortmund in January -- Long came close again.
Arsenal, however, were at last moving forward with greater purpose and seven minutes from halftime Welbeck received the benefit of a deflection off Maya Yoshida to beat goalkeeper Alex McCarthy.
Southampton could claim further bad luck, however, when Wesley Hoedt's header from a corner was going in until Elneny nodded it off the line.
It was no more than Southampton deserved when substitute Austin marked his entrance by tapping in Cedric's low cross.
Welbeck missed badly in front of an open goal but redeemed himself with the winner.
There was still time for Stephens and Elneny to receive straight red cards within a few minutes of each other for raising their hands.
"It's a competitive game, Saints are in a difficult position, and at the end it got a bit out of hand," Welbeck said.
(Reporting by Steve Tongue; Editing by Toby Davis and Ian Chadband)
