The Dutch National Champion was afforded the perfect lead out by LottoNL-Jumbo teammate Tom Leezer and held off McLay (Great Britain Team) over the final 200 metres on the 218km stage.
Behind Swift's third place on the stage and resulting time bonus helped him move up to third overall, while Julien Vermote (Etixx-QuickStep), the yellow jersey came home fifth to retain his race lead for a third day.
"I won in Yorkshire and I've won here too,” Groenewegen said. "Today was a very hard stage with all the climbs, and it was very long, so I knew there was only a small chance I could fight for it. But the team was very strong and I started to think I could do it. It was a very hard final but the team worked very hard and I had four guys for me, so it was a perfect team performance”
"But the team was very strong and I started to think I could do it. It was a very hard final but the team worked very hard and I had four guys for me, so it was a perfect team performance.”
It had in fact been the other Lotto team in the Tour of Britain, Lotto-Soudal who had worked had to set up the sprint over the final 15km, but they were usurped for pole position on the left hand turn into the finish.
Four riders had spent the stage up the road, including Bardiani CSF's Alessandro Tonelli as the longest surviving and winner of the most aggressive rider bib for Stage 4.
The quartet were eventually swept away when the GC contenders began to battle, with Dan Martin (Etixx-QuickStep), Wout Poels (Sky) and the teams of Movistar and Cannondale Drapac all active.
This flurry of attacks split the field, ditching sprinters like Mark Cavendish (Dimension Data), Caleb Ewan (Orica-Bike Exchange) and Elia Viviani (Sky), and briefly Andre Greipel (Lotto-Soudal) who worked his way back across to the leaders.
No great advantage was ever established with some of the dropped riders able to rejoin as the race hit Rhayader and the main roads leading to Builth Wells the GC battle was over, barring a hotly contested intermediate sprint in Newbridge-on-Wye with less than 20-kilometres remaining.
That sprint saw Tony Gallopin (Lotto-Soudal) taking the points, and more importantly the time bonuses, which elevated him into fourth overall behind Swift, but now ahead of Dan Martin.
Jasper Bovenhuis (An Post-Chain Reaction) retook the sprint jersey while the battle between Xandro Meurisse (Wanty-Groupe Gobert) and Nicolas Roche (Sky) for the mountains classification jersey continued apace.
Meurisse has now eked out a two-point lead over Roche and wears the jersey for a third day on Stage 5.
Vermote now holds both the lead and points jersey, while second overall Steve Cummings (Dimension Data) continues as the leading British rider.
The Tour of Britain remains in Wales for the Start of Stage Five in Aberdare Park in Rhondda Cynon Taf, which then crosses South Wales and the Forest of Dean for the finish on the Royal Avenue in Bath.