Coulthirst, 24. kept his cool to convert from the spot in the 21st minute at Bramall Lane and Barnet, managed by Darren Currie, nephew of former Sheffield United great and England international Tony, held on for a classic Cup upset.
Barnet are 84 places below Sheffield United in the English football pyramid but were deserved winners against a side harbouring genuine hopes of promotion to the Premier League.
They are the only team from outside the first four tiers to make the fourth-round draw on Monday.
"You couldn't tell the difference between the teams, especially in the first half," Coulthirst, 24, told the BBC.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
"We knew if we stayed in the game our confidence would grow, and we got the early goal, which made it easier.
"I'm delighted for all the lads and the manager. Who knows how far we can get? I want Arsenal away -- I used to play for Spurs so it would be nice to beat them there."
It is the first time Barnet have reached the fourth round since 2008 while Sheffield United had not lost to a minor-league team for more than 100 years.
"I'm proud of them," Currie, who began his playing career at West Ham United and played more than 100 times for Barnet, said.
"But I'm a little bit wet too because the boys doused me in water when we got back to the dressing room.
"(Sheffield manager) Chris Wilder said we deserved it after too. I can't put it into too many words -- it is such a wonderful occasion for everyone."
Wilder was scathing of his team as they became the fall guys of the round.
"They were better in every single part of the game and we were lucky it wasn't two or three more," he said.
"What we served up there is not acceptable for either the club or for me. That doesn't represent, in any way shape or form, what we are about."
(Reporting by Martyn Herman, editing by Ed Osmond)
