United led 2-0 and then 3-2 at St James' Park but Paul Dummett equalised for the hosts in the 90th minute to leave Van Gaal fuming.
"Of course it feels like a defeat," the Dutchman told Sky Sports after United slipped to sixth place, overtaken by West Ham United.
"We have given it away -- I have told that to my players. When the referee gives a penalty for nothing -- it is a duel I think and you cannot decide who is worse -- but we gave it away."
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
United fans have been highly critical of their team's lack of flair at times this season and even Van Gaal, surrounded by speculation that his time in charge at Old Trafford was coming to an end, made the surprise admission this week that he had found his side "boring" to watch at times.
But United were anything but dull on Tuesday, counter-attacking at pace only to twice throw away a lead.
"We could have scored six goals easily here and we did not," bemoaned Van Gaal.
"If we had, it would have been three successive wins at the start of the year going into Liverpool (on Sunday).
"It is not so interesting to create chances. In every press conference I say you have to score one more goal than the opponent."
Wayne Rooney scored twice, including the 79th minute goal that put United 3-2 up, but the captain agreed with his manager in casting a downbeat tone.
"It does feel like a defeat. After scoring three good goals, it was maybe a lack of concentration -- not staying with runners for the third -- and that is disappointing," he said.
"If you take a two-goal lead, we should be taking all three points and killing the game off. We didn’t take all our chances and we have been punished for it.
"We always want to try and play entertaining attacking football that is what we work on and try and do, and hopefully we can come up with more solutions."
(Reporting by Justin Palmer; Editing by Andrew Both)
