The Rally Cycling rider was followed by team-mate Rob Britton, Lennard Hofstede (Sunweb), Mathias Le Turnier (Cofidis) and Gavin Mannion of the UnitedHealthcare team.
The five-rider break spent almost the entire 159km stage from Santa Barbara to Santa Clarita in the race lead with the time gap cut from a high point of nine minutes to just enough seconds to beat the bunch.
"I guess it just shows that anything can happen in bike racing," Huffman said. "Sometimes you do everything right and you lose and sometimes you get a little bit lucky and today we flipped the script.
"It was a sprint stage on paper but we just forced two of our strongest guys in the break and just rode hard all day and they weren't able to bring us back."
Peter Sagan (BORA-hansgrohe) led the chase across the line while team-mate Rafal Majka retained his overall lead ahead of George Bennett (LottNL-Jumbo) and Ian Boswell (Sky).
The general classification is likely to be decided on Stage 5 as the peloton tackles the 125km from Ontario to the top of Mount Baldy.
"I had a great team supporting me today and I hope to have a great day tomorrow, and hope to keep the jersey," Majka said. "I think it’s going to be a really hard climb, so at the end, we’ll see how my legs respond after five days of racing.
"It will be the first time I do that climb and it all depends on my legs. If I have good legs, for sure I’ll try to attack, otherwise, I’ll try to follow. We’ll see how my condition is after these four days."
