The New York Times reports the three men and a woman were arrested in a search of three Manhattan apartments on Tuesday night.
Investigators recovered the bags of heroin inside the flats, the paper says, quoting a law enforcement official.
The Times says information stemming from the investigation into Hoffman's death led them to the building.
Hoffman died on Sunday in an apparent heroin overdose, with a needle found stuck in his arm.
Also on Tuesday, police said heroin found in Hoffman's apartment did not contain fentanyl, a powerful additive that has been tied to 22 recent fatal overdoses in Pennsylvania.
The city medical examiner has not yet reached a definitive cause of death for the award winning actor, who was 46 and had three children.
He was hailed by many as the finest character actor of his generation.
Police said he was found on the bathroom floor, wearing shorts and a T-shirt, in the apartment he rented at 53 Bethune Street in the West Village.
According to US media, at least 50 envelopes of heroin littered the $US10,000 ($A11,220.20) a month rental.
