Chelsea dominated a scoreless opening period, managing 11 shots on goal to none for the visitors, but Conte was sent to the stands in the 43rd minute by referee Neil Swarbrick after the fourth official called his attention to an angry outburst.
Conte's side finally broke the deadlock in the 55th minute when defender Rudiger reacted quickest to N'Golo Kante's deflected shot by diving to head the ball home.
Chelsea's struggles in front of goal continued for the rest of the game as a combination of solid goalkeeping by Swansea keeper Lukasz Fabianski and wasteful finishing prevented them from winning by a greater margin.
A contrite Conte said it was time-wasting by Swansea, who had only two attempts on goal over the 90 minutes, that led to him being sent off.
"I tried to tell this to the fourth official not only once but a few times," he told the BBC.
"I have apologised because I did the wrong thing. I have been in to see them after the game. It’s right to speak and apologise with them," Conte added.
Third in the table on 29 points and 11 behind leaders Manchester City, Chelsea face a quick turnaround as they play Newcastle United at Stamford Bridge at lunchtime on Saturday, while second-bottom Swansea travel to Stoke City.
(Reporting by Philip O'Connor; Editing by Toby Davis)
