Barty appeared to be careering towards a famous victory before letting slip a one-set and one-break advantage in the loss in Paris on Friday morning (AEST).
Seemingly unnerved by the occasion, Barty broke the 23-times major champion early in the second set to surge further ahead before Williams willed her way back into the enthralling contest on Court Philippe-Chatrier.
Barty's game plan to move Williams from side to side in the hope of exhausting the underdone champion in just her sixth match back since giving birth last September initially paid dividends.
But the grand slam colossus drew on her unrivalled fighting qualities - and energy reserves many doubted she still possessed at almost 37 and eight months into motherhood - to stave off Australia's 17th seed after one hour and 46 minutes.
Barty's exit leaves former finalist Samantha Stosur and 24th seed Daria Gavrilova to fly the Australian flag.
Stosur, runner-up at Roland Garros in 2010 and a semi-finalist in 2009, 2012 and 2016, is through to the third round for the 10th straight year.
