The 19-year-old left-hander, bidding to become the first teenager to lift the Suzanne Lenglen cup since Iva Majoli in 1997, saved three set points on her way to winning the opener.
Then she let slip a 5-2 lead in the second set, double-faulting on a match point at 5-3 as nerves kicked in.
She steadied herself though and broke Croatian Martic’s serve to set up a semi-final against Britain’s Johanna Konta.
“I mean, it’s not happening every day, right? I’m just really happy with my game. And it was a very tough match for me. I beat her for my first time in my life, so it’s amazing,” Vondrousova, leading a new Czech brigade, told reporters.
Martic had beaten top Czech Karolina Pliskova in the third round and had beaten Vondrousova in their four previous clashes, but this time found her a much trickier proposition.
Boasting a game full of variety, Vondrousova, who has yet to drop a set in the tournament, broke for 3-2 with a forehand winner down the line that wrong-footed her opponent.
The 31st-seeded Martic battled back and seemed poised to take the opener when Vondrousova fell 0-40 down when serving at 5-6, only to reel off four clean backhand winners and an ace.
She took the momentum into the tiebreak which she won with ease and then opened a 5-2 lead in the second set.
Martic saved a first match point on her serve before Vondrousova served a woeful double fault on the second in the ninth game, which the Croatian won to put the set back on serve.
But Martic eventually cracked on her own serve, fluffing an easy smash and sending a lob long on the fourth match point.
“In the semi-finals, I think anything can happen. I think it’s going to be a very tough and great match, and I just can’t wait to play,” Vondrousova said.
American Amanda Anisimova will look to become the second teenager into the semi-finals on Wednesday when she faces defending champion Simona Halep.
