Five-times world champion Hamilton set a best time of one minute 04.838 seconds at the sunny Spielberg circuit, 0.144 quicker than Vettel.
Hamilton's team mate Valtteri Bottas ended up third fastest, 0.017 slower than Vettel, with Mercedes carrying out a hurried engine change after discovering an oil leak about an hour before the start of the session.
"It was a bit messy, overall," Mercedes team boss Toto Wolff said of a session that thrust the track's unforgiving kerbs into the debate after several teams suffered costly front wing damage.
The session ended three minutes early with red flags after Renault's Nico Hulkenberg ran over the kerb at the penultimate corner and ripped off half of his car's front wing, scattering debris across the track.
"I think the most spectacular failure was the Renault, where the front wing is completely gone, but we have broken two flaps," said Wolff.
"I believe Racing Point has broken a front wing so I believe we need to do something around the sausage kerbs."
Red Bull boss Christian Horner said the kerbs were too inviting to be a deterrent to drivers and needed a change of angle.
"There's been a lot of discussion about these kerbs here over the last couple of years. I'm sure there's going to be a lot more," he added.
Ferrari used the slower medium tyres, however, and were also running through a list of test items after a disappointing race in France last weekend while the Mercedes were on softs.
Hamilton has won the last four races, and six out of eight this season, and has a healthy 36 point lead over Bottas in the championship.
Bottas has been on pole in Austria, one of the shortest tracks on the calendar, for the past two years and needs another to give himself the best chance of reining in Hamilton.
Mercedes are favourites to take an 11th victory in a row this weekend, equalling the longest winning streak of the modern era that was set by McLaren in 1988, despite suffering a double retirement in Austria last year.
Ferrari's Charles Leclerc was fourth fastest with last year's winner Max Verstappen fifth for Red Bull at their home circuit.
The session followed a vote by teams on whether to return to last year's thicker treaded tyres for the rest of the season, a move seen by some as an attempt to reduce Mercedes' advantage. There was no majority so nothing changed.
"Relying on the teams to agree, that was never going to happen," said Horner.
(Reporting by Alan Baldwin in London, editing by Christian Radnedge)
