In a statement issued on Tuesday afternoon, Mr Quaedvlieg said Mr Dutton’s comments were "disgusting and offensive".
He defended his decision to submit a statement to the parliamentary inquiry into Mr Dutton's actions as a minister.
"It is extraordinary behaviour from a cabinet minister to pre-emptively impugn the character and reputation of a witness," he said.
"I call on him to formally withdraw that comment."
Labor used question time to launch a sustained assault on Mr Dutton regarding the claims.
The Minister responded by bringing thick, conspicuous binders with the names of Labor MPs Chris Bowen and Tony Burke prominently displayed.
Mr Bowen alone had written to him on visa matters 192 times, Mr Dutton told parliament.
“You have been out there with your wet lettuce talking to journalists. You've been issuing angry press releases. And, after all of that time, that is the best you could come up with,” Mr Dutton said.
"They can carry on all they like. I will tell you this much, all I will do is double down.”
Shortly after Question Time, Roman Quaedvlieg tweeted incredulously about the minister’s “rambling” comments.
Earlier, the Senate committee investigating Peter Dutton’s use of ministerial powers to spare au pairs from deportation asked for more time to release its final report.
SBS News understands the Labor chair, Louise Pratt, will seek the Senate’s approval to extend the inquiry by another week to September 19.
The Upper House will need to approve the extension but Labor is confident it will pass with the support of the Greens and the crossbench.
Senators on the inquiry have already received email leaks from within the Home Affairs department.
The inquiry was due to report on Tuesday, but recent revelations from the former Border Force chief Roman Quaedvlieg have raised new questions and sparked a bitter public feud with Mr Dutton.
Mr Quaedvlieg wrote to the inquiry alleging he was asked by the minister’s chief of staff to help a “mate” of Mr Dutton by preventing the deportation of a young European au pair, who had been detained by Border Force on suspicion of a visa breach.
Mr Dutton responded by accusing his former top border officer of working with the Labor party on a smear campaign and pointing out that Mr Quaedvlieg’s dates did not match with the dates he worked at the department.
In turn, Mr Quaedvlieg wrote a second letter explaining the date mixup by suggesting he may have been referring to a third au pair intervention.
Labor senator Murray Watt said there was a “growing scandal around Peter Dutton”.
On Monday, Mr Dutton distanced himself from the Queensland policeman at the centre of the scandal over one of the au pairs.
The dispute boils down to whether Mr Dutton did a favour for a friend, or whether he was simply judging the case on its merits after he was contacted by a man he had not spoken within 20 years.
“He does not have my personal phone number or my personal email address.”
The minister has also released the email he received from the contact, who began with the words: "Peter. Long-time between calls.”
Labor seized on the email as proof of some familiarity between the two men.