“A lot of people have raised this with me. I think they are quite angry that Ms France is using her disability as an excuse for not moving into our electorate," Mr Dutton told The Australian.
Ms France, 45, had her leg amputated after she was hit by a car at a shopping centre in 2011.
"Having a disability is not an excuse, it is a reality," the former journalist and palliative care worker told reporters in Brisbane on Friday.
"It is a reality that many hundreds of thousands of Australians have to deal with every single day."
She was joined by Deputy Labor leader Tanya Plibersek who unleashed on the minister.
"Well I hear from some people that Peter Dutton is a horrible human being - put that on the record," Ms Plibersek said.
"What's Peter Dutton's excuse for saying to people with disability right across Australia that there's somehow something wrong with them?
"What's Peter Dutton's excuse and frankly what's Scott Morrison going to do to pull him into line?"
Prime Minister Scott Morrison dismissed questions from reporters about Mr Dutton's comments, saying only that he had been taken out of context.
Earlier, Labor Leader Bill Shorten said the prime minister should order Mr Dutton to apologise.
Mr Shorten reminded voters of the prime minister's statement when he launched the disability royal commission a week ago about building a culture of respect.
"That didn't last very long, did it? What we have here is Mr Dutton has insulted every Australian with a disability, and Mr Morrison has said, "that's OK."
Other senior Labor figures have also leapt to Ms France's defence against what they labelled a "disgraceful attack".
"Peter Dutton is launching dirty attacks to distract from the LNP’s cuts and chaos," Labor Party president Wayne Swan said.
"Unlike Mr Dutton, who is spending all his time making clear how desperate and out of touch he is, Ms France has been working hard and talking to people in the community about Labor’s plan to fix schools and hospitals."
Asked if he was comfortable with Mr Dutton's comments, Prime Minister Scott Morrison told reporters that Mr Dutton's comments had been taken out of context.
Greens Senator Jordon Steele-John, who also uses a wheelchair, was outraged by Peter Dutton's "ablest comments" and called on the prime minister to take a stand.
"Why does he and his cabinet keep acting in a way which is so ablest and discriminatory towards disabled people," Senator Steele-John told Sky News.