"We don't need to put a handbrake on population growth, we need to manage our population growth sensibly in a country which quite frankly can take a lot more than 25 million people.
"I don't think population policy or immigration intake should be decided by polls, they should be decided on principle."
The latest Fairfax-Ipsos poll also found 45 per cent of voters believe the number of immigrants coming to Australia should be reduced, with 23 per cent arguing for a rise and 29 per cent happy with the status quo.
Australia doesn't have an immigration target, but rather a ceiling, Mr Pyne added.
Mr Pyne confirmed the government is considering ways to settle new arrivals into regional areas rather than congested major cities.
Prime Minister Scott Morrison has recently flagged migrants could be asked to spend five years in a regional area if they want permanent residency.
Mr Morrison wants states and territories to decide where migrants go as they plan roads, hospitals and schools.
Labor leader Bill Shorten says immigration policy should be bipartisan.
Congestion in cities should be eased through better public transport, roads and internet services so new arrivals don't have to travel hours to get to work, he said.
"We've always taken some immigrants from other countries," he told reporters in Melbourne on Monday.
"But we have to make sure that we're not relying on an army of temporary labour where people have temporary visas to work in this country, and they get exploited, and Australians are missing out on jobs."