Mr Shorten pointed to high Indigenous incarceration rates and worse health outcomes as more of a priority for governments.
"They are the real challenges. We all know Mr Turnbull is looking for a summer-time diversion."
He also shut down suggestions by his Labor colleague Linda Burney that there should be a separate public holiday celebrating Australia's First Nations peoples.
"I think we have enough public holidays in Australia, and in terms of going forward, my priority is to make sure we listen to Indigenous Australians," he said.
Mr Shorten's comments come as a new poll shows Australians aren't worried about changing the national day of commemoration from January 26.
There's been a renewed push to change the date, which marks European arrival in Australia, because it is painful for many indigenous people.
The Australia Institute conducted a poll of 1417 people about their attitude to Australia Day.
"This polling shows that while Australia Day is important to most Australians, most people are laid back about the date we celebrate on," Ebony Bennett, Deputy Director of the institute said on Thursday.
"When asked to choose which date Australia Day should be celebrated on, less than a quarter chose the current date from a range of options."
Half agreed that Australia Day should not be on a day that is offensive to indigenous Australians, while 36 per cent disagreed.
However, only 37 per cent agreed that the current date of Australia Day was offensive to indigenous Australians, while 46 per cent disagreed.