The rising Asian superpower also has an "interests partnership" with Australia, rather than one based on values, Mr Abbott told the Japan Institute of National Affairs in Tokyo.
"We aren't entirely confident that, when China's interests differ from Australia's, there is a shared set of values that will allow a mutually satisfactory outcome," Mr Abbott said in a speech on Friday (AEDT).
"The challenge for all of us is to work to ensure that China better appreciates the rules-based international order that's created the stability that's made China's new prosperity possible."
Mr Abbott said China's creation of artificial islands in the South China Sea came at a massive environmental cost, fortified disputed territory and restricted freedom of navigation.
"(It's) putting at risk the stability and security on which depends the prosperity of our region and the wider world," he said.
The former prime minister said Australia has quietly increased its own air and naval patrols in the South China Sea over the past 18 months.
"We should be prepared to exercise our rights to freedom of navigation wherever international law permits because this is not something that the United States should have to police on its own," Mr Abbott said.
"There should be consequences when countries, even very powerful ones, don't play by the rules; but likewise there should be benefits when they do."
Mr Abbott said Australia and Japan had become close allies in the decades after they waged war against each other in the 1940s.
"Our hope must be that, over time, similar trust can develop between other countries that have a difficult history and that don't currently share similar values," he said.
