Currently, anyone visiting, driving through, or working on land owned by indigenous communities has to apply to the local land council for a permit.
The decades-old system is designed to protect the privacy and culture of Aboriginal people.
But Indigenous Affairs Minister Mal Brough has used parliamentary question time to announce that he wants to strip communities of the power to decide who comes on to their land.
Mr Brough said the case of a 12-year-old Maningrida boy, allegedly raped by a group of five adults and five juveniles seven times between April and June this year, had not received enough media attention.
If the permit system was scrapped, journalists and the public would have greater access to communities where abuse and violence were commonplace, Mr Brough said.
"It raised with me the concern I had as to why this hadn't been more widely reported, because I'm sure it would have been in almost any other part of the country," he told parliament.
"One of the conclusions I've come to is that, unfortunately, (in) Maningrida, like so many other remote indigenous communities, people still require permits to go there.”
The time had come to remove the permit system, he said.
Labor said Mr Brough's exploitation of the alleged rape would hurt the Maningrida community.
"The use of this case to validate the removal of permits is just playing dog-whistle politics at its worst," Northern Territory Labor MP Warren Snowdon said.
The Northern Land Council (NLC) said Country Liberal senator Nigel Scullion supported the permit system and journalists were granted ample access to communities.
"The NLC has advised both the commonwealth and NT governments that it will ensure reasonable access to Aboriginal land, including that the media will continue to have access to court proceedings,"
NLC chief executive Norman Fry said in a statement.
The Northern Territory government said it also supported permits.