In brief
- Telstra officials said a Network Time Protocol server that required updates was behind a recent national outage.
- Telstra admitted it had been warned multiple times to update the server before the issue reared its head.
Telstra did not act on repeated warnings to fix an impending issue with a piece of equipment that ultimately triggered a devastating network outage, a parliamentary inquiry has heard.
United States-based supplier Microchip Technology warned the telco multiple times that a Network Time Protocol server required a software update to prevent a "rollover problem" that would take down the system, Telstra officials told a hearing in Canberra.
The snap inquiry was set up in response to the ensuing outage, which left train networks in disarray and customers unable to dial Triple Zero.
The telco has been in damage control since the outage occurred on 8 July.
In a submission to the inquiry, the telco said a GPS card within the server did not operate as expected once power was restored to the device following scheduled maintenance.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
The change had not been properly documented, meaning the maintenance team was not aware the issue would occur when the device was restarted.
A software update, which could have averted the outage, had also not been applied to the device.
An EFTPOS payment system used by about 80,000 businesses confirmed it had been impacted, with a number of customers taken offline and unable to process payments.
Train services had to be cancelled after severe network disruptions in both Victoria and NSW, triggering strong criticism from the federal government.
What is a Network Time Protocol?
A Network Time Protocol (NTP) is a set of rules that govern timekeeping on a computer network. NTP is one of the oldest and most basic computer network protocols still in use today.
It was invented by legendary US computer engineer David Mills in 1985 to ensure that all computers on a given network were synchronised to Coordinated Universal Time (or UTC) within a margin of a few milliseconds. It has been heavily modified and updated over the years to encompass and incorporate new technologies and network infrastructures.
Mills' invention of the NTP was so groundbreaking that it earned him the nickname 'Father Time' among his peers.
This level of synchronisation is important, particularly when working with systems that use timekeeping as a central component — like train networks.
For instance, NTP allows computers on a network to coordinate their logging so that administrators and IT professionals can track error reports or security concerns in the precise chronological order they occurred, no matter where they happened in the network or the world.
It also ensures greater internet and network security and authentication. Many encryption methods depend on very strict time synchronisation to confirm real vs malicious traffic.
NTP is also used in banking, where rapid trading and processing funds transfers rely on pinpoint, millisecond network accuracy.
Telstra received multiple reminders
Telstra chief executive Vicki Brady said she was "deeply sorry" for the failure.
"Telstra let Australians down. We let our customers down, we let the community down, and we fell short of what people rightly expect from us," she told the hearing on Friday.
"Triple Zero sits at the heart of public trust in Australia's communication system. Telstra has a critical role in the system, and we take that responsibility extremely seriously.
"No one should be left wondering whether a call for emergency help will get through."
Gerard Tracey, the executive responsible for end-to-end service performance, said the supplier first warned about the impending issue in the SSU 2000 server, a 15-year-old piece of tech that costs about $30,000, in November 2020.
The issue resulted in the timing server being reset to 2006, causing invalid authentication certificates to spread across the network and preventing customers from making calls and accessing data.
Inquiry chair Sarah Hanson-Young questioned why the repeated warnings hadn't been acted on.
"This doesn't seem like a hardware matter. This seems like incompetence," the Greens senator said.
Telstra received another reminder in January.
The reason why the warnings went unheeded would be probed in an external investigation Telstra has commissioned into the outage, Brady said.
"This is not an excuse, but putting in context, there are many software updates and changes getting made across the network," she said.
"Any software update creates a risk when we put it into testing.
"In this case, they thought, given the design that was in place, it was not relevant because we were not using this equipment in a way that would trigger this risk."
Telstra is accepting compensation claims from affected customers and small businesses who can provide evidence to support their case.
Telstra conducted more than 630 welfare checks on customers whose Triple Zero calls could not be connected due to the outage.
Representatives from the Australian Communications and Media Authority and the communications department will also give evidence at the inquiry.
For the latest from SBS News, download our app and subscribe to our newsletter.
