Telstra staff unaware of mass outage risk as critical software failure ‘rippled slowly across the network’ | Telstra


Telstra has blamed the lack of a software update for a key time-keeping system for outage that caused nationwide chaos last week, with its maintenance teams also unaware of a design change that affected how it would reset.

Ahead of an appearance by its chief executive, Vicki Brady, at a Senate inquiry into the mobile outage on Friday, Telstra provided a written submission that revealed the cause. In it, the company stated it did not lack redundancy in its network – but that such redundancy did not prevent the outage.

The submission confirmed reporting that one of Telstra’s network time protocol (NTP) servers, designed to ensure the systems had the correct time, had reset to be “2006”.

Telstra said it has three NTP servers, in Melbourne, Sydney and Perth.

During maintenance, the company shut down and restarted the server in Melbourne. Due to an “underlying software configuration” within the server, it restarted with the wrong 2006 date.

“Over the next few hours, the incorrect date rippled slowly across the network, causing authentication certificates in other servers to become invalid,” Telstra said.

“Customers were intermittently unable to authenticate onto the network (‘no service’), which affected their ability to place voice calls and use data across Telstra’s mobile network.”

Sign up for the Breaking News Australia email

Telstra said it had made an intentional design change to the equipment to fix an earlier fault – but that this had not been properly documented. This meant maintenance workers arriving at the Melbourne site early last Wednesday were not aware of how the device would be reset.

A software update had also not been applied to the device, and if that had been done, the outage may not have occurred, Telstra said.

The submission also said when the Melbourne NTP server disconnected for maintenance, the other two worked as redundancy and backup as expected.

“The failure mode here was not inherently related to hardware, levels of redundancy, or the architecture of our network,” Telstra said.

But when the Melbourne server supplied an incorrect date once switched back on, “downstream systems used that date in security, authentication, session and policy-control processes.”

“The issue was therefore not simply the loss of one NTP server or redundancy in the design of the configuration of the three NTP servers, but the propagation and acceptance of erroneous date information by interconnected systems that rely on timing as a trust and ordering reference.”

Telstra also said it was taking full accountability for the outage.

“That is clearly unacceptable. If maintenance work can trigger this kind of outage, it suggests our controls were not good enough,” the company said.

Telstra CEO Vicki Brady faces questions on nationwide outage – video

“We are accountable for that, and our investigation will address why that design change was not documented, why the software update was not completed, and what needs to change in our controls so known risks are captured, prioritised and closed before they can affect customers.”

In the course of the outage, Telstra said 58,835 calls to triple zero successfully connected, while 604 experienced an error.

Telstra also operates the triple-zero platform for all telcos, and the company said the platform does not use the NTP servers for synchronisation, so it was unaffected by the incident. Fixed-line callers on the NBN were also not affected.

On Friday, Brady and a number of other executives are due to appear before a Senate inquiry – established into last year’s Optus outage – to explain the outage that wrecked havoc for mobile services, transport systems, retailers and electric-vehicle charging.

The chair of the committee, the Greens senator Sarah Hanson-Young, said the inquiry will seek to get to the truth of the outage and making “sure that we don’t keep accepting a system that leaves Australians vulnerable”.



Source link

  • Related Posts

    How to protect yourself from wildfire smoke

    Around 100 million people could experience unhealthy air as wildfire smoke blows into the Midwest, Northeast and Mid-Atlantic this week. Wildfire smoke contains toxic gases and particulates that can affect…

    Australia to Put Environmental Brakes on A.I. Data Centers

    Australia will require large data centers powering artificial intelligence to generate as much power as they consume, and ensure that creative professionals retain control over work that may be used…

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    You Missed

    Manitoba’s Tristan Peters flourishing in the Majors after off-season trade – Winnipeg

    Manitoba’s Tristan Peters flourishing in the Majors after off-season trade – Winnipeg

    Iceberg lettuce eyed in parasite outbreak

    Iceberg lettuce eyed in parasite outbreak

    Do Face Masks Help With Wildfire Smoke? Yes, But More Is Needed

    Do Face Masks Help With Wildfire Smoke? Yes, But More Is Needed

    How to protect yourself from wildfire smoke

    How to protect yourself from wildfire smoke

    All the Canadian Politics!

    Outrage after ICE agents tackle man at Las Vegas airport

    Outrage after ICE agents tackle man at Las Vegas airport