Identity of ‘third party’ that brought down Optus network revealed

The unnamed “international peering network” that Optus said had contributed to its 16-hour-long network meltdown last week is run by its Singaporean parent company Singtel, it can be revealed.

The embarrassing revelation comes as it can also be revealed a key executive, who left Canadian telco Rogers Communications less than two weeks after it last year suffered an almost identical outage to Optus, is now working for Singtel.

Customers were left in the dark after Optus experienced an outage across the entire country last week.Credit: Louie Douvis

Jorge Fernandes left Rogers in July 2022 and joined Singtel in June this year as its chief technology officer – a role he held at Rogers for more than four years. Fernandes has been contacted for comment and this masthead does not suggest he was responsible for last Wednesday’s outage.

The Rogers outage affected 12 million users, with around 25 per cent of Canada losing internet connectivity for about 15 hours.

Singtel chief technology officer Jorge Fernandes, who formerly held the same role at Canadian telco Rogers, which suffered a similar network meltdown to Optus in July last year.

Both Optus and Singtel were approached for comment. Optus declined to respond on Tuesday evening but on Wednesday morning a spokesperson confirmed that the Singtel Internet Exchange, known as STiX, was the international peering network “which sent changes to routing information” that crippled its network.

The Singtel Internet Exchange is one of the many internet peering exchanges that help move vast amounts of data across the world. Peering allows two networks to connect and exchange traffic directly, sometimes without having to pay a third party to carry data across the internet. Not only does it lower the cost of moving traffic around, but it also improves performance and redundancy.

An Optus engineer, who did not wish to be named due to the sensitive nature of last Wednesday’s outage, told this masthead on Tuesday that at least part of the hesitancy to name Singtel as the “third party” probably had to do with the fact the parent company’s board was visiting Sydney last week to meet Optus executives.

Both revelations come ahead of a Senate inquiry into the outage that Optus chief executive officer Kelly Bayer Rosmarin will front on Friday, where senators including the Greens’ Sarah-Hanson Young are expected to grill the telco boss.

Before Optus confirmed this masthead’s reporting on the identity of the international peering network, a separate source close to Optus, who was not authorised to speak publicly, told this masthead on Tuesday that it was the Singtel Internet Exchange. It was this company, the source said, that Optus was alluding to in a statement on Monday afternoon as the one sending routing information to the telco’s routers on the day of the outage that put them into meltdown.

Optus said in its Monday statement that those updates were sent “following a routine software upgrade”. The Optus source added that the software update occurred at the Singtel Internet Exchange, not on the Optus network.

Analysis from Kentik, a company that monitors the internet, shows the length of the outage in Co-ordinated Universal Time (UTC).Credit: Kentik’s Doug Madory

Once received, the routing information changes then propagated through multiple layers in the Optus network and exceeded “preset safety levels” on key routers that could not handle them, resulting in the routers disconnecting from Optus’ core network “to protect themselves”.

The changes meant technicians had to be deployed to key locations across the country to access routers that had gone offline so that they could restore services, which delayed restoration.

Poor filters

Despite the Singtel Internet Exchange being one of the triggers for the outage, an Optus source said the Australian telco was primarily responsible for the outage.

“It’s still Optus’ fault when it comes to its own network (unless Singtel required Optus to accept routes blindly),” an Optus engineer, who was unclear what commercial agreements may have been in place, said.

The source, as well as other external network engineers this masthead spoke to, said Optus should have had filtering rules in place that dismissed or ignored the update from the Singtel exchange.

On the day of the outage, a senior Optus executive phoned a counterpart at content distribution network Akamai, another of the telco’s peers, believing it had triggered the outage.

By Friday, Optus engineers were still adamant that Akamai was partly to blame. However, Akamai definitively ruled this out in a statement on Monday night. “Akamai did not trigger the outage,” a spokesperson told this masthead.

Optus customers seek an explanation for the outage on Wednesday.Credit: Chris Hopkins

As more details of the cause of the Optus outage emerge, Telstra chief executive Vicki Brady said the nation’s biggest telco had won new customers as a result of Optus’ network meltdown. But she pushed back on calls for extra regulation, arguing that would force it to share its infrastructure in the event of another mass disruption.

Last week’s outage affected about 10 million Optus customers, leading some to desert the company barely a year after it suffered the worst cyber breach in Australian history.

The exact number of customers ditching Optus is yet to be determined, but speaking at an annual Telstra investor day in Sydney on Tuesday, Brady said Telstra had seen an uptick in new customers.

“We take zero pleasure in any competitor or operator experiencing major outages,” the Telstra boss said.

While it was early days since the incident and Telstra was continuing to see elevated levels of new customers signing up, Brady said she “wouldn’t anticipate major shifts in [market] share”.

Telstra’s executives were quick to spruik the resiliency of its network in front of investors, highlighting increased investment and reliability.

Network sharing in the picture

Channa Seneviratne, Telstra’s executive in charge of technology development and innovation, said the telco’s mobile network was superior to competitors due to its “best in class resilience and security”.

Seneviratne said Telstra was in favour of network sharing in the event of a natural disaster like a bushfire, which would enable Optus or Vodafone customers to send texts or make and receive calls.

Communications Minister Michelle Rowland and Emergency Management Minister Murray Watt and their departments have begun developing such emergency roaming capabilities – work that was already underway before last week’s outage.

Minister for Communications Michelle Rowland.Credit: Alex Ellinghausen

But that was a very different proposition to network-sharing in the event of an outage, said Seneviratne.

“Last week we lost one of the national networks, roughly one-third of our carriers. We don’t design our networks to have overhead to manage another carrier’s whole customer base, it just doesn’t work that way.

“I’m sure it’s going to be the topic of conversations with some of the parliamentary inquiries, and we’ll be happy to participate and co-operate with that,” Seneviratne said.

“It’s a very different proposition, emergency roaming in a selected area, versus opening up the possibility of having 11 million Optus customers suddenly come into our network. You probably end up having two national outages.”

Telstra’s executives carry SIM cards of other carriers with them in the event of a Telstra outage, he added. Optus chief Kelly Bayer Rosmarin last week raised eyebrows when she was forced to dial into ABC radio via WhatsApp, given Optus’ networks were down.

The Business Briefing newsletter delivers major stories, exclusive coverage and expert opinion. Sign up to get it every weekday morning.