Network Events

Future / Scheduled events

Co-Location Gateway Migration

Date: 20 Oct 17 06:00 - 20 Oct 17 12:00
Affected services: Hosted

On Friday the 20th at 6:00 AM we will be migrating our Co-Location customer gateways to new hardware. The process should be seamless but some customer firewalls might experience an outage if their ARP tables do not update efficiently.
If you have any connectivity issues after 6 AM please do not hesitate to call us on 04 474 0770 or follow your standard support process.

Currently affecting the system

There are no events currently affecting the system.

Historic events

Chorus network issues

Date: 03 Aug 17 14:15 - 03 Aug 17 16:50
Affected services: DSL

At 14:15 we were alerted to multiple customers on ADSL, VDSL and HSNS in Masterton, Taupo, Taumaranui, Whakatane and Auckland who have lost connectivity.
We have logged a fault with Chorus but as yet have received no update.

15:24 Chorus have confirmed they have a "Nationwide event" and they currently do not have an ETA for resolution.

16:50:28 Chorus have advised this is now resolved: "Unplanned outage occurred due to planned activity under CRQ 161590" was their final update.

CityLink Planned Outage Notification

Date: 19 Mar 17 18:00 - 20 Mar 17 09:00
Affected services: CityLink

CityLink have advised the following:
CityLink will be undertaking Wellington network upgrades during our regularly scheduled maintenance window (1st and 3rd Sunday of each month) as per our standard Terms and Conditions. This window will be between the hours of the Sunday 18:00 to the Monday 06:00.

This planned outage is in relation to the core switch being upgraded, while network configuration changes are carried out to optimise network performance. Note that this work concerns Wellington CityLink Ethernet services only - FibreLINK/dark fibre services are not affected.

Please do not hesitate to contact us on 0800 469 783 on Monday morning if you arrive at work and have no internet connectivity.

Wellington UFB Upgrade

Date: 22 Apr 16 05:30 - 26 Apr 16 12:00
Affected services:

Update: We were unable to stand the connection up on the new router. We have rolled back to the old router and will investigate and reschedule for 5:30 AM on Tuesday the 26th.

On Friday morning we will be migrating our Wellington based UFB connections handover link from our existing Cisco edge router to our new Juniper edge router.
You will experience an outage of up to 5 minutes while the handover link is re-patched to the new Juniper edge router and the new routing changes take effect within our core network.

Urgent Spark Wholesale Outage

Date: 23 Oct 15 01:00 - 23 Oct 15 02:00
Affected services: Cable, CityLink, DSL, E-Mail, Hosted, Wireless

Spark Wholesale have advised us of an urgent outage on our upstream national/international circuits, beginning at 1 AM this Friday the 23rd, for up to 1 hour. This is to replace a failing wavelength router card and clean the fibre terminations.
Xtreme connections may notice a 2-3 minute loss of service as traffic going over our Spark Wholesale connections fails over to our Vodafone Wholesale connections.
No other impact should be noticed unless data is sent to another ISP that depends solely on Spark Wholesale traffic through the failing card for their upstream connectivity, as they will be unavailable for the entire duration of the outage.

Vodafone Packet Loss

Date: 24 Sep 15 10:00 - 24 Sep 15 12:00
Affected services: Cable, CityLink, DSL, Hosted, Wireless

At approximately 10 AM we started experiencing around 40% packet loss on our international Vodafone Wholesale upstream circuit.
To minimise impact we have shut this connection down so all our international traffic is now going through our Spark Wholesale/GGIS circuit.
When Vodafone resolve this we will bring the connection back up.

Update 12:00 Vodafone have advised they had power related issues at Northcote and these have now been resolved.

Core switching modification

Date: 29 Jul 15 22:00 - 30 Jul 15 00:00
Affected services: Cable, CityLink, DSL, E-Mail, Hosted, Wireless

We have recently had two incidences where an outer edge switch has flapped, forcing updates through our core switches, thus resulting in outages for some customers of up to a minute, depending on where traffic is flowing at the time.
On Wednesday the 29th of July, beginning at 10:00 PM we will be modifying the way our core and edge switches communicate with each other.
The process will result in further outages of up to one minute while we apply the modification to each switch, once the process is complete there won't be any further issues.

We would like to apologise to everyone who has been impacted by these outages.

Kind regards,
The team at Xtreme.

Major email systems fault

Date: 17 Jan 14 06:36 - 20 Jan 14 11:30
Affected services: E-Mail

11:30 AM 20/01/14 Update: After further work over the weekend and this morning we are now confident the issues experienced on Friday have been resolved, we apologise for Friday morning's frustrating incident.

9:45 AM Update: Full mail functionality was restored at approximately 9:30. We will continue investigating the root cause and are planning at this stage to bring the full system back online early (~3:30) Monday morning.

6:40 AM: After running Microsoft's Windows updates early this morning our mail systems stopped accepting inbound emails after a short period of time. We have not yet been able to resolve this on the live/backup systems so we are currently bringing a temporary new server online.

We estimate this will take 1-2 hours (as at 6:40), during which no email services will be available unfortunately.

Please be patient as we work to resolve this for you.

CityLink Issues

Date: 10 Dec 12 08:35 - 10 Dec 12 09:07
Affected services: CityLink

CityLink had issues in the Featherston/Johnston Street area due to a faulty UPS. This was resolved around 9:00 AM

Spam filtering issues

Date: 12 Oct 12 05:00 - 12 Oct 12 12:15
Affected services: E-Mail

At 5:00 this morning a spam filter update introduced a new set of checks which caused a lot of false positives (rejects of legitimate email). In all cases the sender would have received a notification that the message was not delivered.

Our attention was first drawn to this at 10:00 and we disabled the primary spam filter system at 10:30 as the full extent of the issue became apparent. This resolved the incorrect rejects, but unfortunately allowed more spam through.

We isolated the problem and restored functional spam filtering at 12:15 PM.

We sincerely apologise for any trouble this caused and are investigation options to ensure this does not happen again.

Firewall migration

Date: 10 Aug 12 03:30 - 10 Aug 12 05:30
Affected services:

Early tomorrow morning (between 3:30 and 5:30) we will be upgrading the firewalls that our protect our web, mail and DNS servers.
There will be a brief loss of connectivity to the servers when the new firewalls are patched in, but the process should be largely unnoticeable from the outside world.

Major CityLink Outage

Date: 02 May 12 07:03 - 02 May 12 16:10
Affected services: CityLink

CityLink have advised that they have a fibre down at the intersection of Wakefield Streetand Cambridge Terrace. The fibre appears to have been brought down by a truck.
Engineering staff have been dispatched to site to investigate.
CityLink are unable to provide an ETA for full resolution at this time, but customers and buildings will regain connectivity as individual fibres are repaired.

7:47 Update:
The damaged fibre is being removed and a replacement section installed. Once the replacement section is ready it will be spliced onto the undamaged sections.

CityLink anticipate having the replacement section ready for splicing by 10am. They will then be prioritising the splicing to get the in use fibres repaired first. All services should be restored by 2pm.

CityLink connection migration

Date: 08 Mar 12 03:30 - 08 Mar 12 05:30
Affected services: CityLink

On Thursday morning between 3:30 and 5:30 we will be migrating our main CityLink customer connection to new hardware as part of our IPv6 migration.

We know from past experience that some firewalls (ZyXEL ZyWALL’s in particular) will need to be power cycled in order to pick up our new hardware.

If you or your customers have no connectivity on Thursday morning please try power cycling the firewall in the first instance and if a connection still cannot be made don’t hesitate to call us on either 04 474 0770 or 0800 469 873.

CityLink Issues

Date: 24 Sep 10 09:02 - 24 Sep 10 12:10
Affected services: CityLink

CityLink is currently experiencing some network issues in the Featherston Street and Customhouse Quay areas of Wellington.

12:10 A dead power phase to a key building has been repaired and no further issues should arise.

Cable modem issues

Date: 27 Aug 10 09:14 - 23 Sep 10 16:00
Affected services: Cable

Cable modem connections (primarily in, but not isolated to) the Kilbernie area are experiencing loss of connectivity that may come and go. The cable modem network provider has advised that an update on a single piece of equipment that should not have caused any issues was performed early yesterday morning, no further information has been provided as yet.
Connecting a laptop directly to the cable modem with the connections IP information makes the connection work again, but may not be a permanent fix. Please contact us if you or a customer is having any issues.

23rd September: We believe the issues have been resolved by rolling back an upgrade, but have not received any official report from the cable modem network provider.

CityLink Switch Software Upgrades

Date: 07 Sep 09 00:00 - 07 Sep 09 06:00
Affected services: CityLink

CityLink are upgrading the software in their switches and have advised:
"This work will necessitate multiple 5 minute outages on the CityLink network and it is expected that services will be restored within 5 minutes."

If you find you have no connectivity on Monday the 7th please do not hesitate to contact us on either 04 474 0770 or 0800 469 873.

Core Firewall Upgrade

Date: 24 Jun 09 16:00 - 26 Jun 09 06:00
Affected services: Cable, CityLink, DSL, E-Mail, Hosted, Wireless

On Friday morning between 4:30 and 5:30 we will replacing our core firewalls, this will result in our web, mail and DNS servers being unavilable for a short period of time.
If everything goes smoothly these services should be unavailable for no longer than one minute.
If you have any concerns on Friday morning please don't hesitate to call us on 04 474 0770 or 0800 469 873.

ADSL Outage

Date: 31 Mar 09 12:05 - 31 Mar 09 14:00
Affected services: DSL

Telecom have advised of issues resulting in loss of connectivity for some Wellington and South Island connections. They currently have no ETR.

Widespread Power Outage

Date: 21 Jan 09 08:48 - 21 Jan 09 12:00
Affected services: CityLink, Wireless

There is currently a widespread power outage in the Wellington CBD, surrounding suburbs and parts of the Hutt Valley, this is affecting connectivity across the board.

Araneo Upgrade

Date: 06 Oct 08 15:28 - 06 Oct 08 20:10
Affected services: Wireless

Araneo are expecting a brief outage on their Wellington Wide Area Network this evening, 6th October. Between 8PM and 8:05PM there could be a period of approximately 30 seconds of packet loss as a planned topology change propagates through the network

Araneo Disruption

Date: 04 Aug 08 06:46 - 05 Aug 08 00:00
Affected services: Wireless

Tonight (4th of August) at 8:00 PM CityLink will be replacing a switch that will cause a break of up to 15 minutes for Araneo Wireless connections fed from Mt Climie and Colonial Knob.

Update on Monday's Power Outage

Date: 28 May 08 08:11 - 28 May 08 08:11
Affected services:

At 11:58 Monday morning the top end of Wellington lost power. Within 30 seconds our generator started and Xtreme itself experienced no downtime. We ran on generator until 12:35 and then cut back to normal mains power. Unfortunately, CityLink’s equipment located at AT&T House did not last through the power cut.
CityLink worked franticly to bring the network up piece by piece and the CityLink network was fully restored at 4PM.

We are meeting with CityLink next week to discuss what they will be doing and what else we can do to ensure future power cuts have less impact on the CityLink network and our customers.

Thank you all for your patience and understanding on Monday.

UPS Upgrade

Date: 13 May 08 15:31 - 19 May 08 06:00
Affected services: CityLink, Wireless

Between the hours of 03:00 and 06:00 on the 19th of May we will be upgrading a UPS in one of our cabinets which feeds our CityLink and Araneo connections. This will result in loss of connectivity for less than 5 minutes while the equipment reboots.

Smartlinx3 in the Hutt Valley

Date: 05 Feb 08 20:20 - 05 Feb 08 22:00
Affected services: Wireless

Smartlinx3 are currently experincing issues in the Hutt Valley resulting in loss of connectivity, they hope to have this resolved by 10:00 PM

Hutt Valley Smartlinx3

Date: 31 Jan 08 07:00 - 31 Jan 08 10:19
Affected services: Wireless

Smartlinx3 are currently resolving an issue that is causing loss of connection for some sites in the Hutt Valley area

Router module upgrade

Date: 27 Aug 07 19:39 - 28 Aug 07 05:45
Affected services: Cable, CityLink, DSL, E-Mail, Hosted, Wireless

At 5:30 am Tuesday the 28th we will be replacing two older modules in our border router with current ones. This will result in an outage of approximately 5 minutes.

Loss of connectivity

Date: 20 May 07 08:41 - 21 May 07 00:00
Affected services: Cable, CityLink, DSL, E-Mail, Hosted, Wireless

A repeat of the issues experienced on the 27th of April have led to the diagnosis of a failing superviser module in one of our Cisco core switches. We replaced it prior to Sunday midnight with a spare off the shelf.