Network Events
Future / Scheduled events
There are no future events.
Currently affecting the system
There are no events currently affecting the system.
Historic events
Urgent outage
Date: 24 Jun 10 17:35 - 25 Jun 10 12:00
Affected services: Cable, CityLink, Wireless, DSL, Hosted, E-Mail
We need to do some urgent troubleshooting for a peculiar issue that has arisen since the installation of our first new Cisco core switch.
Once every minute for a second or two, latency through our core router is jumping. We suspect the core router is receiving something from the new switch that is upsetting it and we need to ascertain what is causing this.
At approximately 5:00 tomorrow morning we will be dropping connections from our new switch to our core router for up to 15 minutes in order to try and pinpoint the cause/source of this regular occurrence.
Core switch upgrades
Date: 09 Jun 10 15:58 - 11 Jun 10 12:00
Affected services: Cable, CityLink, Wireless, DSL, Hosted, E-Mail
This Friday morning between 4:30 and 5:30 we will be patching various border and edge router connections into our first new core switch. This will involve 2 outages of up to 5 minutes per outage. As these breaks are occuring well within our network, customer connections should not need any attention after repatching is completed.
Edge router relocation
Date: 02 Jun 10 04:15 - 04 Jun 10 12:00
Affected services: CityLink, DSL
This Friday morning at approximately 4:30 we will be physically relocating one of our edge routers to make room for new equipment for improved core network redundancy and growth.
This will result in loss of connectivity for approximately 10 minutes on all new EUBA and HSNS connections and CityLink connections with a stub IP address in the 203.171.50.0/24 subnet.
As this is simply a physical relocation all connections should restore automatically, should you experience any connection issues please call us immediately on 0800 469 873 or 04 474 0770 and select option 2.
TelstraClear DoS Attacks
Date: 26 May 10 16:13 - 27 May 10 00:00
Affected services: Cable, CityLink, Wireless, DSL, Hosted, E-Mail
Over the last week TelstraClear has been the target of DoS attacks coming out of the Asia region which in turn, affects all TelstraClear international traffic.
When we become aware that an attack has commenced we are dropping our TelstraClear international BGP advertisements which forces all international traffic (after BGP advertisement propagation) through our Telecom international circuit.
TelstraClear are keeping us up to date regarding mitigating the effects.
Vodafone/ihug issues
Date: 12 May 10 09:33 - 12 May 10 11:40
Affected services: Cable, CityLink, Wireless, DSL, Hosted, E-Mail
It appears that Vodafone/ihug have made some routing changes overnight and are currently advertising their routes over WIX but then losing them in their network, we are trying to get an update.
Vodafone/ihug have confirmed that they have issues but currently have no ETA.
Vodafone/ihug ceased advertising their routes on WIX at approximately 11:40 and traffic is now successfully routing over their national circuits.
Power cut and subsequent systems outage.
Date: 16 Sep 09 16:51 - 16 Sep 09 17:40
Affected services: Cable, CityLink, Wireless, DSL, Hosted, E-Mail
At approximately 16:04 a portion of the Wellington CBD experienced a 3/4 hour power outage.
During this time our UPS's and generator maintained a constant power supply to our core network infrastructure and customer co-located equipment.
Mains power was restored to our premesis at 16:51.
When the mains power was restored our UPS's dropped all power to all equipment briefly as they changed back to mains power supply.
A quick check of our core network infrastructure revealed a core Cisco switch that sits between our core routers (ironically, to provide redundancy to the core routers and border router) was dead and even a complete electrical isolation before repowering was not bringing it up.
At this point we made the decision to reconfigure the primary core router and border router in order to bypass the failed switch and connect the two directly in order to restore connectivity to the outside world. We completed this at 17:40. This restored all services other than unsecure web browsing. We then determined that our port 80 web caching system had not recovered, so we bypassed this system as well. This was completed just prior to 18:00, thus restoring all services.
We also became aware during the network reconfiguration, that our 0800 number and DDI's were not functioning correctly as our VoIP system had not come back up correctly, this system was restored at 17:51.
We are in discussion with our UPS providers regarding why the change back to main supply caused an outage.
We are investigating options to provide complete redundancy from the core to the border, we will post a separate NOC event when we are ready to implement this as there is a good chance that it will require a brief outage.
We need to impliment systems to provice some form of communication with customers should we have another major outage.
We would like to reassure our customers that we strive to cover all bases in our facility. We have two of everything wherever possible. Yesterday morning (as in the morning of the outage) we actually took delivery of a second generater to cope with future growth for power and cooling.
We are pleased that we stayed up for the duration of the power cut, but are bitterly dissappointed that the return to mains power had such disasterous consequences.
Border Router Reboot
Date: 07 Sep 09 16:30 - 07 Sep 09 16:45
Affected services: Cable, CityLink, Wireless, DSL, Hosted, E-Mail
At 16:30 our border router performed 4 unexpected reboots in a row. We have sent the crashinfo files to our Cisco support vendor for analysis and are awaiting their response.
CityLink Switch Software Upgrades
Date: 07 Sep 09 00:00 - 07 Sep 09 06:00
Affected services: CityLink
CityLink are upgrading the software in their switches and have advised:
"This work will necessitate multiple 5 minute outages on the CityLink network and it is expected that services will be restored within 5 minutes."
If you find you have no connectivity on Monday the 7th please do not hesitate to contact us on either 04 474 0770 or 0800 469 873.
Smartlinx3 Hutt Valley Link
Date: 12 Aug 09 00:00 - 19 Aug 09 12:00
Affected services: Wireless
Smartlinx3 have advised for the 19th of August from midnight through to 5:00 AM: "Our backbone fibre provider FX Networks, is working with Ontrack as part of the Wellington Region Rail Program (WRRP) and is required to complete maintenance work along segments of the FX Networks fibre cable between Wellington and Upper Hutt. They regret that this work does mean that some services to you will be impacted, however please be assured that they are working closely with their contractors to keep any downtime to a minimum.
The following services to you will be affected during the above outage windows:
Outage One
• Internet and tunnel services between Wellington and Lower Hutt
• Internet and tunnel services between Wellington and Upper Hutt
Outage Two
• tunnel services between Lower Hutt and Upper Hutt
• Internet and tunnel services between Wellington and Upper Hutt"
Core Firewall Upgrade
Date: 24 Jun 09 16:00 - 26 Jun 09 06:00
Affected services: Cable, CityLink, Wireless, DSL, Hosted, E-Mail
On Friday morning between 4:30 and 5:30 we will replacing our core firewalls, this will result in our web, mail and DNS servers being unavilable for a short period of time.
If everything goes smoothly these services should be unavailable for no longer than one minute.
If you have any concerns on Friday morning please don't hesitate to call us on 04 474 0770 or 0800 469 873.
ADSL Outage
Date: 31 Mar 09 12:05 - 31 Mar 09 14:00
Affected services: DSL
Telecom have advised of issues resulting in loss of connectivity for some Wellington and South Island connections. They currently have no ETR.
Power Change Over
Date: 24 Jan 09 13:00 - 25 Jan 09 00:00
Affected services: Cable, CityLink, Wireless, DSL, Hosted, E-Mail
As a result of a power change over to new UPS's we will have, between 11 PM and midnight, a 20 second outage as our primary core router fails over to our secondary core router, and a 2-3 minute outage as our customer edge routers are restarted.
Widespread Power Outage
Date: 21 Jan 09 08:48 - 21 Jan 09 12:00
Affected services: CityLink, Wireless
There is currently a widespread power outage in the Wellington CBD, surrounding suburbs and parts of the Hutt Valley, this is affecting connectivity across the board.
Araneo Upgrade
Date: 06 Oct 08 15:28 - 06 Oct 08 20:10
Affected services: Wireless
Araneo are expecting a brief outage on their Wellington Wide Area Network this evening, 6th October. Between 8PM and 8:05PM there could be a period of approximately 30 seconds of packet loss as a planned topology change propagates through the network
CityLink cable damage
Date: 14 Aug 08 16:14 - 15 Aug 08 00:00
Affected services: CityLink
Two Citylink overhead fibre-optic cables, one 96-fibre cable and one 24-fibre cable, have been ripped down by a passing bus at the corner of Victoria and Vivian Streets. Citylink engineers are currently on-site assessing the scene and working towards replacement of the damaged sections of cable and restoration of all circuits effected.
Araneo Disruption
Date: 04 Aug 08 06:46 - 05 Aug 08 00:00
Affected services: Wireless
Tonight (4th of August) at 8:00 PM CityLink will be replacing a switch that will cause a break of up to 15 minutes for Araneo Wireless connections fed from Mt Climie and Colonial Knob.
Beijing Olympics Upgrades
Date: 08 Jul 08 17:18 - 11 Jul 08 17:18
Affected services: CityLink
Router upgrades for the Beijing Olympics.
The Beijing Olympics are starting on the 8th and we will all be routing for the Kiwi's.
TVNZ will be streaming this event over the Wellington Internet Exchange (WIX) and to be honest we are not really sure of the increased bandwidth requirements this event will bring.
As both TVNZ and Xtreme Networks peer at the WIX this traffic will not be seen as international.
Each will be Windows Media streams at ~90Kb/s, ~384Kb/s, or 1.5-2Mb/s, and may be some experimental HD Freeview coverage stream at 5-6Mb/s.
Therefore we have decided to bring forward our planned upgrades for our WIX and CityLink Customer Edge connections to 1Gb (1000Mb) to handle the extra traffic we may see.
The WIX upgrade was completed seamlessly today.
Our new CityLink Customer Edge Router is in place and we will be cutting the bulk of our customers over early this Friday the 11th.
The edge router replacement will result in a 5 - 10 second outage for CityLink customers.
We will also be upgrading our main and backup core routers to make sure we have plenty of backbone capacity.
The new core routers are in transit and will be here in the next week or two. We will post another NOC event when we intend to cut these over.
Our core router upgrades will result in 2 x 20 - 30 second outages as the system automatically fails over to the backup router then fails back to the main router.
Our SmartLinx3, Araneo, and Cable Links have plenty of capacity for the number of connections so we will not be upgrading any of these routers or links this month.
CityLink Customer Connection upgrade
Date: 08 Jul 08 17:39 - 11 Jul 08 08:00
Affected services: CityLink
On Thursday the 10th at approximately 5:30 AM we will be transferring customers with IP address starting 203.167.235 and 218.101.3.
On Friday the 11th at approximately 5:30 AM we will be transferring customers with IP addresses starting 203.171.40 and 203.171.41.
This should only result in a 5 - 10 second outage but some firewalls may retain "ARP entries" for a long period of time. If you have no connectivity or your connection appears slow or internittent on Thursday or Friday morning please try turning your firewall off for 5 - 10 seconds and see if this resolves your issues. If you still have concerns please do not hesitate to call us on 04 474 0770 or 0800 469 873
Update on Monday's Power Outage
Date: 28 May 08 08:11 - 28 May 08 08:11
Affected services:
At 11:58 Monday morning the top end of Wellington lost power. Within 30 seconds our generator started and Xtreme itself experienced no downtime. We ran on generator until 12:35 and then cut back to normal mains power. Unfortunately, CityLink’s equipment located at AT&T House did not last through the power cut.
CityLink worked franticly to bring the network up piece by piece and the CityLink network was fully restored at 4PM.
We are meeting with CityLink next week to discuss what they will be doing and what else we can do to ensure future power cuts have less impact on the CityLink network and our customers.
Thank you all for your patience and understanding on Monday.
Power Outage
Date: 26 May 08 11:55 - 26 May 08 16:00
Affected services: Cable, CityLink, Wireless, DSL
Following the power outages this morning CityLink are having ongoing issues at AT&T House and substantial packet loss through their network, we have no ETA at this time.
Services were fully rstored around 4:00 PM
UPS Upgrade
Date: 13 May 08 15:31 - 19 May 08 06:00
Affected services: CityLink, Wireless
Between the hours of 03:00 and 06:00 on the 19th of May we will be upgrading a UPS in one of our cabinets which feeds our CityLink and Araneo connections. This will result in loss of connectivity for less than 5 minutes while the equipment reboots.
Telecom ADSL
Date: 09 May 08 12:00 - 10 May 08 14:45
Affected services: DSL
Telecom advise they have a congestion issue on some equipmnet in Mayoral Drive. This is causing loss of connectivity for our ADSL customers, they currently have no ETA
Telecom National circuit upgrade
Date: 05 Apr 08 08:30 - 05 Apr 08 08:30
Affected services:
We will be dropping our Telecom national fibre circuit in order to perform an upgrade. During this time all national traffic will continue as normal through our TelstraClear national circuit. Our international circuits will not be affected.
Power Outage in Wellington
Date: 21 Feb 08 11:08 - 21 Feb 08 16:00
Affected services: CityLink
A substation has failed near the top end of Cuba Street affecting CityLink connections in the area.
The power technicians plan to bring the power back in stages as they suspect there will be repais needed on the power cables.
Smartlinx3 in the Hutt Valley
Date: 05 Feb 08 20:20 - 05 Feb 08 22:00
Affected services: Wireless
Smartlinx3 are currently experincing issues in the Hutt Valley resulting in loss of connectivity, they hope to have this resolved by 10:00 PM
Hutt Valley Smartlinx3
Date: 31 Jan 08 07:00 - 31 Jan 08 10:19
Affected services: Wireless
Smartlinx3 are currently resolving an issue that is causing loss of connection for some sites in the Hutt Valley area
CityLink Outage
Date: 25 Jan 08 16:07 - 25 Jan 08 17:05
Affected services: CityLink
CityLink currently have an issue which is resulting in loss of connection in the Willis Street/Victoria Street and Cable Street areas.
Smartlinx3 Planned Outage in Lower Hutt
Date: 12 Dec 07 09:47 - 13 Dec 07 00:00
Affected services: Wireless
Due to an essential network infrastructure upgrade, there will be an outage on the Smartlinx3 network in the Lower Hutt area between 07:00, Saturday 15 December and 17:00, Sunday 16 December. Smartlinx3 are removing an existing mast on top of the HCC building and installing two new ones.
More Power Outages
Date: 11 Dec 07 16:32 - 12 Dec 07 08:00
Affected services: CityLink
Continued sporadic power outages throughout the Wellington CBD are currently impacting Ethernet services on the Citylink network.
Power Loss affecting CityLink
Date: 10 Dec 07 16:15 - 10 Dec 07 17:06
Affected services: CityLink
A massive power event in the Wellington CBD has caused loss off connectivity in the Willis Street/Lambton Quay/Featherston Street/Manners areas.
The CityLink network is now completely restored.
Cable Modem Network
Date: 20 Nov 07 08:05 - 20 Nov 07 13:15
Affected services: Cable
We have received multiple calls advising that cable modem connections are not working, we have logged a fault with TelstraClear and will update this when we have more information.
TelstraClear have resolved issue which was caused by an upgrade the night before.
Email issues
Date: 18 Oct 07 14:33 - 09 Nov 07 17:00
Affected services: E-Mail
Update 09 November: After implementing our new hardware and monitoring the system we are satisfied that the issues are now resolved.
Update 26 October: Our new hardware has arrived, we will be performing a restart on the entire mail system around 17:15 this evening to bring the new hardware online to allow us to begin configuration so that we can implement it early tomorrow morning.
We have recently experienced a large increase in both spam and legitimate email which is pushing out systems to their limits. This has urgently brought forward our upgrade schedule. We will be receiving early next week additional hardware to resolve the issue, until then we have a temprary work around that will unfortunately result in additional spam slipping through from time to time. There will hopefully be no more repeated emails with our work around.
Telecom Fibre Work
Date: 12 Oct 07 09:29 - 13 Oct 07 12:00
Affected services: DSL
On Saturday the 13th between 10:00 am and mid day Telecom will be doing some fibre work in Thorndon Quay. This may result in loss of connection for ADSL connections and will result in loss of connection for dial up users during this period.
CityLink issues
Date: 06 Oct 07 09:34 - 06 Oct 07 20:09
Affected services: Cable, CityLink, Wireless
CityLink are currently experiencing issues on their network causing connectivity problems for CityLink and Wireless connections.
Border Router
Date: 21 Sep 07 05:30 - 21 Sep 07 05:45
Affected services: Cable, CityLink, Wireless, DSL, Hosted, E-Mail
We need to replace 2 x ethernet interface cards in one of our border routers.
The current ethernet interfaces are not causing any issues.
This is a recommendation from Cisco.
Router module upgrade
Date: 27 Aug 07 19:39 - 28 Aug 07 05:45
Affected services: Cable, CityLink, Wireless, DSL, Hosted, E-Mail
At 5:30 am Tuesday the 28th we will be replacing two older modules in our border router with current ones. This will result in an outage of approximately 5 minutes.
Cisco Memory Upgrade
Date: 13 Jun 07 17:27 - 15 Jun 07 03:30
Affected services: Cable, CityLink, Wireless, DSL, Hosted, E-Mail
At approximately 3:20 on Friday morning we will be upgrading the memory in our Cisco Border router. This will result in appriximately 5 minute outage across all our external connections.
Smartlinx3 UPS Upgrades
Date: 22 May 07 13:04 - 27 May 07 00:00
Affected services: Wireless
The Lower Hutt section of the Smartlinx 3 network will have a few short outages on Sunday 27/05/2007 between 8am and 1pm.
Smartlinx3 will be installing some additional UPS's at a couple of their Lower Hutt POP sites to keep the network running during this period.
The outages will occur while the temporary UPS's are installed and then removed.
Border router migration
Date: 21 May 07 08:48 - 24 May 07 06:00
Affected services: Cable, CityLink, Wireless, DSL, Hosted, E-Mail
On Thursday the 24th beginning at 4 am we will be migrating to our new Cisco 7206 Border router. Loss of connectivity should be minimal and will occur as the physical connections are moved from the old to the new. There should be no ongoing impact on end user connections.
Loss of connectivity
Date: 20 May 07 08:41 - 21 May 07 00:00
Affected services: Cable, CityLink, Wireless, DSL, Hosted, E-Mail
A repeat of the issues experienced on the 27th of April have led to the diagnosis of a failing superviser module in one of our Cisco core switches. We replaced it prior to Sunday midnight with a spare off the shelf.
Intermittent connectivity
Date: 27 Apr 07 17:30 - 27 Apr 07 19:30
Affected services: Cable, CityLink, Wireless, DSL, Hosted, E-Mail
One of our Thorndon core switches began to intermittently stop forwarding traffic. We believe we have traced the cause to a trunk port link to one of our Porirua core switches and the issue should now be resolved.
Customer CityLink Router Upgrade
Date: 26 Apr 07 13:32 - 27 Apr 07 05:00
Affected services: Cable, CityLink, Wireless
This a reminder as per the information sent with last months accounts:
On Friday the 27th of April begining around 3:30 am we will be upgrading our Customer CityLink router. This will result in a temporary disconnection of CityLink, Cable Modem and Wireless connections. Should you find your internet connection performs poorly or is non existent on Firday morning, please try a powercycle of your firewall/router device in the first instance. If you are still experiencing trouble please call us on either 04 474 0770 or 0800 469 873.
The Upgrade has been completed successfully.
ADSL Connectivity
Date: 18 Apr 07 09:13 - 18 Apr 07 12:00
Affected services: DSL
Telecom have advised of problems on their ADSL network which is affecting email and internet connectivity, they have not given an ETA for resolution at this time.
Thorndon-Porirua Link
Date: 03 Apr 07 09:00 - 03 Apr 07 14:09
Affected services:
Our link from Thorndon to Porirua is currently down due to a fibre cut in the Porirua area. This will only affect customers connected directly to our Porirua Data Centre. We have no eta from TelstraClear for resolution at this time.
Resolved at 14:09
International issues
Date: 30 Jan 07 14:45 - 30 Jan 07 15:10
Affected services: Cable, CityLink, Wireless, DSL, Hosted, E-Mail
TelstraClear's international connection is currently performing very poorly. TelstraClear are aware of this and are investigating. This will affect all international traffic.
CityLink Wakefield Street Switch
Date: 10 Nov 06 13:00 - 10 Nov 06 16:18
Affected services: CityLink
A power outage at Actrix Building has caused a number of Actrix, NZWireless and other customers in the area (Wakefield St, Blair St, Allen St, Kent Tce, Cambridge Tce) to lose connectivity. It is anticipated that a generator will arrive by 2:30pm and they hope to have all connections restored this afternoon
Mail Delivery
Date: 09 Nov 06 16:00 - 09 Nov 06 17:30
Affected services: E-Mail
Our mail system become severely swamped around 4:00 PM, the system as been restarted, is now operating properly and the backlog cleared.
Power down in The Terrace
Date: 23 Oct 06 10:48 - 23 Oct 06 15:00
Affected services: CityLink, Wireless, DSL
A power transformer under the James Cook Hotel failed this morning causing loss of power to a large portion of the Wellington CBD.
CityLink outage
Date: 04 Oct 06 10:31 - 04 Oct 06 12:00
Affected services: CityLink
CityLink are currently attending to a number of switches which have been affected by power spikes.
ETA Within the hour
Smartlinx3
Date: 15 Sep 06 13:03 - 15 Sep 06 14:02
Affected services: Wireless
Point Howard AP is currently down due to a power outage.