System Status

Current Issues

International Speed issues

Friday May 9
Status: Ongoing
Services affected: Access to New Zealand Hosted websites from offshore and some New Zealand ISPs
Several customers have noticed slow speeds when accessing websites hosted on our New Zealand servers from offshore.  This problem also seems to affect at least one New Zealand ISP (callplus/slingshot).

We are investigating the issue with our upstream provider.

Recently Fixed Issues

Thursday May 1 15:00 - 15:35
Status: Fixed
Services affected: Websites hosted on brisbane server
Websites hosted on our brisbane-based Australian server (note, unless you have specifically requested this, most websites are not hosted there) were unavailable due to a failure of the Optus Uplink in the australian datacentre.

---

Friday March 8 02:09 - 08:56
Status: Fixed
Services affected: Website outage for some websites
Due to a large load spike on one of our websites, one of our database servers ran out of disk space.  This in turn led to a problem loading websites.  As all relevent services were still running however, our monitoring systems were unable to detect the fault.

Once the problem was detected at approximately 7:30, steps were taken to add disk space to the affected server to get it operational.  Then a full server restart was required to get the affected websites live
Tuesday December 4 10:00 - 10:20
Status: Fixed
Services affected: Website outage for some websites, reduced performance for up to an hour afterward.  Page logging disabled for three hours.
Effect: Intermittent ability to access websites for some customers
Details:
A routine database change was made to one of our servers early this morning - however this had unexpected performance implications that slowed this server to the point where it began causing errors on websites & eventually crashed under the built-up load.

This problem was fixed twenty minutes later (by restarting the affected server).  However flow-on performance effects resulted in reduced performance for several of our larger websites (especially for clients in the admin, and users while completing an order) for up to an hour afterwards. 

In order to reduce load to within acceptable limits we were forced to disable page logging during this time, meaning that clients will notice a three-hour "gap" in their page view statistics for this day.

---

Sunday 2007-11-25 - Wednesday 2007-11-28
Status:Completed
Services affected: Website hosting for some websites
Effect: Possible slowdown as your site is copied
Details:
In order to prepare for the christmas rush we are in the process of performing server upgrades in our hosting platform.  This may involve us moving some websites to new servers, a process which should be transparent to the user.  However while this is happening users may experience some slow-down, and in some cases a website may need to be switched to read-only mode to avoid order inconsistencies.  At most this move generally should take less than three minutes per website.

---

Friday 2007-11-16 8:01 AM - 9:05 AM , again at 9:45 - 10AM
Status: Fixed
Services affected: Website hosting for some websites (Other websites were unaffected)
Effect: Website intermittently slow/inaccessible
Details:

Disk space was exhausted on one of the three "overflow" servers, which are used to temporarily provide extra hosting capacity to websites experiencing higher-than average load. These servers have recently been added to help handle the Christmas rush & other unexpected load spikes that sometimes affect our clients.

This resulted in page requests to that server hanging, causing 1 in 3 page requests to fail.  This server was disabled while the problem was diagnosed, but was later re-enabled by another engineer causing the problem to recur later that morning.

To avoid this problem in future the scheduled task that cleans up used disk space on these servers has been modified to run more frequently, and more sophisticated monitoring of disk usage has been implemented for these temporary servers.

---

Thursday 2007-9-20 9:30 - 10:24 AM
Status: Fixed
Services affected: Website hosting for 20% of websites (Other websites were unaffected)
Effect: Intermittently, access slow or site inaccessable
Details:

Unexpected load overwhelmed one of our five file servers, causing it to run extremely slowly.  This results in the websites being slowed down (in some cases to the point where people we unable to access them at all).

To resolve this we are moving some of our website's file hosting off this server to increase its spare capacity.  Within the next month we plan to replace this server with a much more recent and faster system as part of our ongoing upgrade processes.

Archived/Historical Issues

Tuesday 2007-9-11 2:38 - 2:50 PM
Status: Fixed
Services affected: Website hosting for 20% of websites (Other websites were unaffected)
Effect: Access slow or site inaccessable
Cause:
A backup script hanging because of excessive disk usage on one server caused scheduled tasks that should run overnight to run in the middle of the day during peak website load.  This overloaded one file server, slowing down the websites running off it to the point where they no longer responded at all.

To fix this issue this scheduled task is being moved to another server where it cannot be blocked by this backup script.

Friday 2007-8-31 10:30 - 10:35 AM
Status: Fixed
Services affected: Website hosting for 20% of websites (Other websites were unaffected)
Effect: Website access blocked for 5 minutes
Cause:

A hardware failure of one of our fileservers blocked website access for approximately five minutes.  Automatic failover of the machine to its backup failed due to the nature of the hardware error (a partial failure rather than a complete failure that our failover system could not detect), however our engineers were notified and were able to manually fail-over the server.

Saturday/Sunday 2007-04-29 10:40PM-2:06AM

Status: Fixed

Services affected:

Website Hosting: yes

Email: No

Domains: No

A database server outage meant that many websites were unaccessable during this period.

Detailed description

Our main database server filled a disk with logs while experiencing much-higher than average load.  This outage also affected the SMS notifications from our monitoring system meaning only email notifications worked.  Combined with the fact that this outage occured overnight on a Sunday this resulted in the long outage window.

2007-04-26 9:06 - 13:01
Status: Fixed

An issue with our email provider meant that all customers were being asked for thier passwords whenever they try to check thier email.  

Affected Services
Hosting : no
Email :  yes
Domains : No