Start: 2020-09-14 @ 00:00 UTC / September 13, 2020 @ 7:00PM CDT End: 2020-10-02 @ 00:00 UTC / October 01, 2020 @ 7:00PM CDT
Our Data Center Provider will be rolling out updates across there entire server fleet to improve networking performance and stability on our servers. We do not have a more specific date or time for this maintenance window, we will update this notice when we know more.
This update should have limited to no downtime associated with the maintenance.
Affected Service
Managed Web Services
Managed Mail Services
Webmail
Send & Receiving
If you have questions or concerns, please feel free to contact us.
First off, I would like to apologize for this unexpected outage to our valued clients.
Our recent managed mail service outage that was between 12:17A - 12:35A on September 10, 2020, was a result of a missed announcement of an upcoming maintenance window by our Data Center Provider to resolve an issue with the underlying server that our Managed Mail Service is hosted on. This maintenance window was missed due to multiple maintenance notices for the Month of September.
Live migration was attempted, however, failed and an offline migration had to be performed. An offline migration requires the Managed Mail Server to be powered off before the migration process. This offline migration caused the outage and extended the downtime.
Timeline of Events:
12:17A - Data Center Provider Attempt Live Migration of Mail Server to the new host, but fails
12:22A - System Monitoring: Notification that Mail Server is unavailable
12:24A - Started investigating
12:28A - Found that Data Center is performing offline migration of Server
12:30A - Migration Completed
12:35A - Server back online
As mentioned above, there are other upcoming maintenance windows scheduled for September 2020, below is the upcoming maintenance windows:
September 13, 2020 (CDT) - Hypervisor/Network Upgrades
September 17, 2020 (CDT) - General Data Center Maintenance
Again, I would like to apologize again for this outage and I take full responsibility for this error.
If you have any questions or concerns, please feel free to contact us.
Start: June 8, 2020 11:15 PM CDT (June 9, 2020 04:15 UTC) End: June 9, 2020 7:15 AM CDT (June 9, 2020 12:15 UTC)
Our Data Center Provider will be doing maintenance on the underlaying server that we are using, to resolve a potential issue with the system.
The expected downtime is currently unknown, but may last the full maintenance window. We expect to only see a minor interruption to services - including packet loss - while our system is moved to a new system.
Affected Services
Managed Web Services
Managed Mail Services
All our Online Services
We will update this post with additional information as we get more information.
If you have questions or concerns about this maintenance window, please feel free to contact us.
Start: May 24, 2020 11:15 PM CDT (May 25, 2020 04:15 UTC)
End: May 25, 2020 7:15 AM CDT (May 25, 2020 12:15 UTC)
Our Data Center Provider will be doing maintenance on the underlaying server that we are using, to resolve a potential issue with the system.
The expected downtime is currently unknown, but may last the full maintenance window. We expect to only see a minor interruption to services - including packet loss - while our system is moved to a new system.
Affected Services
Managed Web Services
Managed Mail Services
Our Online Services
We will update this post with additional information as we get more information.
If you have questions or concerns about this maintenance window, please feel free to contact us.
Starting at 11:00PM on July 26, 2019, many of our online services became unavailable due to expired security certificates.
This issue was caused by a silently failing scheduled task that would normally renew the certificates automatically and a notification system failing to send warnings about the expiration of said certificates.
It came to our attention at 1:30AM July 27, 2019 that the services were unavailable, we remedy the issue by 1:40AM. We will be implementing additional safe guards to prevent this issue in the future.
We would like to deeply apologize for this problem.
Stage Two may be delayed until Sunday, we are looking into the matter to see if it would be better to delay this Maintenance Window. We will update this post with more details later today (May 18 2019).
We will be doing some major maintenance this weekend on all of our online services. We have divided this maintenance into two stages, due to the extensive maintenance that is required.
We expect up to 14 hours of possible downtime, however, we estimate the actual downtime will be around 4 hours for each service or a total of 8 hours.
We may not need the Stage Two Maintenance Window, however we wanted a fallback window just in case problems arised.
We will update this post during the maintenance and also on Twitter (@MadSciTechCloud).
If you have questions or concerns, please feel free to contact us.