Start: 2021-07-16 at 3:15PM CDT
End: 2021-07-16 at 11:50PM CDT
During a normal maintenance window our Software Defined Network (SDN) Controller went offline for upgrades, however it failed to start after the upgrade.
We are in the process of recovery the controller.
What does that mean we manage your network devices, we are unable to change settings on your devices. However, the devices do not need the SDN Controller available to continue function. Your equipment will continue to work, however any statistics will be lost.
We will update this post with additional information, when it becomes available.
Update
2021-07-16 at 3:50PM CDT We have found the cause of the problem and are attempting a fix now.
2021-07-16 at 11:50PM CDT We have resolved the issue with our SDN Controller and has returned to nominal state.
If you have questions or concerns, please feel free to contact us.
On June 8, 2021 starting at approximately 5:39PM CDT, our alerting system informed us that are primary web and database server became unresponsive from the greater Internet.
Upon investigation, we found that our Data Center (DC) provider is having a connectivity issue across there entire global footprint. The DC provider has started investigating this large scale issue.
As of 11:30PM on June 8, 2021 we do not have any further information to share.
UPDATE: June 8, 2021 @ 11:55PM: Web Sites & Services Started to be restored and coming back online. We are still waiting for additional information from the DC about these restorations. We are assuming this may be temporary and some or all services may become intermittent at times. Additional Updates when available.
Once we know more, will update this post and our twitter account.
Timeline of Events (Time in Central Time)
June 8, 2021
5:39PM: Monitoring system indicate multiple failures in our web services.
6:20PM: Investigation into cause, found possible DC Provider failure.
9:00PM: DC Provider investigating outage
11:55PM: Web Sites and Service are coming back online.
Effected Service
Managed Web Service
Managed Mail Service
Incoming mail is spooling on backup servers, will be delivered when back online
Server is accessible, but authentication and delivery failures.
Online Invoice & Billing System
When additional information is provided, we will publish updates here and on twitter.
If you have questions, please feel free to contact us.
On May 20, 2021 we were informed that our Managed Web Service Server has been blacklisted from sending emails directly. This was a change by Spamhaus by placing the server address on a Policy Based Blacklist due to bad neighbors in our data center provider.
We are looking into options to remove our IP address from Spamhaus - this is unlikely to happen.
However, we have decided to continue to implement our plans to switch to using our third-party mail delivery service for all web applications.
We will post updates on this situation as they come available.
If you have any questions, please feel free to contact us.
Start: 2020-09-14 @ 00:00 UTC / September 13, 2020 @ 7:00PM CDT End: 2020-10-02 @ 00:00 UTC / October 01, 2020 @ 7:00PM CDT
Our Data Center Provider will be rolling out updates across there entire server fleet to improve networking performance and stability on our servers. We do not have a more specific date or time for this maintenance window, we will update this notice when we know more.
This update should have limited to no downtime associated with the maintenance.
Affected Service
Managed Web Services
Managed Mail Services
Webmail
Send & Receiving
If you have questions or concerns, please feel free to contact us.
First off, I would like to apologize for this unexpected outage to our valued clients.
Our recent managed mail service outage that was between 12:17A - 12:35A on September 10, 2020, was a result of a missed announcement of an upcoming maintenance window by our Data Center Provider to resolve an issue with the underlying server that our Managed Mail Service is hosted on. This maintenance window was missed due to multiple maintenance notices for the Month of September.
Live migration was attempted, however, failed and an offline migration had to be performed. An offline migration requires the Managed Mail Server to be powered off before the migration process. This offline migration caused the outage and extended the downtime.
Timeline of Events:
12:17A - Data Center Provider Attempt Live Migration of Mail Server to the new host, but fails
12:22A - System Monitoring: Notification that Mail Server is unavailable
12:24A - Started investigating
12:28A - Found that Data Center is performing offline migration of Server
12:30A - Migration Completed
12:35A - Server back online
As mentioned above, there are other upcoming maintenance windows scheduled for September 2020, below is the upcoming maintenance windows:
September 13, 2020 (CDT) - Hypervisor/Network Upgrades
September 17, 2020 (CDT) - General Data Center Maintenance
Again, I would like to apologize again for this outage and I take full responsibility for this error.
If you have any questions or concerns, please feel free to contact us.
Start: June 8, 2020 11:15 PM CDT (June 9, 2020 04:15 UTC) End: June 9, 2020 7:15 AM CDT (June 9, 2020 12:15 UTC)
Our Data Center Provider will be doing maintenance on the underlaying server that we are using, to resolve a potential issue with the system.
The expected downtime is currently unknown, but may last the full maintenance window. We expect to only see a minor interruption to services - including packet loss - while our system is moved to a new system.
Affected Services
Managed Web Services
Managed Mail Services
All our Online Services
We will update this post with additional information as we get more information.
If you have questions or concerns about this maintenance window, please feel free to contact us.