We've noticed significant delays in our geo distributed database cluster over the last couple of days, our investigation concluded that a bit over 50% of the packages sent between the nodes where dropped.
How will this be solved?
During Sunday, Jan 16th 5 new nodes will be deployed on 4 continents, these will be connected via a private network, which according to our tests provides less than 250ms latency from the two nodes furthest away from each others, 50-100ms on average. For read operations we'll get less than 10ms latency.
How will this affect the service?
During the day, each and every one of our websites, including the api will have a few minutes of downtime during the day. The new cluster has been up and running and stress tested to ensure that it can handle the load after the update.
When will you be back?
The scheduled maintenance is estimated to be over at 17:00 UTC Sunday Jan 16th
Is there any risk of data loss?
No, this only affects our own sites, your servers and cloud resources will be completely unaffected, tho during maintenance operation via api will not be possible.