Your browser is no longer supported! Please upgrade your web browser now.

System Maintenance February 29th, planned at 7pm EST

Update: This maintenance has been completed. Thank you for your patience as we upgraded servers affected by this critical performance bug in the Linux kernel. We totally understand the poor timing of this issue, this maintenance was not typical in any way, however our hand was forced by the severity of the issue.

Recently Harvest infrastructure has been affected by a Linux kernel bug, which causes poor performance on servers which otherwise are operating optimally. Upgrades have rolled out to all servers which can be updated without any impact to our application traffic. This bug is starting to affect enough servers to introduce risk for our application performance, such as our database servers and the clustering servers which keep the database cluster online.

Because of the number of servers affected by this bug, we are forced to take the unusual step of upgrading our entire database cluster in one go to return to acceptable performance. This unfortunately means we need to take Harvest offline for up to 30 minutes while the servers are upgraded.

During the day today, we’ll be keeping a close eye on server performance and plan to perform this upgrade tonight February 29th at 7pm EST. Harvest will be offline for maintenance for 30 minutes or less at this time. What time is that for you?.

Due to the nature of this issue, we may be forced to perform the upgrade sooner than 7pm EST today, in order to preempt unacceptable performance levels. This issue is not affecting the integrity of our customers’ data in any way, rather producing unacceptable levels of performance for affected servers.

We apologize in advance for this outage and assure you that Harvest performance is our top priority. Thank you for your patience while we upgrade our infrastructure.

We will keep you informed of the progress of the maintenance on and via @harvest on Twitter.

Thoughts or questions about this post? Need some help?
Get in touch →

This was posted in Product News.
  • Thanks for the notification and not surprising us. It is greatly appreciated as is your product.

  • Service going down in the middle of the week during business hours? Not cool, especially for people on the west coast. I cant imagine that this couldnt have waited another couple of hours.

  • I agree with Chris. This should not happen at 4pm (West Coast, Pacific Time) on a Friday. We rely on Harvest for our business, and scheduled maintenance should happen over the weekend.

  • Wait, I misinterpreted that this was going to be Friday. But, it’s actually *today*, and with less than one day’s notice. If this is truly urgent and there’s no way it can be done after U.S. business hours, then at least an explanation of the severity and/or extenuating circumstances around scheduling would be appreciated.

  • Karen Schoellkopf on February 29, 2012

    @Chris @Jim This is happening tonight, FYI, not Friday. We try to avoid downtime during the workweek, at times that people are actually working. Unfortunately, it is unavoidable that we must take action as soon as possible, which is why we scheduled the maintenance this morning, for later today. We take your businesses very seriously, and we sincerely apologize for the interruption. Stay tuned to @harvest on Twitter or for the latest updates. And thank you for your patience.

  • Amit Dunsky on February 29, 2012

    What about redundancy/DRP?
    You found a bug which you need to deal with. But can’t you upgrade the servers using your DRP or simply have another server/s running the system while you perform the upgrade?

  • I have a team of people tracking time, estimates to put out, invoices to create. You know, that sort of essential daily business operation stuff. Late afternoon is prime time for us. All I’m saying is, if it could wait until 7, it could wait until 9.

  • Chris and Jim,

    I totally understand. Due to the severe nature of this issue, several affected servers have become unstable and require immediate kernel upgrades to return to stability. We have already upgraded most of our infrastructure seamlessly before this point, but the remaining servers require special consideration due to the number and type of servers affected. The timing and the notice period are both unfortunate, this is not how we generally approach maintenance. But this is required at this point to return Harvest to optimal performance. We’ll update you as we go.

  • That sounds reasonable, given the issues, but using the word “planned” for “emergency” maintenance (needing to be done on the same day it’s announced and during business hours) is misrepresenting it and reflects badly on Harvest’s planning process.

Comments have been closed for this post.
Still have questions? Contact our support team →