Your browser is no longer supported! Please upgrade your web browser now.
Posts by Warwick Poole:

Harvest Is Moving Datacenters and Your Data Is Coming With Us!

Post Migration Update:

If you are reading this, then you are hitting Harvest servers in our new datacenter. The migration went well, except for some performance issues which took us a while to investigate. Harvest was offline from 9am EDT until 1:55pm EDT. This was a very long outage, and not something we are looking to repeat anytime soon. We look forward to serving your needs from our new datacenter and thank you very much for your patience while we made this switch.

On our Maslow’s hierarchy of needs, we place our customers squarely at the base of our pyramid. We simply don’t exist without the support of our customers and our part in that relationship is to provide useful applications reliably. We are pretty proud of our reliability thus far, but there is always room for improvement. To that end, we have been working feverishly behind the scenes to bring online a whole new set of servers with a new hosting provider. We are pleased to be partnering with our friends at Voxel to make Harvest more reliable and more responsive for our global customers.

Moving hosting providers is no mean feat and we’ve done the heavy lifting over the course of the last few months. To flip the switch we need to take Harvest offline for a few hours and sync up our two datacenters. Downtime is a curse word around here, but sometimes it is unavoidable.

We’ll be taking Harvest offline for 4 hours on Sunday, September 11th between 9am  – 1pm EDT.

What time is that for you?

We are sorry about the downtime and we will work to minimize the outage window, but we trust this move will mean more stability for our platform as it grows. Thanks for your patience as we make this important switch.

If you have any questions or concerns, please email us at and we’ll be glad to talk with you about this move.

Scheduled Maintenance Sunday July 3rd, 8am – 9:30am EDT (complete)

Update at 9:55am EDT: We have completed this maintenance. We had to take a little extra time to finish it up and Harvest came back online at 9:54am EDT. Thanks for your patience while we upgraded.

On Sunday July 3rd, 2011 we plan to take Harvest offline beginning at 8am EDT and ending before 9:30am EDT for some database upgrades. What time is that for you?

Sorry for the short notice on this one, and please follow @harvest on Twitter for real time updates from the team during this work. Thanks for your patience, folks.

Maintenance: Sunday March 6, 1am – 2am EST

Harvest will be offline for 60 minutes between 1am EST to 2am EST on Sunday March 6th, 2011. What time is that for you?

This coming weekend, our datacenter provider is performing network maintenance to improve network redundancy. Many core services in the datacenter will be offline for 5 hours, between 1am – 6am EST, Sunday March 6th.

We have reconfigured a number of our systems so that we will be able to stay online during most of this maintenance, even when a large set of datacenter resources go offline. That said, we plan to take advantage of this maintenance to perform some upgrades of our own, increasing our storage capacity by more than 400%. For this reason, Harvest will be offline for 60 minutes.

During the extended 5 hour network maintenance we will be monitoring the progress and doing everything possible to keep Harvest systems online.

Follow @harvest on Twitter for real time updates from the team during this work, and thanks for your patience.

Secure Connection for All Accounts

Last year, we activated SSL on all Solo, Basic, and Business accounts.  As we continue to ensure the highest standards of security, we will now automatically enforce SSL on all plans, starting September 1st.  This means all trial and free accounts, as well as paid plans that previously did not take advantage of it, will now have SSL enabled by default. We want to share how this will affect you, your Harvest account, and any Harvest add-ons you may rely on.

For the non-technically inclined, HTTPS (SSL) is a security layer which ensures that the server your browser is communicating with belongs to Harvest (and is not an impostor) and also encrypts your communication with Harvest servers, to prevent eavesdropping.

What does this mean for you?

For the majority of Harvest customers who access Harvest using a modern web browser (Mozilla Firefox, Internet Explorer 7 and above, Chrome, Safari, etc) this change should be totally seamless and you’ll not need to do anything.  Depending on which browser you use, you might notice a new hint in your address bar that your connection to Harvest is now secure.

Firefox HTTPS hint display

Here is an example of an HTTPS hint in Firefox 3

Do you use one of our Harvest widgets?

  • If you use the Mac widget, you don’t need to do anything differently, it’ll work the same as it does now.
  • If you use the Yahoo widget, you’ll need to check the “SSL” checkbox on the widget options after we make the change September 1st.
  • If you use the Vista gadget, you may actually need to re-install the Vista gadget after we make the change September 1st.

Have further questions?  Feel free to contact us.

Technical Information for Harvest API integrators and authors of Harvest API clients

For Harvest customers who have integrated Harvest into other systems using an API client we need to make sure that your integration continues to function properly. We’ve begun reaching out to authors of popular Harvest API clients already. Many thanks to Zach Moazeni, Brian Glass and Matthew Denton for helping test their Harvest API clients so far. We will update this post with any update news on these clients.

If your Harvest API integration has in the past communicated over HTTP (ie: not HTTPS) with Harvest, from September 1st onwards these requests will be met with an HTTP 302 redirect to the HTTPS location of your Harvest subdomain. Your API client will need to support SSL as well as follow these HTTP 302 redirects, to continue to communicate with Harvest. Please open a support ticket if you need any help with this, or have any questions about supporting SSL.

Technical note: Why are we using an HTTP 302 redirect and not an HTTP 301 or an HTTP 303? We suspect there may be API clients which will follow a 302 and may not follow a 301 or 303. An API client which will obey any redirect on the 3xx class is a good idea. Please contact us for more details on this issue.

Developers, have any further any questions? Please visit the Harvest forum.

When Networks Attack

Not everyone on the Internet plays nicely. Recently Harvest has been affected by DDoS (Distributed Denial of Service) network attacks directed at sites with equipment physically near our equipment in our primary datacenter. There are some bad forces on the Internet and in the last few weeks there have been two incidents where those bad forces have attacked sites which have nothing to do with Harvest, but unfortunately happen to share a network path into our primary datacenter with our equipment. When these attacks have taken place, the sheer volume of bad traffic has saturated some of those network paths and caused Harvest to be slow for some users. Usually the effects last only a few minutes until the source and destination of the attacks are identified by the datacenter personnel and are removed from the network. Nevertheless, that’s a suboptimal situation for Harvest and our customers.

We’ve taken steps to mitigate the risk of these DDoS issues affecting us in future. Part of this involves moving some of our front facing load balancers to new locations in the primary datacenter where the network pipes are so large that issues of the type we have seen recently will not have any negative impact.

The first set of servers was moved this evening, some users may have noticed Harvest run a little slow for 20 minutes while this work was done. Co-op in particular was slower while we moved. Next week we’ll be moving the rest of the servers and I’ll update you when that work is complete.

Brief scheduled maintenance Saturday May 22 at 2am EDT

Sorry for the late notice on this one folks but we’ve got some features coming down the pipeline which require us to make changes to our databases. Sometimes this can be done while Harvest operates normally and sometimes we need to take Harvest offline to make these changes. We’ll be taking Harvest offline at 2am EDT on Saturday May 22nd for approximately 60 minutes. What time is that for you?

Follow Harvest on Twitter for realtime status reports. Thanks for your support as we improve Harvest!

Weekend network maintenance could impact some users

Our datacenter provider, Softlayer, is performing network maintenance over the course of this weekend (Saturday April 24th, Sunday April 25th) which could make Harvest and Co-op slow, or inaccessible, for some users for very short intervals. Softlayer assure us that all care is being taken to ensure minimal impact, if any, to the network. We apologize for any inconvenience caused if you experience any difficulty accessing Harvest. This maintenance is required to introduce improved network infrastructure in the datacenter.

Brief service interruption: Friday April 23 (updated)

Update 6am EST: A second network attack was directed at our datacenter in Dallas, effectively taking Harvest and Co-op offline for a further 60 minutes between 4:45 am EST and 5:45am EST. All service should now be restored once again. Softlayer, our datacenter provider, have assured us that the particular customer involved here has been removed from their network. Really sorry that Friday started out this way, folks. Thanks for your understanding.

Original incident: We are really sorry for a brief service interruption which affected some Harvest and Co-op users over the last hour or so. Around 2:05 am EST on Friday April 23 a directed network attack was launched against our datacenter provider in Dallas, TX. The target of the attack is unknown to us currently. Our datacenter provider had to take steps to mitigate the problem and this included interrupting many network services upon which Harvest depends. This meant that for 65 minutes between 2:05 am EST and 3:10 am EST Harvest and Co-op were unavailable, or incredibly slow, for some users. We are waiting for more details on the issue and hopefully steps which will be taken by our provider to mitigate these types of problems in the future.

Thanks for your patience and as always, for realtime status information we suggest you follow @harvest on Twitter.

Scheduled Maintenance: Thursday 18th, 1am EST

You can no doubt tell that Harvest takes our responsibility of hosting our customers’ data very seriously from the recent “behind the scenes” posts on maintenance and service availability. Yesterday we discovered a bug in the database software we use, which can cause intermittent speed problems for parts of the site. Harvest has been built to be fast, and we love how frequently Harvest users remark on the responsiveness of the application. We initially planned to upgrade this database software as part of the upcoming infrastructure upgrade but we have decided to bring the upgraded database software to Harvest sooner rather than later.

At 1 am EST on Thursday 18th (what time is that for you?) we’ll briefly take Harvest offline for 30 minutes to upgrade the database software.

Thanks for your support, it’s much appreciated.

Server Management Made Simpler

Many people assume that under the hood there are mystical unicorns and magical elves powering Harvest, but the reality is that we simply use regular servers in a datacenter to bring our applications to you. No elves involved. Anyone who has ever had to look after more than a few servers knows that getting all the servers into line is a bit like herding cats, and can be time consuming. It’s always important to know exactly what state your server configurations are in, and usually very important to know that all servers are in the same configuration state at all times.

It’s no secret that many of the fine folks who work on Harvest are Ruby experts (and thus quite easy on the eye). So when looking around at tools we could use to make our servers simpler to manage, we decided to try Chef. If you are not familiar with Chef, it comes from the folks at Opscode and is a powerful tool to manage server configurations, and perform common system administration tasks on a set of servers. Chef is written in Ruby and allows you to accomplish a fair amount of your system administration tasks by writing Ruby recipes to execute on remote servers. Chef makes system administration a bit more like application programming, in a sense.

While we work on getting Chef into daily use, and indeed while we improve our systems platform in general, we plan to share some of our experiences with Harvest users and readers of this blog. You can read up on some of the details of how we got Chef up and running quickly, a little look at how a Chef cookbook, recipe and role work together and we plan to bring you some more details in future posts. We’d love to hear from any Harvest users who are using Chef, or doing interesting systems or server management work (and invoicing clients via Harvest, naturally). What systems projects have you been working on?