tools.assembla.com is down

If you are an early user of trac and svn on Assembla, your server is down. I apologize for the inconvenience.

Tools.assembla.com is an old server that does not have the same failover capability as our new subversion servers. We shut it down to check some disk errors, and it did not come back correctly. So, we are now working hard to move the files onto a server that works correctly. The server could be down for another three hours.

Some details: we had some IO errors in server logs, this usually is due HDD failure. The fix is simple – the data center needs to identify bad hdd from raid and replace it with new hardware. what they did: they ran 3 times fsck on partition with errors in last 18h and update support ticket hourly with progress – 50%, 60%, 87% and so on.

2 years the planet hosting works very well, but after the Houston explosion, things changed. Explosion took 3 our servers down more than 24 hours, after 4-5 days something happens and network cable was disconnected from db server – another 6hours of downtime, now with this hdd failure they cause our tools.assembla.com to be down for more than 18h with simple hardware replacement.

We can not work anymore with them after such incidents and we will move our services to amazon EC2 ASAP.

waiting…

Please, can you estimate how long the server will be down?

Now I have information that we need to build a new server. We are working on two different servers and we will release the one that comes up first. Unfortunately, this takes a lot of time because we have a large volume of backup files to move. It will not be working before tomorrow morning – Tuesday morning US time.

Hi Andy and Vitalie.

Thank you for providing an excellent free project management and hosting solution. I’m quite new here, so I haven’t come to depend on it too much yet (lucky for me only my blog is down). But I have a good feeling you’ll sort out this mess soon and migrate to the amazon service.

Keep up the excellent work.

- Steve

Hi Andy and Vitalie, is there any update on this yet? It’s becoming a little awkward not having access to my repositories… I appreciate the updates you’ve provided but Tuesady morning (US) has been and gone and still no access.

Any news yet?

Also waiting on an update. Please let us know where you guys are on this issue. Some sort of updated timeline would be useful.

I’m not a fan of the whole me too posts, but.. I can haz codez?

We expect to have the server back online about midnight GMT, 8:00 US EDT. After some misguided attempts by the datacenter to repair the server, we configured a new server. We are now copying backup files to that server. We are copying 200 Gigabytes of files over a 100 mbit network. This is taking a long time.

If you are desperate I cane move your repositories to one of the new servers that works correctly. You will get a new subversion url. You can email me at info@assembla.com with your space URL name.

Retaining this old server without failover options was a mistake. We will move all of these tools repositories to virtual servers with gigabit networking in the next few weeks.

My sympathies. There’s nothing worse than when a hosting vendor lets you down. It wasn’t so long ago that my own public servers were knocked offline for 6 hours when my colo provider was doing some electrical work, killed the main power, and then found out their “redundant” UPS didn’t work when the whole building went dark and quiet. Oopsie! The power surge even fried one of my load balancers. So much for “conditioned power”. This was a Tier 1 telco provider, mind you. They even took down the city’s E911 for a few minutes, which needless to say is a very serious boo-boo.

My lesson learned: you can’t depend 100% on any vendor. The only remedy is to make sure you’re not completely dependent on any one vendor for anything. This is very hard to do, and usually rather expensive.

The amount of dedication from the Assembla team, the amount of money they spend on providing what many probably use as a free service, the constant updates…

Nothing else to say than thank you very much for providing this excellent service even throughout circumstances that were due to no fault of your own, you have my utmost respect.

I just wanted to say hanks for tackling this issue. I know it’s a pain when stuff like this happens, especially on a Monday. Your work and service are appreciated.

My only concern is: the URLs are going to be the same, specially for the trac? Because these URLs are on my resume, and right now I’m searching for a new job…

I’m not in a hurry, to say the truth, just need to have the same URL for the trac.

Hi, any update on that midnight GMT time?

Thanks,

There’s no chance of data loss is there?

Hi,

First thanks for tackling the issue. Its much appreciated. Any updates on the issue or new estimates ?

Thanks.

Are we gona have our workspaces back again? or can you move the workspaces into an other server?

Andy and Vitalie!

Thank you very much for the prompt and detailed status updates! It makes me feel confident that everything will be fine eventually.

Victor

Server is back,
read more .

Thank you! Guys your are the best :) Best service, best project. I galb what I founded you some years ago.

server back but workspace destroyed :/