"The more I find out, the less I know."

Thursday - December 11, 2003 at 03:37 AM in

Bad karma

Yesterday was the Bad Karma day. Someone in my company must have driven over a squirrel or swatted an endangered housefly, because the universe clearly did not like us.
To begin with, we were still cleaning up the mess our development server left when it died a week and a half ago. Then, two of our five people were out (one because of illness, the other because the snowstorm made him decide to work from home).

Early in the morning the main database server went down after a cooling fan failed.

Next, both our primary DNS (which we host) and our secondary DNS (which is hosted by a service provider) failed. This left us off the air.

One of our production voice T1's failed due to a faulty line card at Qwest.

A subtle change at AOL caused one of our normally automated processes to go manual.

An overly aggressive mailing caused our E-mail server to thrash, nearly eliminating E-mail services for a short time.

By the end of the day, everything (OK, nearly everything) was working again. But yesterday was one of those days which left me feeling like I'm in charge of a sausage factory (i.e. "If you like the product, don't watch to see how it's made").

Posted at 03:37 AM | Permalink | | |

Powered By iBlog, Comments By HaloScan
RSS Feed