HOME

OUR STORY
PAYMENTS
WHAT HAPPENED
EWONCALLS
YOUR FEEDBACK

LETTERS
O = out, I = in

MEDIA

COMPLAINTS
LODGE
OMBUDSMAN
MANAGEMENT

ABOUT
EWON
RIGHTS
ENERGY LAW

LEGAL ACTION

COMPLAINTS
HOTLINE
(02) 8825 6119

This was no good. I hadn't had a shower yet!  I figured we'd leave the tank for the day and see if it warmed up later. Maybe the power outages on Saturday (27 January) had caused Integral Energy's ripple transmitter to fail.  No doubt they'd be aware of it.

Wednesday morning came and still no hot water.  Now it was getting serious.   Two kids a wife and a slowly becoming stinky hubby!  I boiled water for the bath tub, that got me washed.  The kids had a splash around and Suzi had a COLD shower!

We were very busy on the Wednesday (31 January) and I promised I go and look at the fuse box and water heater sometime during the day.  It never happened.

Thursday came and I had a meeting in the morning.  Just after I left I thought that I should have instructed Suzi to call Integral and get them to check and test the Ripple Receiver.  I couldn't imagine it being the water heater to be really honest.   Later that day, Greg (one of my assistants) went out and ran a meter across the water heater, checked the circuits and all looked fine.  Suzi called Integral and asked for someone to come out.  And this was the beginning of the third worst day of my life.

I'd arrived back from my meeting and was in a phone call with a client.  It was exactly 16:28.  We were getting into deep and technical conversation about an upgrade path for a project.  Suddenly the phone went dead, my desk lamp went out, my monitor went off.  Then it all came back on again.  Then it went off again.  Then on, and off, and on and off.

I panicked, this wasn't good!  I dropped the now dead phone, raced to the front door, found the keys and rushed to the fuse box.  Worst case I was going to turn off the main breaker to stop any more damage being done.

I stopped dead in my tracks meters from the fuse box.  There was a guy standing there with the door open and fiddling inside.

"What the hell?" I shouted.  "Who the hell do you think you are?"

He said "I'm from Integral Energy, come to check your hot water fault."

I won't repeat my very uncharacteristic string of profanities in the next two sentences but in short I asked, between expletive, what the hell this guy thought he was doing flipping the Main switch without first coming to the door of the house and talking to someone, yet along flipping it a dozen times.  I tried hard to tell him he just killed 20 computers and heaven only knows what other damage has been done, that he was the most incompetent idiot I'd ever seen.  My wording was a lot stronger than that.

His only response was "Well do you want me to check your hot water or not?   I'm just trying to fix it for you?"

I asked for his name.  He said "Brian."  Oh boy, this guy has no clue!

I asked for his last name, he refused to give it to me.  I asked for his managers name, he refused to give it to me (some of you will find this all very familiar! Only this time I used rude words!) so I asked for his managers phone number.  He gave me 131003.  Great.  Waste of brain space this guy is.

I went inside and dialled Integral and was put on hold.  This didn't help me at all.  I threw the phone to Suzi who by this stage was wondering what was going on, because even whilst I was running outside, power was going on and off, and when I was on my way back, he'd turned it on and off and on just one more time.  Probably to be sure everything in the house was destroyed.

I turned all the main power off to the office.  Systems were screaming, power supplies were hissing the noise was terrible.  I knew I was in for a long night.   Suzi started talking to someone.  I told her to tell the person I wanted to speak to the general manager.  She wasn't getting far so I took the phone. I insisted on speaking with a supervisor or manager.

I spoke with Lochlan Morrison, Customer Service Manager (??).  By this time it was around 16:45.  I had no time services, all our servers were offline, the PBX had a scrambled time stamp and I was in a panic.

I told Lochlan that I wasn't impressed and that I wanted to lodge a formal complaint.   I detailed the problem.  I was put on hold.  I grabbed another phone and called Greg (my assistant) who'd just left only an hour earlier to pick up his kids.  

Lochlan came back to the phone.  I used a string of words, no rude ones, but alluded to the fact that this onsite engineer was the biggest incompetent I've ever had near me in my life.  Even Telstra aren't that incompetent.

Lochlan agreed with me and told me that I'd need to put the matter in writing to the Integral Energy Claims department addressed to PO BOX 6366 Blacktown NSW 2148.  We discussed compensation for the damages.

I wasn't posting any letters.  I knew this was going to be painful as it was.   I asked for a fax number.  I also suggested he get a manager to call me ASAP as I wasn't going to be very happy about this at all.

As soon as I got off the phone, I called Ross Wheeler from Albury Local Internet and asked him to log our offline status and to monitor for our network returns.  And if there was anything he could think of doing that might resolve our longer offline status, as I suspected it was going to be a while, then call me.

I turned each individual server switch off.  I was going to have to go through each machine, one by one.  But where to start?

Root servers first, without those we don't even work.  I power on RS2, it whistled and you could smell the silicone smoke.  Dead.

I moved to RS1, it booted.  I thought this was a good sign, but then the Hard Drive test failed.  I could hear the hard drive, it was clunking.  Not a good sign.  I switched it off.  Both RS2 and NS2 were at the bottom of the rack. Most of the other servers were on top. They had been successfully running for the last 5 and a half years without a hitch.They were the most reliable, longest running servers I have. Brownouts and power surges never hurt them.  But what happened this day in those 30 seconds was nothing like a brownout or power surge.  It was total negligence, gross incompetence and totally ignorant.

I decided to check the web server next.  I powered it up, it started, but once the Operating system started to load and check the hard drive, it looked bad.  Hundreds of bad sectors, lots clusters, block duplicates and much more.  I felt like collapsing on the floor and crying.  This wasn't going to get any easier and there were more than a dozen servers and workstations left to check.

Greg his wife and kids were great.  They arrived about 15 minutes later and Greg started to assess the damage.

His wife and kids, and my wife and kids vanished into silence in the lounge room.   They knew within the next two hours, Greg and I were going to be wrecks.  We had Root Servers, DNS servers, Web Servers, Dialup Access Servers, Accounting Servers, SQL Database Servers, Fax Servers, video servers, security systems and god knows what else offline and in unknown states.

I briefed Greg on the current status and asked him to scrounge parts.  I'm fortunate that we were about to deploy a number of new servers for new projects and there were parts laying around.  Greg and I looked at how to get the bottom two servers out of the rack.  Something I'd never considered I'd ever have to do.

We decided to slide a sheet of timber over top of RS2 and jack the timber and servers upwards. This worked and we slid RS2 out of the nest.  It wasn't pretty. 

We then pulled RS1 out.  RS2 was a desktop server, RS1 was a mini tower.  If worst came to worst we'd alias the IP addresses, and make one server from the two. This wasn't so easy.

Greg found a standby power supply suitable for the mini tower case.  His son (9 year old) Matthew came into the Workshop and wanted to help.

I gave him a screwdriver and told him to remove the covers off RS1 and RS2.  The to carefully remove the Hard Drive from RS2 and the power supply from RS1.  I could see we were all going to be very confused very quickly.

Greg swapped the power supplies.  I found a spare hard drive suitable for RS2 and we powered the machines up.  Not looking good.  The Hard drive controllers in both machines were shot.  I went back into the office and climbed up to the spare parts, finding two spare (thank goodness) Hard Drive controllers.

I threw the boxes at Greg and told him to label and toss all the old parts into a big box.  Worry about status later. 

We converted RS2 into RS1 as we still had the original RS1 drive working.  We then copied the new RS1 drive to the new RS2 drive. reconfigered the server specific details, adjusted the DNS configuration so RS2 was the primary and RS1 was the secondary and fired them up.  This whole procedure took a LONG time.  There wee so many fiddly things and we were getting confused between which system was which as labels said one thing and drives said the same or different.  It was the hardware that was the most concerning.

I started the CISCO routers and brought them online.  I was just hoping Telstra hadn't done the usual thing and screwed up our ISDN and DIGITAL services like they usually do.  Fortunately all our uplinks came up.  The Routers are pretty stable, but I had no idea what damage had been done to them.

I started to check the Name Servers.  These are critical to our operation because without them people can't communicate with us at all.  NS2 and NS4 were okay as far as booting and loading were concerned. There were a number of bad blocks and duplicate sectors, which I force cleaned using the file system tools.  This had never been necessary on these servers before.  They had been running since the mid 1990's, without ever having a problem.  They came up, to the point of needing the root servers to work. 

I then checked the SQL server.  This is critical for logging web server activity, authorising users of the hundreds of secure web sites we operate, authenticating dialin users and granting access and accounting their usage and activities.  I was really nervous, although I'd made a backup of the data, it wasn't current for that days activities. 

SQL booted fine, it came up, had a few false starts, grunted and stopped.  It wasn't looking good.  I powered down, let it rest and went t the next server.

I decided my dial server was next most important.  Lucky enough it just checked the file system and was pretty happy - except it couldn't talk to the SQL server and that meant modems were going to answer and users weren't going to get access.  I shut down all the services and left DIAL running idle with nothing happening.  I felt it would be good to see if it stayed active for an hour!

I moved to the next few servers and mostly things were OK.  Some of the hard drives sound pretty bad, and still do today, but they are running - for now.

I went back to the SQL server. It was critical now.  I restarted it and up she came.  Loaded all modules, SQL server started up and it connected to the network.   I was starting to feel a little better, but was very concerned about the heads crashing and slamming on the platters in the drives as the power was turned on and off repetitively by the incompetent "Brian."

The web server was next.  It's a very new server, only 6 months old, if that.   It's very critical.  It handles e-mail for thousands of people, has thousands of web sites hosted on it and is a fairly busy machine.  It also provides a backup server for our video services.

I started my main workstaiton, also NT Workstation.  It was horrible.  It did four systems checks, rebooting each time.  This machine is only 8 months old.  I was not happy again.  I was losing faith in achieving a most system up status.

I decided to leave it for a bit.  I'd connect later.

It started up ok, but when it came to the hard drive it was a mess.  Lost chains, bad blocks, noisy grinding sounds.  It wasn't good.  After an hour going through and cross checking every bad block and lost chain, I managed to get it to clean boot.   It came up, but the lost data that had been moved to the lost+found space was huge.   Log files, billing data, e-mail, web sites.  I felt like crying again.  I needed Coke, Chocolate and Ginger Beer.

After lots of other systems and service checks, I managed to get the Web Server partially running.  I particularly wanted the IRC servers up so I could talk to Ross and other ISPs about our status and get any other help I could.  Getting it up partially and shutting down all the other services worked - so far.  I went back to my workstation.  It had finished its endless rebooting and systems checks but was complaining abut not being able to find servers and resources.  DNS problems, we still didn't have any!  I connected to our IRC server using IP addresses and started to chat with the ISPs as they came online.  They were all puzzled as to why we'd been offline and why nothing was resolving properly.  They could of course still resolve my domain names because their DNS servers still had the data in it - at least for a little while.

Greg yelled out that RS1 and RS2 were ready to go - he thinks!

We gave them a quick test on the lab network and they seemed to work.  We had a few services problems, like telnet into RS2 wasn't working, but that's hardly a concern as we have local console access.  It could be fixed later.  What was important was that the Root Services resolved.

It was around 9 PM now.  We slid RS1 (which use to be RS2) into place.   Lowered the timber, slid it out and installed RS1 (which was pretty much a new box now.)  Added network cables for visible and private networks and powered them on.

HOORAY!  They came up and connected.  A few quick tests and we were happy. Make that happier.

Things were looking better and it was getting late.  Greg asked if there was anything else he could do to help.  I was less stressed now and whilst he'd fixed the Root Servers, I'd managed to get most other essential services going.  Thanking him and his family I sent them home.  Greg could always access the network from home (when the dial up services were working again) and connect to servers and fix things as needed.

I restarted NS2 and NS4 so they knew they should talk to RS1 and RS2.  They both came up fine.  I went back to the SQL server and restarted it.  I also started ALPHA, our back office firewall machine, but it also caters for special services.   Alpha and SQL were having problems.  They have a  dependency with the Web Server.  Security things weren't working and they didn't want to work either.

I restarted the Web Server again.  It came up and was much happier now it had two DNS and two root servers to talk to.  Alpha restarted and came up fine, although it had clock synchronisation problems and shut itself down.  SQL was desperate to find Alpha as it's not connected to the Internet and needs Alpha to actually handle special security issues to allow it to sync it's clocks and start up properly.  I was starting to see a problem.

After about an hour, I managed to get Alpha and SQL up and running.  That meant I could restart the Dial server and most things would be working.  I had to restart some of the services on the Web Server as they need SQL as well.

Dial came up and connected to SQL.  WEB was already talking to SQL.

Next was our Adult Content Server.  It was only days old. I'd spent the last six months working on this project, blood sweat and tears were in it, and plenty of hubby-wife arguments.  Sexy started up and checked it's file system.  I was really nervous.   Backups are backups, but production servers aren't always the same after events like this.  Sexy passed most file system checks, had a few things to fix, raised a few things to my attention and started almost fine.

There were some destroyed photo libraries and a couple of the online videos were corrupted, but that wasn't too much a concern, it was getting the users back into it - after all, that's what pays the bills!  Sexy connected to the SQL server and ALPHA just fine.  No network problems.  The Web service itself didn't start, I had to push that along a bit.

I then started up the fax servers, and workstations.  Most came up OK.   Suzi's PC runs NT Workstation, it grumbled a lot about not being able to connect to the Web and Alpha servers.  So something we still wrong, but I wasn't too concerned about it right now. 

I checked our Video Server, which ahs really noisy drives in it now.  We're trying not to use it too much for live or real data as it's likely to crash it's heads at any time.  Even the CPU is running flaky.

With most things running now, it was around 11:30 PM.  Greg dialled in and connected.  He fished around testing things, I probed the network for problems reading log files and other things.

I noticed a few problems on the WEB server over the following hours.  I had to restart a lot of services and even reboot a few times.  Seems the memory DIMMS were flaky.  I managed to create a clever work around that stopped the operating system using suspect sections of memory.

That was pretty much it for Thursday, February 1, 2001.

But I had no idea I was going to go through the Public Service Hoop Jumping process as I had previously with the kidnapping of my son, all over again.  But I'm no fool and I keep records and detailed notes.  And the rest of this section of the site will be copies of those as I put them online.

When AJ was kidnapped we built the site on a daily basis.  We've been a little busy this time, and I really should have started this site straight away.  Needless to say, there is a story here to be told and it will be told in my usual detailed and intricate style.

What will please some of you most are the RealAudio files of conversations with Integral Energy and the Audio version of the EWON meeting on 19 February (also on Video, but our Video systems aren't working still.)

So watch for the calendar and Chronology of events to come.  You'll be shocked to see what I found out about some Integral Staff and more shocked about how the Customer Services staff, treat their customers.

When we eventually win our damages against Integral, we'll be installing 6KVA of Solar Power to the premises and Integral can pay us for power.  We really don't want to ever risk this kind of thing again.  With Deregulation of the power industry next year - I'd recommend everyone be afraid.  Be VERY afraid.