Air conditioning woes

[KA1ZGC]

From the "don't build it this way" department:

Woke up this morning to a barrage of emails from equipment overheating and shutting down. Called into work to find out what the hell was going on, only to discover something a little on the amazing side.

It turns out that the majority of our lab AC units do not have their condensors located on the roof, they are in fact coupled to one big evaporator fed by one big glycol loop to one big condensor on the roof.

Well, the glycol pipe burst this morning, taking out most of my lab facilities in the Framingham office, and flooding rooms on three floors with glycol.

Oh, joy.

Fortunately, a couple of the 10-ton units are not on this secondary glycol loop, so some stuff is still running, but otherwise this place is a mess. All I can do is wait for the HVAC guys to finish repairing and recharging the glycol loop so we can check pressures on the subordinate loops and start bringing stuff back up again.

We almost certainly lost some equipment, one of the last units to shut itself down due to ambient temps over 45 deg. C was only a foot off the floor!

This is what happens when you try and make data centers out of office buildings. I knew something like this would eventually happen, and it picked a day when potential customers are here.

Now, let's see... where did I put that resignation letter?

--Thom
Kelvin Accumulation One Zero Glycol Cooler

[W3SLK]

Thom said:

Well, the glycol pipe burst this morning, taking out most of my lab facilities in the Framingham office, and flooding rooms on three floors with glycol.

Your state department of environmental resources will love you for that! We let anything close to the amount of a pint of glycol loose, its a 5-star all hands emergency that starts first with contacting PA DEP. What sucks is that a farmer can lose a hydraulic hose dumping 100 gal. + of fluid on the road and nothing is even considered. But damned if we do anything resembling that, we are nailed to the proverbial cross.

[wb1aij]

Your state department of environmental resources will love you for that! We let anything close to the amount of a pint of glycol loose, its a 5-star all hands emergency that starts first with contacting PA DEP.

At Pratt & Whitney we get water from the MDC water company, the same water we drink in our homes, but before we can dump it back into the MDC sewers we have to remove the fluoride that the MDC put in.

[Jim, W5JO]

At Pratt & Whitney we get water from the MDC water company, the same water we drink in our homes, but before we can dump it back into the MDC sewers we have to remove the fluoride that the MDC put in.

Are we stupid or what? 

[John Holotko]

Our data (operations) center is located underground, i.e. lower basement of a old Manhattan building. We have had our share of problems. Like one night a couple of months ago when a water pipe broke cascading water down onto racks of equipment and forcing the center to be systematically shut down to minimize damage. Fortunately if we go off line we are backed up via facilities elsewhere. Surprisingly the water damage was minimal and we mainly lost only a couple ofline conditioner/backup power supplies. We were back up in relatively short order. However, that water pipe has since been rerouted away from where our equipment is located as to avoid a repeat of the same near disaster.

Anytime you fill some location with sensitive electronics there is not telling what could happen. Even my home "data center"is not immune to problems.  I noticed that when the AC cycles on and off frequently the initial current draw causes my UPS supplied to go on battery momentarily. This is not much of a  problem but, if the AC cycles on and off frequently it can reduce backup battery capacity. Good data center design is quite tricky.

[KA1ZGC]

Our data (operations) center is located underground, i.e. lower basement of a old Manhattan building. We have had our share of problems. Like one night a couple of months ago when a water pipe broke cascading water down onto racks of equipment and forcing the center to be systematically shut down to minimize damage. Fortunately if we go off line we are backed up via facilities elsewhere.

We also have redundancy, at least in terms of production. We're leasing space in a "real" data center (they've got a raised floor, other than that it isn't much more real), and I installed a complete clone of our production infrastructure there. To be honest, the main reason I did so was because the inter-site link is a single T1 and the NFS traffic alone will kill the link, but it does have the added virtue of disaster recovery.

I go through the water drill pretty regularly, one of the (ceiling mounted) AC units in question has a way sucking air in through some service panel (or something) when too fine a filter is placed in the return (for those keeping score at home: the return is the warm air entering the unit). That mysterious "other intake" is above the suspended ceiling somehwere, where the air is guaranteed to be hot and damp. Pull that across the evaporator: instant iceberg.

Same thing happens when some douchebag leaves the damn door open for more than a few minutes. That's why I'm probably known as something along the lines of "door nazi".

The sad thing is that it took several doused systems before people realized that I had a point when I told them they shouldn't park racks under those particular AC units.

Anytime you fill some location with sensitive electronics there is not telling what could happen. Even my home "data center"is not immune to problems.  I noticed that when the AC cycles on and off frequently the initial current draw causes my UPS supplied to go on battery momentarily. This is not much of a  problem but, if the AC cycles on and off frequently it can reduce backup battery capacity. Good data center design is quite tricky.

I'd agree somewhat, but phrasing it the other way around states the case better: bad data center design is altogether too easy, and you can't make a good data center where one just wasn't meant to be.

If your interested, I can elaborate to you in a non-public kinda way. I've probaby said too much already.

Just curious, is your UPS an APC unit? The reason I ask is because I've never seen an APC UPS that didn't cycle to the batt for several seconds during initial powerup, 120V or 208V, in any building I've plugged one in. Those particular units just seem to do that by default, I always assumed that it was part of the power-up self-test.

Your state department of environmental resources will love you for that!

I won't tell them if you won't! 

As it stands now, we've recharged the secondary loop with water. It seems everybody and his dog within twenty miles of us had AC failures over the weekend and today (which doesn't suprise me at all), and our normal contractor had already used all their glycol on Sunday. It'll do for right now, water has better thermal absorbtion and conductivity than glycol, it's just more corrosive and freezes at 0 C.

I realize, reading back over what I wrote initially, that I may have given the impression that the glycol directly took out more than it did. Three racks took a bath, everything else just got really, really hot. I did have an awful lot of fun strategically powering racks up and down, moving fans about, and slip-sliding on films of glycol all day.

Thank God I'm off the meds and can drink again. I really needed that beer tonight!

--Thom
Killer Album One Zappa's Greatest Compositions

[John Holotko]

Just curious, is your UPS an APC unit? The reason I ask is because I've never seen an APC UPS that didn't cycle to the batt for several seconds during initial powerup, 120V or 208V, in any building I've plugged one in. Those particular units just seem to do that by default, I always assumed that it was part of the power-up self-test.

Yes, it is an APC unit.  It seems that all the newer APC power supplies perform that routine where they cycle to battery momentarily for a few seconds after  they are  first turned on. . And from what I read in the manual it is part of the initial self test. I also think I remember reading that it will perform a self test like that periodically (every week or so)  even when there are no power interruptions or problems.   Now I have a couple older APC units and they don't do the self test. However all the newer ones made within the past couple years  or so that I have seen seem perform that self  test.
 

[KA1ZGC]

Just curious, is your UPS an APC unit? The reason I ask is because I've never seen an APC UPS that didn't cycle to the batt for several seconds during initial powerup, 120V or 208V, in any building I've plugged one in. Those particular units just seem to do that by default, I always assumed that it was part of the power-up self-test.

Yes, it is an APC unit.  It seems that all the newer APC power supplies perform that routine where they cycle to battery momentarily for a few seconds after  they are  first turned on. . And from what I read in the manual it is part of the initial self test. I also think I remember reading that it will perform a self test like that periodically (every week or so)  even when there are no power interruptions or problems.

Well, I don't know what models you had, but we shipped our product with APC UPSes, 208VAC mostly, with a few 120VAC models for development systems. All rackmount jobbies. They all had ethernet ports and would at least throw SNMP traps when things got out of whack, which they do a decent job of detecting during those periodic self-tests. If it's running something mission-critical on it, it's a handy feature to have. Otherwise, it's just annoying for about 7 seconds.

APC makes a good UPS. As I found out today, however, they don't make two 3-ton ceiling-mount air conditioners alike. I never did find the reset button in the same place twice on four units bought in one purchase. 

Oh, well. It's working for now.

--Thom
Killer Aircraft One Zeppelin Goes Crash

[John Holotko]

Your state department of environmental resources will love you for that! We let anything close to the amount of a pint of glycol loose, its a 5-star all hands emergency that starts first with contacting PA DEP.

At Pratt & Whitney we get water from the MDC water company, the same water we drink in our homes, but before we can dump it back into the MDC sewers we have to remove the fluoride that the MDC put in.

Hah ! did it ever dawn on those mensa geniuses that the flouride is still there when everyone in town flushes the bowl or lets it go down the drain ?

[flintstone mop]

My big UPS in the basement is always on-line , none of that switch over stuff. it's running on the battery all the time. If the AC, quits the charger stops.  Once a week it goes to battery for about 15 minutes and will beep if the battery fails. The charger is turned off. It's a BEST ferro somethng or other. Before I retired my company was throwing these things out because of battery failure!!!! My unit is a 2KW back breaker.

Fred

[KA1ZGC]

Thank God I'm off the meds and can drink again. I really needed that beer tonight!

Another bad idea on my part. I wound up getting seriously ill about 3am last night off only a couple of beers. Guess the metronidazole wasn't totally out of my system after all. 

Your state department of environmental resources will love you for that! We let anything close to the amount of a pint of glycol loose, its a 5-star all hands emergency that starts first with contacting PA DEP.

At Pratt & Whitney we get water from the MDC water company, the same water we drink in our homes, but before we can dump it back into the MDC sewers we have to remove the fluoride that the MDC put in.

Hah ! did it ever dawn on those mensa geniuses that the flouride is still there when everyone in town flushes the bowl or lets it go down the drain ?

That reminds me, one of these days I've got to find out when K1DEU's birthday is so I can send him a bottle of flouride. 

My big UPS in the basement is always on-line , none of that switch over stuff. it's running on the battery all the time. If the AC, quits the charger stops. Once a week it goes to battery for about 15 minutes and will beep if the battery fails. The charger is turned off. It's a BEST ferro somethng or other. Before I retired my company was throwing these things out because of battery failure!!!! My unit is a 2KW back breaker.

The 208V APCs we have are pretty hefty, too. 208V 30A input, two 208V 20A outputs, one 208V 30A output, and two 120V 20A outputs.

These days I'm deploying them more as line conditioners than UPSes. Muscling these things around is how I screwed up my back last year. Now I leave it to the hourly workers. Back pain and tooth pain are two things I could use a lot less of.

--Thom
Kerolene Apparatus One Zooming Golf Cart

[k4kyv]

The window a/c unit in my ham shack just crapped out.  The compressor seems ok, but smoke poured out of the fan motor.  The problem is (1) getting the original motor out - the way the unit is constructed, it's much like trying to work on a car as far as removeability is concerned, (2) finding a replacement, since the unit is decades old and the model # didn't even show up on an appliance parts website I checked out, and (3) if I need to replace the unit (I have a working spare a/c in storage), it is mounted in a hole in the wall about 10 ft. above the floor.  How do I safely get the old one down and the new one up?

[KA1ZGC]

The window a/c unit in my ham shack just crapped out.  The compressor seems ok, but smoke poured out of the fan motor.  The problem is (1) getting the original motor out - the way the unit is constructed, it's much like trying to work on a car as far as removeability is concerned, (2) finding a replacement, since the unit is decades old and the model # didn't even show up on an appliance parts website I checked out, and (3) if I need to replace the unit (I have a working spare a/c in storage), it is mounted in a hole in the wall about 10 ft. above the floor.  How do I safely get the old one down and the new one up?

You might want to consider knocking an a-frame together out of some 2-byes or 4-byes with a block-and-tackle arrangement at its apex. That's too much wheight too high up to be safely done on a ladder. You and one other person of reasonable strength should be able to winch it down and winch the new one up.

Unless you've got one of those scissor-jack tables laying around, which would also do the trick.

--Thom
Kilimanjaro Africa One Zulu Goat Cheese

[Bill, KD0HG]

The window a/c unit in my ham shack just crapped out.  The compressor seems ok, but smoke poured out of the fan motor.  The problem is (1) getting the original motor out - the way the unit is constructed, it's much like trying to work on a car as far as removeability is concerned, (2) finding a replacement, since the unit is decades old and the model # didn't even show up on an appliance parts website I checked out, and (3) if I need to replace the unit (I have a working spare a/c in storage), it is mounted in a hole in the wall about 10 ft. above the floor.  How do I safely get the old one down and the new one up?

Don, check out the WW Grainger catalog. They've got all sorts of fractional horsepower motors.

Do you have any idea how the A/C unit was originally installed? Without knowing the layout there, maybe you could rig two tall ladders with planks in between to make a scaffold. That's how I worked on way high fixtures when we built our house.

..

[w1guh]

The window a/c unit in my ham shack just crapped out.  The compressor seems ok, but smoke poured out of the fan motor.  The problem is (1) getting the original motor out - the way the unit is constructed, it's much like trying to work on a car as far as removeability is concerned, (2) finding a replacement, since the unit is decades old and the model # didn't even show up on an appliance parts website I checked out, and (3) if I need to replace the unit (I have a working spare a/c in storage), it is mounted in a hole in the wall about 10 ft. above the floor.  How do I safely get the old one down and the new one up?

I replaced the motor in an A/C circa 1963 in 1996.  I first tried to match it up with something from Grainger and went up to one of their outlets.  The guy there couldn't help me match up the unit, so he sent me to an electric motor place in Ardsley, NY.  There the guy took my old motor, figured out its ratings, fitted it with the proper cap, did the wiring, and cut and finished the shaft so all I had to do was put it back in where the old one was.  If I remember correctly the delta in price between what he charged me and what I would have paid at Grainger for just a raw, unprepared motor was well worth it. 

Yea, working on an AC is nasty work...much like a car.  To get the motor out and back in took many, many cuss words.

And the glycol leak.  Funny, we had one of those in one of the labs at work around 2000-2001.  Is there something about those systems that's inherently unreliable, or are we talking about two incompentant installers?  The clown who supervised the installation where I was at (who no longer worked there) could easily have done a bad job.  The good news was that that lab was inactive at the time.

Finally, about AC's, the vintage '63 Welbuilt that I put a new motor in in '96 started blowing the circuit breaker the next year.  Grrr.  I suspected the capacitor for the compressor, but I was totally burnt out working on it so I bought a new unit.  That unit started blowing the breaker two years ago ( only seven years old).  It's a moot point now, I'm in a new place and my old building is a pile of rubble...but I've since learned that circuit breakers can and will get weak with age (probably especially if you use it as a remote control switch).  It's probable that neither A/C was bad...it was probably the 40 year old circuit breaker.

[KB2WIG]

       "How do I safely get the old one down and the new one up?"

Well..... disconect everything, tie a rope around it and your pick up truck axle mount, make sure you have some slack, and let her rip.... it'll pop out like a dead tooth..

Or, Disconect everything, spike some 2x4 or 2x8 below the unit and give her a good push

Or, see " A slight mishap" ,   WD8BIL has a sure fire method to reduce the objects weight..

[KA1ZGC]

And the glycol leak.  Funny, we had one of those in one of the labs at work around 2000-2001.  Is there something about those systems that's inherently unreliable, or are we talking about two incompentant installers?  The clown who supervised the installation where I was at (who no longer worked there) could easily have done a bad job.  The good news was that that lab was inactive at the time.

Well, I know the HVAC guys said the roof units were pretty cruddy, which I'm inclined to beleive, given that a room with 9 tons of cooling on that loop was getting hot from 7.5 tons of heat load for quite some time before the blowout.

Clog your condensor and you'll build up high head pressure. I wouldn't think that would be enough to blow out a 4" copper pipe, but I suppose anything is possible.

The glycol loop was already in place before we moved the company to this building, so I honestly couldn't speak to the quality of the install.

As Mike(y) pointed out earlier, we've got the state (oh, I'm sorry, the commonwealth) DEP guys crawling all over us now. Best guesstimate is well over 100 gallons of glycol spilled. It was a 4" line hundreds of feet long and burst at its lowest point, draining the entire loop.

I haven't heard the estimates on cleanup and repair, but I know how much we charge for our systems, and the 4 that took a glycol bath add up to just over $125,000 in lost parts and labor, and just over $3 million in lost revenue. In reality, the revenue is just delayed, we'll simply build 4 more systems to replace the ones that were destroyed.

The good news is that nobody got hurt (nor did anyone break out in a glycol-induced rash), and we managed to get things back online by the end of the day.

Still quite a mess to clean up. I'll have to see what the company policy is on hazmat pay.  

--Thom
Keep Away One Zorched Ground Coductor