I have just gone through the *worst* tech support nightmare of my life. In the spirit of Halloween, I present to you...THE TECHNICAL SUPPORT DEPARTMENT FROM HELL!!!!
WARNING : May be too intense for some readers. Parental guidance is suggested. Some names have been changed to protect me from Google.com.
The system I am describing below actually consists of two machines. One is the voice mail machine we commonly call "the Blue Thing™," and the other is a standard PC running Windows NT. Every night, the Blue Thing uses FTP to transfer any changes that have been made to voice mail to the PC. And that's where the trouble begins...
2:00pm Friday : The IBX programmer reports that PC-Talk, a program that lets people check their voicemail from their computer, has had its name/extension database wiped out. I check it, and sure enough, it's only showing extensions and not the names assigned to those extensions.
2:10pm : I reboot the PC. Same problem.
2:20pm : I run the Names Download applet. It tells me there is an FTP Write Error, so I run it again. And again. Same problem.
2:30pm : Knowing full well that The Blue Thing Company always blames "IIS misconfigurations," I double-check our IIS setup against BTC's specs. Just as I suspected, everything was perfect. The PC checks out OK, so I inform our IBX programmer that since I don't know a damn thing about the Blue Thing (hey, I'm an admin, not a telecom guy) we'll need to bring in BTC support.
3:00pm : BTC wants to know if anything was done to the PC before it stopped working. Since I hadn't even touched it in three weeks, I tell them "no," carefully avoiding any mention of the hotfixes I applied to it three weeks before.
3:30pm : After going around in circles with BTC, our now-frustrated IBX programmer grabs me and demands to know the last thing I did to the PC. I tell her about the hotfixes and ask her "please, do NOT tell BTC that. They are immediately going to tell you that the hotfixes broke it, which is B.S. because it's been working *fine* with the hotfixes for three weeks now."
3:45pm : Arguing from her office. I start having a very, very bad feeling about all this...
3:50pm : I'm asked to go up into our switch room and talk with BTC support. The BTC technician has me check IIS settings, and then says "okay, well, Sandy tells me you installed some hotfix or something. Can you uninstall it?"
I tell him no, our security department wouldn't like that very much, and for that matter neither would I. Is he sure that it's a problem with the hotfix, and not something else?
4:00pm : He's sure it's the hotfixes. He wants me to stay on the line and uninstall them one by one, until the system starts working again. I tell him "no", because a. I applied a hotfix bundle that consisted of several dozen hotfixes so we'd be on the phone all week, b. the thing worked just dandy for 3 weeks after they were applied, and c. removing them one at a time would most likely render the PC unbootable.
I ask him if there's something else we could check.
4:10pm : He's still insisting I remove the hotfixes, and telling me "just don't tell your security department." I tell him I wouldn't have to tell them anything, they'd figure it out as soon as my machine started attacking them with Nimda and Code Red.
4:30pm : He has me reading off the name of every single file on the PC-Talk install CD, because he can't find the program he thinks will do something or other that I don't understand. I don't think he does, either, but I patiently read off a couple hundred filenames like "7ffy397.oc_".
5:15pm : I decide that my first gut feeling was correct. The guy is a moron. I tell him "look, it's 55 degrees in this switch room, my skin is cracking from the dry air, and I can't really see because my contacts have welded to my eyeballs. I have to get off the phone now. Send someone out Monday." He schedules a call for Monday, says something nasty like "this will probably never get fixed because you can't stick things out" and hangs up.
9:00am Monday : I show up at the office. Sandy tells me that BTC technicians logged into the Blue Thing over the weekend and verified that the problem was on the PC - "they're getting an FTP write error," she says.
9:10am : I ftp into the PC using the BTC machine's username and password, and upload 100 megs in 15 seconds without a problem. I tell Sandy that the FTP service is working fine.
10:00am : I remove IIS and reinstall it from CD, sending it back to the days of Service Pack 3. I go back to my cubicle expecting the security department to have my head on a stick by lunchtime.
3:00pm : We get BTC on the phone again. They still insist it's an IIS problem - "we're getting an FTP Write Error. So it has to be your PC."
When I explain that IIS has been uninstalled and reinstalled, they tell me it must be one of the other hotfixes and to pull them all off.
3:10pm : BTC support is now telling Sandy that I am a bad system administrator because I install hotfixes without knowing what they do and that I'm not a team player because I won't work with them.
3:20pm : My boss is now involved, and wanting to know why I don't just remove the hotfixes. I tell him the same thing I told BTC, and add that I'm not real sure the BTC techs have a clue about what they're doing. He seems to be satisified with that.
3:30pm : I tell Sandy to get a BTC technician on-site ASAP, because their phone support tech keeps having me verify that IIS is working, and then making up some random thing to check. Furthermore, I am completely sick of dealing with them, and if this turns out to have *anything* to do with IIS or the hotfixes, The Boss can fire me and I won't say a word.
3:40pm : A BTC technician is scheduled to arrive at 7am Tuesday. Sandy tells me to show up at my normal time and try to avoid the BTC people.
8:00am Tuesday morning : Sandy calls me at home and asks if it is at all possible to remove the hotfixes. I tell her it's not a hotfix problem, but to go ahead and reinstall Service Pack 6a.
8:10am : My boss calls wanting to know why there is absolutely nothing in my logbook about the PC for the past three weeks. I tell him that it's because I haven't *done* anything to it in the past three weeks.
8:50am : I arrive at work. The BTC technician is insisting that it's a problem with IIS because she's getting an FTP Write error.
9:10am : I disable IIS entirely and replace it with FTP Serv-U so I can watch every step of the process. I start the names download, watch the Blue Thing log in, starts transmitting data at 10 bytes a second, and then - lo and behold - THE FUCKING BLUE THING CLOSES THE SOCKET!!!!! THE BLUE THING DOES THIS, NOT MY PC!!! AND FURTHERMORE, ALTHOUGH THEY HAD ME CHECK ANONYMOUS FTP ACCESS 18 ZILLION TIMES, THEIR MACHINE NEVER EVER LOGS IN ANONYMOUSLY!!!
9:20am : I point these things out to the BTC technician. She insists that FTP Serv-U is lying and it doesn't matter anyway because it's not IIS.
9:40am : Our department head is now involved, and the BTC technician is explaining to her that I am throughly incompetant. I completely ignore everyone around me and start pinging the Blue Thing and discover - surprise, surprise - it's dropping packets. The BTC technician insists it's because IIS isn't running. I point out that since the Blue Thing runs a proprietary OS, whether or not it's dropping packets has NOTHING to do with IIS. I ask her to check the network interface on the Blue Thing. She tells my boss and the department head that I don't know anything and that because she's getting an FTP Write error it's obviously all my fault but I just can't admit it and she's never met anyone so incompetent.
9:50am : I grab my friend from the networking department and have him bring in a traffic analyzer, on the basis that if the network interface on the Blue Thing is going bad we'll be able to see that and at least prove to my boss that something is wrong with it. The BTC technician is pissed because I won't let her do her job. My boss is wisely letting me stomp about and take care of business without any interference.
10:00am : The traffic analyzer is up. Something is jabbering at 10Mbps, and the Blue Thing is one of the very few machines left that still uses 10baseT. "I'm surprised you could get a ping through this mess at all," my friend says. I start grinning in anticipation of being proved right.
10:10am : It's not the Blue Thing. Something else is jabbering, and the broadcasts are going all over the place - we have a bona fide broadcast storm. The BTC technician is still mumbling that I don't know what I'm doing and I shouldn't have applied the hotfixes. My friend and I give each other the look that means "everything we're saying right now is flying over this dumbasses' head."
10:20am : As we're tracking down the source, we find a loop in the network. We remove the cable that makes the loops, and the storm stops.
10:25am : PC-Talk successfully downloads the names directory and returns to operation. The BTC technician is now throughly pissed off at me. I just smile at her. She stomps out of the office and leaves for her next job. Everyone in the office is glad.
10:40am : I reapply all the hotfixes, reboot the system, and verify that it's still working properly. I then stop IIS and put FTP Serv-U back on, and retest. The system STILL works properly. I disable anonymous access and retest - it STILL WORKS!!!!
3:00pm : My report on the incident is complete. I point out that it never even occurred to BTC support to even LOOK for network problems, that at NO point did they ever suspect that something could be wrong at the Blue Thing, and that most of the things they had me check (such as anonymous FTP access) didn't even APPLY to their software. I mention that the next time something happens to PC-Talk, calling BTC should be considered the absolute last resort. I remember to thank my friend by name.
This message has been edited by damien_s_lucifer on October 16, 2001 at 09:23 PM
