A post on a board reminded me of one of my "favorite" stories about AOL and Sprint. I used to program AOL's call centers, and deal with Sprint quite frequently during that time. I'd say my assessment of them were mixed.
Because we were one of their biggest accounts at the time, Sprint had AOL liaisons. These people were in a team of about 4-5, consisting of a master account manager and his minions. One of them would usually be technically proficient, but it became clear that although the guy knew his stuff, Sprint did their best to keep him quiet.
First story: Keith Jenkins and his Rough Riders.In 1997, AOL went down for about 16 hours. Back in those days, AOL actually shut down from about 4am-6am a few days a week for maintenance. So no AOL for YOU! This had to stop when we went international, of course, because that was late morning in the UK and peak evening for Asia. But going international was still on the drawing table back then, so the 2 hour outage was a great chance to fix whatever ailed the system. On this one morning, however, something went terribly wrong.
Horribly wrong.
The story that was told to me was they went from Old System A to New System B. New System B launched and crashed horribly. No matter, we have a contingency plan, which was restore Old System A. But Old System A also crashed horribly. It would seem that when New System B went up, it made permanent changes that Old System A couldn't deal with, so what happened was the entire system because a freakish hybrid mutant that lay in a puddle of melting DNA with warped teeth and grayed-out eye sockets, hoarsely whispering... "Kill... me..."
By 7am, a huge amount of people using to dialing in to get their daily news got some errors; I forget what they said, but apparently it wasn't descriptive enough to prevent people from calling our call centers. The phone queue because outrageous by 9am, and when I got in at 3pm that day, they had anyone with a phone answering the huge wave of angry people. I had stopped taking tech calls for a while, but they told me I still had a headset, sign in, log on, and take calls. Man, people were angry! One woman was crying hysterically her husband was on a Navy ship and e-mail was the only contact they had.
But my misery was short lived. I noticed that some calls had no voice, and other calls dropped unexpectedly after a minute or so. Finally, the whole phone system toppled over like a dinghy filled with 100 Cuban refugees. We were asked to log off, and "Oh well."
By the time I started being a telecom programmer, that event was several months in the past. But it was still our number one beef with Sprint, and one of my first jobs was to sit in on the conference calls with Sprint about what happened. They blamed us, and I had to gather stats to show that we had far more agents waiting for calls than calls coming in, which was usually the other way around. After a few bad conference calls where nothing got done, a man named Keith Jenkins got involved.
Keith was, at the time, head of AOL's Member Services. He was an older guy, and you either loved him or hated him. He was like our Teddy Roosevelt, and he had a team of rough riders around him at all times. Keith and I had a "bonding moment" when I was a callback specialist for a new "Your AOL account has been hacked"
queue we were testing out, and him, my boss, and I had to go through a call where it was obvious their underage son was downloading pr0n. So he knew me already and kind of liked me. Keith said, "I am surrounded by bullshit," and demanded a face-to-face meeting with our Sprint liaisons because he felt nobody was giving him straight answers.
So, in AOL style, we rented the finest meeting room in our Herndon office, had it catered (AOL usually had meetings catered), and all of our Sprint people were asked to be there. Okay, they were told be there or else. AOL has a LOT of clout at the time, especially with Sprint. The meeting started with Sprint giving is their interpretation of the events, peppered with complete superfluous garbage that pissed Kieth off right away. This is kind of how the conversation went:
Sprint: So, when we formed back in the 1980s, we had a vision--
Keith: Skip it. Why did we lose calls to our call center?
Sprint: ... um, so in 1995, we forged a contract with... um, AOL. You know, you used to be Quantum Link, and we had --
Keith: Why did our calls go down?
Sprint: That's a very, um, good question, Mr. Jenkins. We have compiled some data that should be of interest--
Keith: Why did your system fail? Were was the point of failure?
Sprint: A system like ours is very hard to see at a microscopic level--
Keith: Why won't you answer the question?
Sprint: It's not a yes or no question, Mr. Jenkins--
Keith: Are you saying you don't know?
Sprint: No, no... see, we have... our network is comprised of [this goes on for a few minutes]
Keith: Let me stop you there, because we're again off track. Where is our interface between you and us? Grig?
Me: Um, well, it goes from several T1's and as far as I can tell, it goes through a MUX and a DMS 250. That's where we stop having control, and Sprint begins.
Keith: What's a DMS 250?
Me: A major phone switch. Sprint owns it.
Keith: Was it working when the call centers went down?
Me: It stopped sending data.
Keith: How many calls per second can the switch take?
Me: Our contract states 45 calls per second.
Sprint: Now hang on.
Keith: I gave him that data, leave him alone. Grig, how many calls were coming in before the call centers went down?
Me: 8 calls per second.
Keith: It never went over 8 calls per second?
Me: No. Usually it's 25 cps, but shortly after 1pm, the calls went from 30cps to--
Keith: I though you said we could take up to 45cps.
Me: It never got that high.
Sprint: You were rejecting calls when the queue was over 5 minutes! That's not our fault!
Keith: Grig, where does the decision to drop after 5 minutes get made?
Grig: On our end. We have to actually *get* the call first to determine whether to drop it or not.
Sprint: [seethe]
Keith: How many calls are we allowed per second?
Sprint: Well, that's compl--
Keith: How many... calls are we allowed... per second?
Sprint: Um... 12?
Keith: That seems far less than 45. How many calls, according to YOUR data, did you allow?
Sprint: [sigh] 8.
Keith: Did anyone ask to lower this limit?
Sprint: YOU WERE CLOGGING THE SWITCH!!
Keith: Isn't it a dedicated switch?
Sprint: Yes. I mean, no. It's COMPLICATED!
Keith: It seems simple to me. So far, it seems like you are not only restricting calls we're entitled to receive, but we're not on a dedicated switch.
Sprint: WHAT DO YOU WANT FROM US???
Keith: Pfuh... what you promised?
That Sprint guy was "retired" from our contract after this incident. I recall him hitting the table, nearly in tears when he screamed that line. Very unprofessional. Later Keith (and my boss Lou and his boss Rob) explained to me that Sprint had their back to the wall because they were in clear violation of contract, and we could sue.
Second Story: Sprints Complete Lack of Customer TactSo we got a new guy to be our main liaison, I'll call Roy. He was far more suited for the stress of the job, but was a little clueless when it came to hosting meetings. After several meetings where we went over call volume and network choke points, they invited the entire programming department to a meeting discussing the upgrade we decided upon based on the events now over a year in the past. Yes, it took that long. We had also gone from 6 call centers to 13 because of the CompuServe merger, plus a few new ones we were outsourcing. Our volume had increased substantially with 10 million members to almost 25 million at that point.
So me, my boss Lou, and two other programmers Liz and Mel were there with me. The meeting started at 9am, and was very boring and droned on and on. It was one of those typical meetings where we kept getting sidetracked on minor points, but at least no one was yelling or confrontational in any way. Just hardware and programming geeks geeking at one another. By noon, however, Lou asked, "Um... not to interrupt, but do we break for lunch or what?"
Roy looked startled, like he had forgotten something. "Lunch. Yes. Yes, we have lunch! Let me go... find ... it." And he left the room quickly. After about 15 minutes, he came back and said lunch was delayed. "About 30 minutes or less," he promised.
By 1pm, Lou asked again, politely (on the weekends, he was a pastor of a church in DC, great guy), "I hate to bring this up again, but I haven't eaten since 7 this morning. I am a little hungry."
Roy, again, looked like he'd been woken up very suddenly. "Yes! let me... check. It should be here by... now?" He quickly left. A few minutes later, he came back with several boxes of Domino's Pizza and a few 2 liter bottles of coke. Some had already been opened and slices were missing. Roy apologized, stating that the staff thought the pizza was for them when it arrived. He asked for some of his staff to let us have the pizza first, and they could have what was left over.
Now, keep in mind, we used to cater from the Marriott to Sprint when they visited. Maybe it's snobbish of us, but when we got some boxes of take out pizza, that was a little underwhelming. We exchanged raised eyebrows across the table. Liz was a little more than put off. See, she was Muslim. She couldn't have pork. And the pizzas were all pepperoni. She didn't say anything, of course, she just declined politely while her decades of religious upbringing were making her quite ill.
Lou was not so calm about it. "Liz can't eat this, it has pepperoni."
"Is she allergic?" Roy asked.
"Not exactly. It's her religion."
Roy looked concerned and sincere, but didn't quite get it. "Well, Liz, can't you eat around the pepperoni?"
Liz shook her head. "No."
"No," Lou repeated, a little sternly. "She can't drink the cola, either, it has caffeine."
"Is that a Jewish thing, too?" Roy asked.
Liz shook her head. "No, I am Muslim. I just can't have caffeine because it gives me headaches. It's fine, it's okay, I'll eat a big dinner or something."
"Muslims can't eat pork?" Roy asked. He had never heard of this, and I think he was trying to sound interested and accepting, but it was coming off so tactless.
"No. They cannot," Lou said. "Liz, you want to leave the room or something? You don't look well."
Liz insisted she was fine, despite the revulsion she was trying to hide.
"I can't drink this cola either," Mel said with a chuckle. "But that's because we don't have cups."
"CUPS!" Roy said. "Let me get some."
Some of the Sprint folks had already left to eat at their desks. Mel, Lou, and I ate pizza, while Liz tried hard not to look put out. The pizza was already cold and greasy. That's when we quickly realized there were no napkins. I had some Kleenex in my backpack I handed around. It felt like some crash meeting in a warehouse more than a proper conference. Roy eventually came back with... assorted coffee mugs. I could tell they were the kind from the communal office kitchen because they all had worn logos and coffee stains. I chose a "clean" one and was dismayed when I poured Coke in mine, an oily film covered the foam that smelled suspiciously like ballpoint pen ink. Roy had probably used someone pen holder. Lou asked if his mug was previously on display in the awards cabinet we saw when we came in. Roy said it wasn't, and Lou commented with a jocular manner he enjoyed the gold trim.
When a lot of the Sprint techs didn't come back, we decided to end the meeting early, having gotten the information we needed for the most part. Lou immediately said, "Meet me at TGI Fridays." And AOL paid for our late lunch/dinner while we all guffawed at the meeting we lust went through.
"I checked the awards cabinet when we left," Lou said, "...and sure enough, the mug was missing."
Posted in:
aol
,
work
![[info]](http://p-stat.livejournal.com/img/userinfo.gif){kind=small}
stodgycat asked me what was on that old backup CD I found in a CD Drive that has been sitting in a box. Well, here's the contents.![[info]](http://stat.livejournal.com/img/userinfo.gif){kind=small}
