Apr. 15, 2004 Issue of CIO Magazine
Project Management | AT&T Wireless
AT&T Wireless Self-Destructs
The story of a botched CRM upgrade that cost the telco thousands of new customers and an estimated $100 million in lost revenue. Hard lessons learned.
BY CHRISTOPHER KOCH
Reader ROI
How an upgrade meant to improve
customer service crippled it instead
Why having an alternative plan is
essential when you're up against a specific deadline
Why outsourcing in the middle of an
implementation can be disastrous
JACK LEE MARKED Nov. 24, 2003, on his calendar
because that was the day he would finally be able to change his cell
phone carrier without losing his phone number-thanks to a Federal
Communications Commission ruling. But Lee, president of Tangara
Technologies, a company that develops software for forms, decided to
wait a day before switching to AT&T Wireless to let the chaos of
"number porting" die down a little. Little did he know that the chaos
was just beginning.
Lee ordered his new phone on Nov. 25. When he went to the AT&T Wireless website to check the status of his order a day later, he was greeted with a message: "We could not find a porting request for this number in the system. Please contact Customer Care." It was the beginning of a two-month odyssey in which Lee estimates he made 15 to 20 calls to AT&T Wireless, sent nearly as many e-mails and spent 60 hours on the phone dealing with customer service representatives or waiting on hold-with the line often going dead when AT&T Wireless's customer service lines became overloaded.
After being routed all over the company, Lee finally discovered what was going on. A major CRM system had crashed during an upgrade, and customer service representatives could not set up or access new accounts. The system breakdowns, which continued through February 2004, swamped other AT&T systems, gridlocked customer service phone banks and sent furious customers scurrying to other providers.
The breakdown couldn't have come at a worse time for AT&T Wireless. It deprived the telco of thousands of potential new customers and cost the company an estimated $100 million in lost revenue. But that wasn't all. The failure so damaged AT&T Wireless's reputation that many analysts believe it hastened its sale to Cingular in February for $41 billion, or $15 per share, which was just under half the value of AT&T Wireless's shares when it went public in April 2000. While an AT&T Wireless spokesman says the company would have been sold regardless of the fiasco because "it was the right thing to do," the crash and resulting confusion could not have helped AT&T cut a good deal. "The system problems made AT&T look like a wounded provider, and the sharks smelled blood and circled," says Roger Entner, a wireless and mobile services analyst at The Yankee Group, a research company.
AT&T Wireless's mistakes offer valuable lessons for CIOs. For one, it's unwise to freight major system upgrades with external complications. AT&T Wireless's CRM upgrade was hamstrung from almost the very beginning by rumors of outsourcing deals and future layoffs. These rumors generated pervasive morale problems that hurt the productivity of project staff.
Second, it should be understood that complex projects require flexible deadlines. AT&T Wireless undertook a difficult upgrade that affected roughly 15 systems just before it was faced with an immovable deadline-the federally mandated Nov. 24 number portability date.
Finally, it always pays to have a plan B. Without one, AT&T Wireless was forced to move forward even as it became apparent that its upgrade would not be completed in time.
Playing Catch-Up with the Competition
When AT&T Wireless began its Siebel CRM system upgrade in 2003, it was a company that had slipped from unquestioned market leader to middle of the pack. Its overall market share had slid from an industry-leading 25 percent at the end of 2001 to 17 percent in 2003, third behind Verizon and Cingular.
Worse, AT&T Wireless was playing catch-up on its most important
technology asset, its phone network. One of the older wireless
companies, AT&T Wireless made an early bet on a technology called
TDMA that could not handle data transfer over cell phones-the next big
thing for business customers. Even before AT&T Wireless was spun
out from its parent AT&T in 2001, it had begun a furious buildout
of an expensive new network-global system for mobile communications, or
GSM-that could not only handle data but had the added advantage of
global compatibility with overseas providers. Only one other major
carrier, Cingular, was saddled with the challenge of building out a new
GSM network while still servicing the old one. The other carriers had
all chosen network technologies that could handle data transfers.
GSM was a great opportunity for AT&T Wireless, but it was also a huge CRM challenge. The company had to convince its old customers to move off TDMA, which worked as well as most other carriers' networks for voice calls, and onto GSM, which had poorer voice quality, according to Morgan Stanley. It also had to convince new customers that GSM was the wave of the future, that they would soon be shipping data over their phones instead of their laptops. But customers didn't buy the pitch. By 2003, AT&T Wireless's percentage of customers on GSM was hovering at 15 percent, according to analysts, while Cingular had 35 percent of its customers moved over (thanks, in part, to its acquisition of a cellular provider with an existing GSM network). And things weren't improving. In the third quarter of 2003, less than half of AT&T Wireless's new customers were choosing GSM, while Cingular was signing up 75 percent, according to Morgan Stanley.
Everyone at AT&T Wireless agreed that the company would keep its existing customers and add more new ones if its customer representatives could handle more calls and get customers up and running faster. Customer service representatives needed about 20 minutes, on average, to work through five or six screens that fetched information from about 15 legacy systems, say former employees. Slow access to customer information records was just one of the reasons why AT&T Wireless possessed the second-highest cost per subscriber (behind Sprint PCS) of the top six national carriers in 2003, according to financial services company UBS.
Why AT&T Needed to Upgrade
AT&T Wireless installed Siebel's CRM system in 2001 to be the front end of its customer service process. The back end, however was a complex mishmash of systems, say former employees. Telco billing systems, for example, were stuffed full of different rate plans and arcane metering processes. Systems that tracked calls and set up new phone numbers (provisioning) communicated with hundreds of thousands of different telephone switches around the country and the world. To work for AT&T Wireless, Siebel's version 6 had to be highly customized. Though the software came with integration tools, consultants usually resorted to writing point-to-point scripts to hook the systems together. Policing the overall integration in a scenario like this is difficult at best. Indeed, a former AT&T Wireless employee who worked on the project recalls the test system crashing and remaining down for six weeks during the summer of 2002 when AT&T Wireless began preparing Siebel version 6 to deal with number portability. And when Siebel 6 was finally up and running, it still couldn't handle all the information that customer service representatives needed.
The next release of Siebel, version 7, was much more powerful. Siebel developed an industry-specific version of the new software that had many more features and could capture more telco-specific information. And new Web capabilities meant that AT&T Wireless could potentially reduce the number of customer service screens to one by building a Web portal that put all the different systems and information sources together in one place. "That was the point of the upgrade," says a former employee. "They wanted to get everything on one screen so they could sign up customers faster."
Marc Siegel, a spokesman for AT&T Wireless, agrees:"As we needed to handle more transactions, more customers, we needed a more robust system. So we upgraded. [The upgrade] was going to make it easier for our representatives to get access to information and give them a fuller array of information on the customer."
In the spring of 2003, the company decided to upgrade to version 7.5 for the roughly 3 million customers on the GSM and general packet radio service (the data portion of the new network). TDMA customers would continue to be serviced through AT&T Wireless's legacy Axys CRM system until they could be moved over to GSM and the new Siebel system. The project was called Odyssey.
The Upgrade Begins, Badly
Though the upgrade promised gains in speed and simplicity, getting there was neither speedy nor simple. AT&T Wireless's IT staff had to rip apart old links from the different legacy systems to the client/server-based Siebel 6 system and rewrite them to work over the Web. The back-end systems-integration work was so complex that it wasn't unusual to see teams of 20 or more people assigned to write connections for a single system, says a former employee. Coordination between the teams-the responsibility of the lead integrator on the project, Deloitte and Touche-quickly got out of hand. (Deloitte and Touche did not respond to repeated requests for an interview.)
"Everything was siloed among the different groups, and we all worked independently of each other," says a project team member. Teams would work on a revision to their piece of Odyssey, for example, only to find that when they finished testing, code had changed elsewhere in the system, rendering the testing meaningless. "In other projects I've worked on, the project managers would freeze code while the teams did revisions to their pieces so you could test everything against the same code base," he says. "I didn't hear of that happening on this project."
Meanwhile, rumors of layoffs and offshore outsourcing began swirling around Odyssey. "[The rumors] slowed things down," says a former employee. "When stuff like that happens, people start looking for other work. I know I was looking for other work when I should have been testing."
Meet the New Boss
On April 10, 2003, Mike Benson, a 15-year AT&T veteran, sent a memo to employees announcing that he was leaving after three years as CIO. AT&T Wireless's Siegel says the 48-year-old Benson "decided to retire." Christopher Corrado, the head of the security solutions practice for Wipro, an Indian offshore outsourcing company, was named executive vice president and CIO. Before joining AT&T Wireless, Corrado had been CTO at two divisions of Merrill Lynch, where he presided over the offshore outsourcing of portions of Merrill Lynch's IT. Employees speculated that it was just a matter of time before offshoring began at AT&T Wireless.
"Corrado was the hatchet man-everyone knew it," says a former employee.
"We'd see our people going into these long meetings with people from
Indian companies. We'd see whiteboards that had questions like, 'What
opportunity do we have to offshore/outsource?'"
Former employees say morale wasn't helped by Corrado's first presentation to the IT group, in which they say he proclaimed, "Come in every day and expect to be fired." Intended to inspire the troops to greater effort, the talk backfired, says another former employee. "We all came away saying, 'Who is this arrogant jerk?'" Corrado could not be reached for comment.
The Deadline Looms
As November approached, AT&T Wireless was one of the last industry holdouts opposing the Federal Communications Commission's new rule on number portability, along with Cingular and Alltel. The industry had succeeded in delaying the change since 1998 on legal technicalities, and AT&T Wireless was counting on another postponement, say industry sources. But on Halloween, the U.S. Court of Appeals for the District of Columbia Circuit rejected the appeal to move back the FCC's Nov. 24 deadline. An AT&T Wireless spokesman told Bloomberg News that the company was ready for porting.
It wasn't.
Things were going so badly with the Siebel CRM upgrade that there was talk on the project of trying to roll back to the old Siebel 6 system before the porting date. "Two weeks before launch there was rollback talk," a former employee says. But project managers hadn't preserved enough of the old system to make it feasible. There was no plan B. Project members estimated that the team needed two or three more months of bug fixing and testing to get the system running reliably.
And the Siebel system wasn't the only thing AT&T Wireless needed to get working before the deadline. Wireless number porting required new systems that had to be integrated with the different carriers' CRM systems. Five of the six top providers outsourced the administration of the number changes to TSI Communications and used software built by Telcordia. But in June 2002 (before the other carriers had announced they were going with TSI), AT&T Wireless chose its longtime software and administration provider, NeuStar. Ultimately, this would create serious interoperability problems for AT&T Wireless.
AT&T Wireless defends its decision by saying that NeuStar was the most experienced provider in the market, having worked on landline portability in the early '80s. It also says NeuStar's software offering was more complete than TSI's at the time AT&T made its decision.
No Treat, Just Trick
On Halloween evening, the project team tried to bring up the new Siebel system for the first time. It crashed. "It went down and stayed down," says a former employee. "They couldn't get it working again." Project staffers were brought in to work on the system for three straight days to try to rescue it, with no luck, say former employees. The system was unstable.
Though observers could not trace specific causes for the crash, a few factors played a role. As the go-live date neared, former employees say that Deloitte and Touche project managers relaxed testing requirements for various pieces of the system. Rather than freeze the code for the system once problems began, teams continued to add new pieces to the project in an attempt to get it all working. But the new pieces simply added more errors to an already bug-ridden system, which complicated the process of finding root problems and fixing them.
"[Integration was] far more technical and far more difficult to test in advance and also fix problems once they appeared," acknowledged Michael Keith, president of AT&T Wireless Mobility Operations, in a conference call on the company's fourth quarter results.
Compounding these difficulties was the hit employee morale had taken on
Nov. 16, when AT&T Wireless President and CEO John Zeglis confirmed
the layoff rumors. Zeglis told analysts in a third quarter conference
call that the company would lay off 1,900 workers. He did not say where
the cuts would come or when. But IT managers started telling employees
that layoffs would begin in February 2004, according to former
employees.
Some AT&T Wireless IT employees had gotten hold of a report prepared by Tata Consulting Services of India outlining plans for offshoring pieces of AT&T Wireless's IT infrastructure. The report, obtained by CIO, also lays out plans for offshoring a portion of the support and maintenance for the new Siebel system. "You had to wonder why you were working so hard if the whole thing was going to go away anyway," says a former project employee.
On Nov. 19, The Wall Street Journal ran a story on planned layoffs and outsourcing at AT&T Wireless, and CIO Corrado responded in a memo to the staff: "We are in the awkward position of having reports in the press about a contract with HP before we have communicated with the employees who will be affected. We have just today signed a letter of intent with HP to deliver the services provided by the following teams: Desktop Services; Retail Field Services; Business Office WAN/Telecom; MSI Packaging; Service Desk-password reset and desktop calls." But as for bigger questions, namely that support and maintenance for the Siebel system would be outsourced to offshore provider Wipro and Tata, Corrado did not address the rumors directly. Instead, he concluded: "We currently outsource work throughout the company, including work within both Customer Services and IT. We will continue evaluating the best mix of internal and external resources as we work to achieve best-in-class margins."
In fact, AT&T Wireless was planning to move overseas more than 3,000 positions in its computer operations and customer service.
The Second Crash
The Odyssey team was still struggling to get the Siebel system up and keep it up when, on Nov. 24, customers began trying to port their numbers between carriers. That's when the software for handling number portability crashed.
TSI and NeuStar had arrived at "differing interpretations" of the industry's data standard for exchanging and verifying porting requests, known as Wireless Intercarrier Communications Interface Specifications (WICIS). The Yankee Group's Entner says that TSI modified its version of the WICIS standard without telling NeuStar. "They made changes in fields and reporting that didn't impact all their five customers but no longer made it possible to communicate with [NeuStar]," says Entner. TSI's Michael O'Brien, vice president of marketing, denies that changes were made without telling NeuStar but acknowledges that the two vendors had different interpretations of WICIS. (NeuStar declined to be interviewed for this story.) AT&T Wireless says that TSI crashes prior to the deadline prevented NeuStar from being able to test the process adequately, and on Nov. 24, the electronic flow of porting requests between NeuStar and TSI broke down right away.
The industry had estimated that automated ports would require 2.5 hours to complete, so the porting messages were programmed with time and date limits. If the other carrier's system didn't respond within the allotted time, the port that failed was supposed to drop into an electronic bin to be handled manually.
NeuStar's software could not respond to TSI's systems before time ran
out. And the error messages weren't all getting through either. When
AT&T Wireless's Porting Administration Group tried to track and
resolve errors using a workflow management tool, it crashed or froze.
That meant AT&T Wireless customer service representatives were in
the dark when their phones lit up. Combined with their continued
problems accessing Siebel, there was little they could do to help their
customers. In that first week of porting, more than half of the
complaints filed with the FCC singled out AT&T Wireless, according
to The Associated Press. The FCC sent a letter to AT&T Wireless on
Dec. 4, asking for an explanation, triggering a torrent of bad
publicity.
"There were 50,000 [AT&T] customers per week that didn't have service because of this," says Phil Cusick, a telecom industry analyst with Bear Stearns. And many of them, like independent consultant Ian Drake, didn't stick around for a fix. After three weeks of trying to track his newly purchased AT&T Wireless phone, Drake gave up and signed on with Verizon.
A Company Fire sale
As word of AT&T Wireless's difficulties spread, customers began deserting at a rapid clip. The problems continued through December, according to AT&T Wireless President and CEO Zeglis. "It took most of December to expand the systems capacity so that the care reps could have as much access as they needed in order to handle customer calls at the same time the salespeople were processing new orders," he said in a conference call with analysts on Jan. 22. "Frankly, our customer service went pretty far south."
For independent cell phone retailers, which account for about 60 percent of new sales in the industry, there was no choice. "The indirect sellers pushed other brands when they heard that AT&T was having problems," says Greg Teets, telecom analyst for A.G. Edwards. AT&T Wireless was forced to bump up commissions to independents and lower prices to try to stem the tide. It didn't work. After adding 229,000 customers in the third quarter of 2003, fifth out of the top six providers, AT&T Wireless added just 128,000 in the fourth quarter, last among the top providers and less than one-tenth the number of industry leader Verizon, which added 1.5 million new subscribers.
By January, AT&T Wireless was in the advanced stages of plans to move overseas more than 3,000 positions in its computer operations and customer service, according to The Wall Street Journal. (It backed off from the plan only after agreeing to be purchased.) Some employees became part of AT&T Wireless's "buddy program," in which consultants from Indian outsourcing companies Tata Consultancy Services and Wipro were assigned to AT&T Wireless employees to learn their jobs.
In February, 220 AT&T Wireless IT employees were told that they would be released from the company in March. According to one of the terminated employees, another round of 250 layoffs is planned for June and the same number for September. "It's tough," an employee said in February. The Indian workers "basically follow people around all day and pepper them with questions." Other former employees say some staffers resisted helping the consultants. "People would make project decisions when the Indians weren't around," a former employee says.
On Feb. 17, Cingular announced that it was buying AT&T Wireless.
That same day, Deborah Sawyer's voice-mail box began filling up with messages. Sawyer, a partner for executive search firm Morgan Howard Worldwide, says she received inquiries from 12 different AT&T Wireless IT executives looking to move on.
But for their company, the odyssey was over.
Executive Editor Christopher Koch can be reached at ckoch@cio.com.
PHOTO-ILLUSTRATION OF BROKEN CELL PHONE BY JOHN WEBER
Lee ordered his new phone on Nov. 25. When he went to the AT&T Wireless website to check the status of his order a day later, he was greeted with a message: "We could not find a porting request for this number in the system. Please contact Customer Care." It was the beginning of a two-month odyssey in which Lee estimates he made 15 to 20 calls to AT&T Wireless, sent nearly as many e-mails and spent 60 hours on the phone dealing with customer service representatives or waiting on hold-with the line often going dead when AT&T Wireless's customer service lines became overloaded.
After being routed all over the company, Lee finally discovered what was going on. A major CRM system had crashed during an upgrade, and customer service representatives could not set up or access new accounts. The system breakdowns, which continued through February 2004, swamped other AT&T systems, gridlocked customer service phone banks and sent furious customers scurrying to other providers.
The breakdown couldn't have come at a worse time for AT&T Wireless. It deprived the telco of thousands of potential new customers and cost the company an estimated $100 million in lost revenue. But that wasn't all. The failure so damaged AT&T Wireless's reputation that many analysts believe it hastened its sale to Cingular in February for $41 billion, or $15 per share, which was just under half the value of AT&T Wireless's shares when it went public in April 2000. While an AT&T Wireless spokesman says the company would have been sold regardless of the fiasco because "it was the right thing to do," the crash and resulting confusion could not have helped AT&T cut a good deal. "The system problems made AT&T look like a wounded provider, and the sharks smelled blood and circled," says Roger Entner, a wireless and mobile services analyst at The Yankee Group, a research company.
AT&T Wireless's mistakes offer valuable lessons for CIOs. For one, it's unwise to freight major system upgrades with external complications. AT&T Wireless's CRM upgrade was hamstrung from almost the very beginning by rumors of outsourcing deals and future layoffs. These rumors generated pervasive morale problems that hurt the productivity of project staff.
Second, it should be understood that complex projects require flexible deadlines. AT&T Wireless undertook a difficult upgrade that affected roughly 15 systems just before it was faced with an immovable deadline-the federally mandated Nov. 24 number portability date.
Finally, it always pays to have a plan B. Without one, AT&T Wireless was forced to move forward even as it became apparent that its upgrade would not be completed in time.
Playing Catch-Up with the Competition
When AT&T Wireless began its Siebel CRM system upgrade in 2003, it was a company that had slipped from unquestioned market leader to middle of the pack. Its overall market share had slid from an industry-leading 25 percent at the end of 2001 to 17 percent in 2003, third behind Verizon and Cingular.
| AT&T Wireless vs. Verizon Wireless: Losing Ground |
Between 2001 and 2003,
AT&T Wireless lost significant ground to Verizon Wireless (now the
industry leader). That's what its CRM upgrade was intended to fix. It
didn't. 
Read More  |
GSM was a great opportunity for AT&T Wireless, but it was also a huge CRM challenge. The company had to convince its old customers to move off TDMA, which worked as well as most other carriers' networks for voice calls, and onto GSM, which had poorer voice quality, according to Morgan Stanley. It also had to convince new customers that GSM was the wave of the future, that they would soon be shipping data over their phones instead of their laptops. But customers didn't buy the pitch. By 2003, AT&T Wireless's percentage of customers on GSM was hovering at 15 percent, according to analysts, while Cingular had 35 percent of its customers moved over (thanks, in part, to its acquisition of a cellular provider with an existing GSM network). And things weren't improving. In the third quarter of 2003, less than half of AT&T Wireless's new customers were choosing GSM, while Cingular was signing up 75 percent, according to Morgan Stanley.
Everyone at AT&T Wireless agreed that the company would keep its existing customers and add more new ones if its customer representatives could handle more calls and get customers up and running faster. Customer service representatives needed about 20 minutes, on average, to work through five or six screens that fetched information from about 15 legacy systems, say former employees. Slow access to customer information records was just one of the reasons why AT&T Wireless possessed the second-highest cost per subscriber (behind Sprint PCS) of the top six national carriers in 2003, according to financial services company UBS.
Why AT&T Needed to Upgrade
AT&T Wireless installed Siebel's CRM system in 2001 to be the front end of its customer service process. The back end, however was a complex mishmash of systems, say former employees. Telco billing systems, for example, were stuffed full of different rate plans and arcane metering processes. Systems that tracked calls and set up new phone numbers (provisioning) communicated with hundreds of thousands of different telephone switches around the country and the world. To work for AT&T Wireless, Siebel's version 6 had to be highly customized. Though the software came with integration tools, consultants usually resorted to writing point-to-point scripts to hook the systems together. Policing the overall integration in a scenario like this is difficult at best. Indeed, a former AT&T Wireless employee who worked on the project recalls the test system crashing and remaining down for six weeks during the summer of 2002 when AT&T Wireless began preparing Siebel version 6 to deal with number portability. And when Siebel 6 was finally up and running, it still couldn't handle all the information that customer service representatives needed.
The next release of Siebel, version 7, was much more powerful. Siebel developed an industry-specific version of the new software that had many more features and could capture more telco-specific information. And new Web capabilities meant that AT&T Wireless could potentially reduce the number of customer service screens to one by building a Web portal that put all the different systems and information sources together in one place. "That was the point of the upgrade," says a former employee. "They wanted to get everything on one screen so they could sign up customers faster."
Marc Siegel, a spokesman for AT&T Wireless, agrees:"As we needed to handle more transactions, more customers, we needed a more robust system. So we upgraded. [The upgrade] was going to make it easier for our representatives to get access to information and give them a fuller array of information on the customer."
In the spring of 2003, the company decided to upgrade to version 7.5 for the roughly 3 million customers on the GSM and general packet radio service (the data portion of the new network). TDMA customers would continue to be serviced through AT&T Wireless's legacy Axys CRM system until they could be moved over to GSM and the new Siebel system. The project was called Odyssey.
The Upgrade Begins, Badly
Though the upgrade promised gains in speed and simplicity, getting there was neither speedy nor simple. AT&T Wireless's IT staff had to rip apart old links from the different legacy systems to the client/server-based Siebel 6 system and rewrite them to work over the Web. The back-end systems-integration work was so complex that it wasn't unusual to see teams of 20 or more people assigned to write connections for a single system, says a former employee. Coordination between the teams-the responsibility of the lead integrator on the project, Deloitte and Touche-quickly got out of hand. (Deloitte and Touche did not respond to repeated requests for an interview.)
"Everything was siloed among the different groups, and we all worked independently of each other," says a project team member. Teams would work on a revision to their piece of Odyssey, for example, only to find that when they finished testing, code had changed elsewhere in the system, rendering the testing meaningless. "In other projects I've worked on, the project managers would freeze code while the teams did revisions to their pieces so you could test everything against the same code base," he says. "I didn't hear of that happening on this project."
Meanwhile, rumors of layoffs and offshore outsourcing began swirling around Odyssey. "[The rumors] slowed things down," says a former employee. "When stuff like that happens, people start looking for other work. I know I was looking for other work when I should have been testing."
Meet the New Boss
On April 10, 2003, Mike Benson, a 15-year AT&T veteran, sent a memo to employees announcing that he was leaving after three years as CIO. AT&T Wireless's Siegel says the 48-year-old Benson "decided to retire." Christopher Corrado, the head of the security solutions practice for Wipro, an Indian offshore outsourcing company, was named executive vice president and CIO. Before joining AT&T Wireless, Corrado had been CTO at two divisions of Merrill Lynch, where he presided over the offshore outsourcing of portions of Merrill Lynch's IT. Employees speculated that it was just a matter of time before offshoring began at AT&T Wireless.
CIO Christopher Corrado never directly addressed rumors that AT&T
Wireless planned to outsource much of the support and maintenance for
its CRM system.
|
Former employees say morale wasn't helped by Corrado's first presentation to the IT group, in which they say he proclaimed, "Come in every day and expect to be fired." Intended to inspire the troops to greater effort, the talk backfired, says another former employee. "We all came away saying, 'Who is this arrogant jerk?'" Corrado could not be reached for comment.
The Deadline Looms
As November approached, AT&T Wireless was one of the last industry holdouts opposing the Federal Communications Commission's new rule on number portability, along with Cingular and Alltel. The industry had succeeded in delaying the change since 1998 on legal technicalities, and AT&T Wireless was counting on another postponement, say industry sources. But on Halloween, the U.S. Court of Appeals for the District of Columbia Circuit rejected the appeal to move back the FCC's Nov. 24 deadline. An AT&T Wireless spokesman told Bloomberg News that the company was ready for porting.
It wasn't.
Things were going so badly with the Siebel CRM upgrade that there was talk on the project of trying to roll back to the old Siebel 6 system before the porting date. "Two weeks before launch there was rollback talk," a former employee says. But project managers hadn't preserved enough of the old system to make it feasible. There was no plan B. Project members estimated that the team needed two or three more months of bug fixing and testing to get the system running reliably.
And the Siebel system wasn't the only thing AT&T Wireless needed to get working before the deadline. Wireless number porting required new systems that had to be integrated with the different carriers' CRM systems. Five of the six top providers outsourced the administration of the number changes to TSI Communications and used software built by Telcordia. But in June 2002 (before the other carriers had announced they were going with TSI), AT&T Wireless chose its longtime software and administration provider, NeuStar. Ultimately, this would create serious interoperability problems for AT&T Wireless.
AT&T Wireless defends its decision by saying that NeuStar was the most experienced provider in the market, having worked on landline portability in the early '80s. It also says NeuStar's software offering was more complete than TSI's at the time AT&T made its decision.
No Treat, Just Trick
On Halloween evening, the project team tried to bring up the new Siebel system for the first time. It crashed. "It went down and stayed down," says a former employee. "They couldn't get it working again." Project staffers were brought in to work on the system for three straight days to try to rescue it, with no luck, say former employees. The system was unstable.
Though observers could not trace specific causes for the crash, a few factors played a role. As the go-live date neared, former employees say that Deloitte and Touche project managers relaxed testing requirements for various pieces of the system. Rather than freeze the code for the system once problems began, teams continued to add new pieces to the project in an attempt to get it all working. But the new pieces simply added more errors to an already bug-ridden system, which complicated the process of finding root problems and fixing them.
"[Integration was] far more technical and far more difficult to test in advance and also fix problems once they appeared," acknowledged Michael Keith, president of AT&T Wireless Mobility Operations, in a conference call on the company's fourth quarter results.
While employees struggled to fix glitches in the CRM upgrade, CEO John
Zeglis confirmed the layoff rumors, saying AT&T Wireless would soon
lay off 1,900 workers.
|
Some AT&T Wireless IT employees had gotten hold of a report prepared by Tata Consulting Services of India outlining plans for offshoring pieces of AT&T Wireless's IT infrastructure. The report, obtained by CIO, also lays out plans for offshoring a portion of the support and maintenance for the new Siebel system. "You had to wonder why you were working so hard if the whole thing was going to go away anyway," says a former project employee.
On Nov. 19, The Wall Street Journal ran a story on planned layoffs and outsourcing at AT&T Wireless, and CIO Corrado responded in a memo to the staff: "We are in the awkward position of having reports in the press about a contract with HP before we have communicated with the employees who will be affected. We have just today signed a letter of intent with HP to deliver the services provided by the following teams: Desktop Services; Retail Field Services; Business Office WAN/Telecom; MSI Packaging; Service Desk-password reset and desktop calls." But as for bigger questions, namely that support and maintenance for the Siebel system would be outsourced to offshore provider Wipro and Tata, Corrado did not address the rumors directly. Instead, he concluded: "We currently outsource work throughout the company, including work within both Customer Services and IT. We will continue evaluating the best mix of internal and external resources as we work to achieve best-in-class margins."
In fact, AT&T Wireless was planning to move overseas more than 3,000 positions in its computer operations and customer service.
The Second Crash
The Odyssey team was still struggling to get the Siebel system up and keep it up when, on Nov. 24, customers began trying to port their numbers between carriers. That's when the software for handling number portability crashed.
TSI and NeuStar had arrived at "differing interpretations" of the industry's data standard for exchanging and verifying porting requests, known as Wireless Intercarrier Communications Interface Specifications (WICIS). The Yankee Group's Entner says that TSI modified its version of the WICIS standard without telling NeuStar. "They made changes in fields and reporting that didn't impact all their five customers but no longer made it possible to communicate with [NeuStar]," says Entner. TSI's Michael O'Brien, vice president of marketing, denies that changes were made without telling NeuStar but acknowledges that the two vendors had different interpretations of WICIS. (NeuStar declined to be interviewed for this story.) AT&T Wireless says that TSI crashes prior to the deadline prevented NeuStar from being able to test the process adequately, and on Nov. 24, the electronic flow of porting requests between NeuStar and TSI broke down right away.
The industry had estimated that automated ports would require 2.5 hours to complete, so the porting messages were programmed with time and date limits. If the other carrier's system didn't respond within the allotted time, the port that failed was supposed to drop into an electronic bin to be handled manually.
| Five Lessons from AT&T Wireless's Project Failure |
| It did a lot wrong.
Here's what you can do right.
Read More |
"There were 50,000 [AT&T] customers per week that didn't have service because of this," says Phil Cusick, a telecom industry analyst with Bear Stearns. And many of them, like independent consultant Ian Drake, didn't stick around for a fix. After three weeks of trying to track his newly purchased AT&T Wireless phone, Drake gave up and signed on with Verizon.
A Company Fire sale
As word of AT&T Wireless's difficulties spread, customers began deserting at a rapid clip. The problems continued through December, according to AT&T Wireless President and CEO Zeglis. "It took most of December to expand the systems capacity so that the care reps could have as much access as they needed in order to handle customer calls at the same time the salespeople were processing new orders," he said in a conference call with analysts on Jan. 22. "Frankly, our customer service went pretty far south."
For independent cell phone retailers, which account for about 60 percent of new sales in the industry, there was no choice. "The indirect sellers pushed other brands when they heard that AT&T was having problems," says Greg Teets, telecom analyst for A.G. Edwards. AT&T Wireless was forced to bump up commissions to independents and lower prices to try to stem the tide. It didn't work. After adding 229,000 customers in the third quarter of 2003, fifth out of the top six providers, AT&T Wireless added just 128,000 in the fourth quarter, last among the top providers and less than one-tenth the number of industry leader Verizon, which added 1.5 million new subscribers.
By January, AT&T Wireless was in the advanced stages of plans to move overseas more than 3,000 positions in its computer operations and customer service, according to The Wall Street Journal. (It backed off from the plan only after agreeing to be purchased.) Some employees became part of AT&T Wireless's "buddy program," in which consultants from Indian outsourcing companies Tata Consultancy Services and Wipro were assigned to AT&T Wireless employees to learn their jobs.
In February, 220 AT&T Wireless IT employees were told that they would be released from the company in March. According to one of the terminated employees, another round of 250 layoffs is planned for June and the same number for September. "It's tough," an employee said in February. The Indian workers "basically follow people around all day and pepper them with questions." Other former employees say some staffers resisted helping the consultants. "People would make project decisions when the Indians weren't around," a former employee says.
On Feb. 17, Cingular announced that it was buying AT&T Wireless.
That same day, Deborah Sawyer's voice-mail box began filling up with messages. Sawyer, a partner for executive search firm Morgan Howard Worldwide, says she received inquiries from 12 different AT&T Wireless IT executives looking to move on.
But for their company, the odyssey was over.
Executive Editor Christopher Koch can be reached at ckoch@cio.com.
PHOTO-ILLUSTRATION OF BROKEN CELL PHONE BY JOHN WEBER
