Schwab's website went down twice after two 'denial of service' attacks -- so what was up?

The San Francisco-based broker showed that even it's not immune to web fritz; RIAs take it in stride though not without administering a healthy dose of schadenfreude

Wednesday 4.24.13 by Brooke Southall and Lisa Shidler

Brooke’s Note: We wrote this article yesterday, but then the web traffic to it started soaring again today. So I checked Schwab’s website and, once again, it wouldn’t come up. So I e-mailed Schwab spokesman Greg Gable, who responded by affirming that indeed his company had suffered a second attack on its systems. “We’re having intermittent access issues to our website due to a denial of service attack similar to yesterday which we’re actively addressing. We’ve asked clients who are affected to please try to log in again or if there matter is urgent, to please call.” What is clear from what our sources say is that denial-of-service attacks are dashedly difficult to counter. See: Walt Bettinger apologizes. Our expert source in this article said it’s like you having breakfast with a friend and having 20,000 people talking at you at the same time. Making out what your breakfast partner is saying is … very difficult. A detailed explanation is spelled out at the end of this article.

The Charles Schwab & Co. website(s) went down toward the end of the trading day Tuesday and right before Apple announced its earnings — and the site had just come back up as of publication of this article. The mega-site absorbed a second attack Wednesday as mentioned in the note above, and the site has been down intermittently again.

The San Francisco-based online broker’s ability to do business through the Internet — either with consumers or through their advisors — crashed at about 3:45 p.m. EDT (according to the company) and company spokesman Greg Gable confirmed the trouble. See: Why Chuck Schwab is fine with boosted taxes — and even Dodd Frank — and believes RIAs should be, too. Schwab has about 7,100 RIAs who trade through its custody unit with nearly $800 billion of assets.

“I can confirm that we are experiencing technical issues with and our mobile applications, and are working on a resolution. We’re asking clients who have urgent needs to please reach us by phone while we resolve the underlying issue.”

In a follow-up e-mail, Gable said that the problem concerned a web-tier issue and not a back-end issue, so trades and other functions can still be accomplished by calling the company.

This was posted in Schwab’s press room this afternoon:

“Shortly before the stock market closed today, we experienced an exceptionally high volume of website traffic which we believe was related to a denial-of-service attack. At all times, phone access to Schwab service professionals (800-435-4000) was available, although for a brief time immediately before market close call volumes were high. Web access was largely restored in approximately one hour and 40 minutes. We deeply apologize to our valued clients for the inconvenience.”

The term denial of service indicates a deliberate attempt to take a network offline.

Bad timing

The website problems happened at the worst time for advisor Heather Locus, a partner with Balasa Dinverno Foltz LLC, based in Itasca, Ill. She was meeting with a client at 2 p.m. CST and he had a question on his statement regarding the success of one of Schwab’s index funds. See: 10 reasons why Schwab’s move into ETFs may be an even bigger deal than it appears.

Heather Locus: Talk about bad timing.
Heather Locus: Talk about bad timing.

“I went to the website to get a copy of the statement to see what he was referring to and couldn’t access it. I think it is the first time in 17 years I wanted to access a statement during a client meeting. Talk about bad timing,” she says. “Another team member accessed the statement at 2:45 p.m. and brought it in while the client was still here.”

Then, Locus was able to answer all of the questions about the client’s fund, the Schwab 1000 Index.

“While it did cause some delays for our back office, it was later in the afternoon after we had done most of our trades,” Locus says. “So, it was an inconvenience but not catastrophic. We use APX for our CRM and portfolio management software and had downloaded this morning, so it didn’t affect a large majority of our work.” See: RIABiz takes a peek under the hood of Advent Portfolio Exchange.

One financial IT expert had this to say: “I can tell you that Schwab is getting slammed right now. I wouldn’t be surprised if there are extended delays and hold times, and Schwab has activated its “all hands on deck” mode, where even lower-level executives are being asked to man the phones.”

The expert continued: “Schwab loses a lot of money when these events happen. If their back-end system is down as well, it’s not a good day. To be fair to Schwab, it’s been a pretty long time since they had an event like this. Nobody is perfect, and outages are an unfortunate part of being a web provider. Shit happens sometimes.”

This was what could be found of Schwab's website in the wake of the attack.
This was what could be found
of Schwab’s website in the wake
of the attack.

Generic 'oops’

The chat room, of the website Is it down right now has dozens of comments about the Schwab site being down including colorful remarks referencing long wait times on the phones. For instance: “Very uncool to not even have a recording on their phone center. Permanent “hold” is not a good thing for a brokerage!”

(RIABiz has had its fair share of downtime — luckily mostly at night.)

One criticism leveled at Schwab surrounding the crash is that punching up produces nothing except a generic “oops!” from the browser.

That prompted this tweet from CFP John Friedman.

“Very surprised that Schwab does not have a separate status domain to update customers when its site is down. $10k a year to do that.”

Something major

Jon Yankee: Usually, our people trade in the morning.
Jon Yankee: Usually, our people trade
in the morning.

The IT expert source commented: “They’re either experiencing a DDoS (distributed denial of service attack), have suffered some major kind of outage at a co-location provider, or have lost a key piece of their web infrastructure [its DNS services, routing, or something critical]. Typically you’d fail over to your disaster recovery site, but since they haven’t been able to that probably means something major has happened.”

Schwab’s former chief information officer Bradley Peterson went to Nasdaq OMX in January.

Calling for backup

Surprisingly, there’s been no issues with the Schwab site being down at RIA Fox Joss & Yankee based in Reston, Va., says Jon Yankee. He wasn’t aware of the problem on Tuesday until he received an e-mail from RIABiz.

“We had no impact,” he says. “The people who traded with Schwab did it before 1 p.m. EST, and we’re not market-timers and we’re not big enough to have a trading desk. Usually, our people trade in the morning.”

About 70% of the firm’s assets are under custody at Schwab.

Yankee says that even if the firm did need to trade when the site was down, it would have reached out manually to an institutional service team at Schwab.

“We’d just call the team at Schwab that serves us, and they’d do the trades for us manually. They do have a backup in place if the website is down.”

'Not a good day’

One RIA, who asked not to be identified, wrote in an e-mail that the site’s having been down can have both real and psychological effects.

“It certainly has a few implications, though: 1) Reliability for Schwab for advisors 2) Reflects badly on advisors who directed their clients to Schwab for custody 3) Makes everyone more nervous about web-based systems, especially when it’s on the day of the Twitter flash crash.

Advisors at Hewins Financial Advisors LLC also say they experienced minimal impact.

“While the site disruption was inconvenient, having a great service team at Schwab to help during the disruption helped keep things on track for our firm and our clients. We had all of our trades in prior to the site disruption, so no issues.”

The IT expert source adds that it’s not exactly a bowl of cherries for Schwab either.

“Schwab loses a lot of money when these events happen. If their back-end system is down as well, it’s not a good day.”

Schwab (SCHW) shares finished up 1.04%, or 17 cents, for the day.

What next?

Experts say that the attack could have been perpetrated by almost anyone.

The IT expert says: “There’s a hacktivist group called Izz ad-Din al-Qassam Cyber Fighters that’s been making a lot of noise recently and have launched major DDoS attacks against a lot of financial services companies, including Citigroup, BofA, Wells Fargo, and others. Their beef is an offensive [to Islam] video that’s on YouTube. They want it taken down. Could be related to that, as they’ve basically said that 'no financial company is safe.’”

Raising defenses against this kind of attack is difficult, he adds.

“DDoS attacks are very difficult to protect against. Basically what’s going on is that a whole army of zombie computers (which could numbered in the tens of thousands) are programmed to basically 'attack’ Schwab by requesting a lot of data from their web servers at the same time. Called 'botnets,’ these computers overwhelm the web host by sending multiple requests for information.

“There are some strategies to deal with DDoS, a host of them require you to reroute malicious requests into what’s called a 'black hole IP’ and get them away from your primary infrastructure. The problem is that, at first, it’s really difficult to identify what’s real versus what’s malicious. Usually your colo (co-location) facility handles this. There are also commercial firms such as Cloudflare that claim to be able to help you respond and defend against DDoS attacks quickly. Not sure if Schwab uses something like this or not. At the end of the day, though, these are really hard to defend against and require rapid triage and action. I can guarantee you that Schwab is working with their colo to update their policies and procedures to be able to respond and mitigate these attacks much more quickly in the future.”

Final Brooke’s Note: I contacted our expert after today’s attack to ask him whether Schwab was at the mercy of its attackers. He reponded by saying that the company was, in effect, developing immunity to this particular threat. He said: Schwab will continue to take steps to mitigate this, and as the attacks continue it will be less and less “at the mercy” of whoever is behind the DDoS. I noticed earlier that even while the site was down and/or slow, Schwab Advisor Center was purring along just fine. So Schwab has clearly been working with its colo to route the malicious traffic away from its infrastructure and is having some success. As I said, though, these events are incredibly difficult to defend against and respond to quickly. Sadly, a DDoS takes a bit of time to respond to. With each attack, though, Schwab becomes less and less vulnerable as more data are collected, allowing it to accurately reroute the malicious traffic away from to the “black hole.”

Heather Locus | Bradley Peterson | Jon Yankee

Schwab | Balasa Dinverno Foltz | Fox Joss & Yankee | Hewins Financial Advisors