Archive for the ‘Behind’ Category
- In: Admin | Behind | Clickry | network | news | Rogue | San Francisco | story | The
- Leave a Comment
Last Sunday, Terry Childs, a network administrator employed by the City of San Francisco, was arrested and taken into custody, charged with four counts of computer tampering. He remains in jail, held on US$5 million bail. News reports have depicted a rogue admin taking a network hostage for reasons unknown, but new information from a source close to the situation presents a different picture.
In posts to my blog, I postulated about what might have occurred. Based on the small amount of public information, I guessed that the situation revolved around the network itself, not the data or the servers. A quote from a city official that Cisco was getting involved seemed to back that up, so I assumed that Childs must have locked down the routers and switches that form the FiberWAN network, and nobody but Childs knew the logins. If this were true, then regaining control over those network components would cause some service disruption, but would hardly constitute the “millions of dollars in damages” that city representatives feared, according to news reports.
Apparently, I wasn’t far off the mark. In response to one of by blog posts, a source with direct knowledge of the City of San Francisco’s IT infrastructure and of Childs himself offered to tell me everything he knew about the situation, under condition that he remain anonymous. I agreed, and within an hour, a long e-mail arrived in my in box, painting a very detailed picture of the events. Based on this information, the case of Terry Childs appears to be much more — and much less — than previously reported.
A Man and His Network
It seems that Terry Childs is a very intelligent man. According to my source, Childs holds a Cisco Certified Internetwork Expert certification, the highest level of certification offered by Cisco. He has worked in the city’s IT department for five years, and during that time has become simply indispensible.
Although Childs was not the head architect for the city’s FiberWAN network, he is the one, and only one, that built the network, and was tasked with handling most of the implementation, including the acquisition, configuration, and installation of all the routers and switches that comprise the network. According to my source’s e-mail, his purview extended only to the network and had nothing to do with servers, databases, or applications:
“Terry’s area of responsibility was purely network. As far as I know (which admittedly is not very far), he did not work on servers, except maybe VoIP servers, AAA servers, and similar things directly related to the administration of the network. My suspicion is that you are right about how he was “monitoring e-mail”; it was probably via a sniffer, IPS, or possibly a spam-filtering/antivirus appliance. But that’s just conjecture on my part.”
Like many network administrators who work in the rarified air of enterprise network architecture and administration, Childs apparently trusted no one but himself with the details of the network, including routing configuration and login information. Again, from the source’s e-mail:
“The routing configuration of the FiberWAN is extremely complex. Probably more so than it ought to be; I sometimes got the feeling that, in order to maintain more centralized control over the routing structure, [Childs] bent some of the rules of MPLS networks and caused problems for himself in terms of maintaining the routing.
“Because the system was so complex (and also because he didn’t involve any of the other network engineers in his unit), Terry was the only person who fully understood the FiberWAN configuration. Therefore, to prevent inadvertent disruption of this admittedly critical network, he locked everyone else out. I know most of the networking equipment … does use centralized AAA, but I get the impression he may have configured the FiberWAN equipment for local authentication only.”
Childs’ attitude toward other administrators is by no means unusual in the IT industry. This is generally due to the fact that admins who are tasked with constructing and maintaining networks of this size and scope care for them like children, and eventually come to believe that no one else could have the knowledge and skills to touch the delicate configurations that form the heart of the network.
Sole Administrator
A key point made in the e-mail is that Childs’ managers and co-workers all knew that he was the only person with administrative access to the network. In fact, it was apparently known and accepted in many levels of the San Francisco IT department. Again, quoting from the e-mail:
“This is where it gets tricky for the prosecution, IMO, because the localized authentication, with Terry as sole administrator, has been in place for months, if not years. His coworkers knew it (my coworkers and I were told many times by Terry’s coworkers, “If your request has anything to do with the FiberWAN, it’ll have to wait for Terry. He’s the only one with access to those routers”). His managers knew it.
Other network engineers for the other departments of the City knew it. And everyone more or less accepted it.
No one wanted the thing to come crashing down because some other network admin put a static route in there and caused a black hole; on the other hand, some of us did ask ourselves, “What if Terry gets hit by a truck?” If a configuration is known and accepted, is that “tampering”?”
My source appears to believe that Childs’ motivation was the antithesis of tampering, and that Childs did everything possible to maintain the integrity of the network, perhaps to a fault:
“He’s very controlling of his networks — especially the FiberWAN. In an MPLS setup, you have “provider edge” (PE) routers and “customer edge” (CE) routers. He controlled both PE and CE, even though our department was the customer; we were only allowed to connect our routers to his CE routers, so we had to extend our routing tables into his equipment and vice versa, rather than tunneling our routing through the MPLS system.”
Valleywag - valleywag.wordpress.com 
Recent Comments