Last year I celebrated 20 years of CCIE certification. No one threw me a party, there was no ticker tape parade, hell, if I didn’t rescue the email from my spam folder I wouldn’t have known that it even happened. So I celebrated content that I no longer have to pay $80 every 2 years to Cisco, and then I downloaded a CCIE Lifetime Emeritus logo to put on my LinkedIn page (fancy huh?). Getting that certification opened a ton of doors in my career, and I am indeed thankful. But now I no longer need to recertify, or pay dues, or do anything really, and for all that I am also thankful.

Being a network expert when there was insatiable demand for elite network skills was certainly fun. I got to work in a lot of cool places, jetted around the world in economy seating, made some great friends, and most importantly, I helped a lot of good people.

Networking has changed dramatically in the past 20 years, it makes me feel like a dinosaur to look at a new Segment Routing topology or try to understand the intricacies of the latest hot ASIC that all the cool kids are using. But the more it has changed the more it has stayed the same. We still use CLIs, syslog and SNMP. That should give everyone pause, the majority of networks are still managed with protocols and tools created in the 70’s.

— Disclaimer: One paragraph of marketing —

Recently I’ve been working on some technology that is designed to empower DevOps teams to build and operate data center and edge networks with their preferred tools, namely Kubernetes and the wide world of Cloud Native software. Will this exciting new approach to networking eliminate the need for CCIEs? Unlikely. But I don’t need to do anything at all to achieve that, it’s going to happen on its own.

— End Marketing —

This series of posts will briefly describe some things from my life on the CLI. First off, let me be clear, I’m going to be talking about network experts as “CCIEs”. If you’re a JNCIE or simply someone who has earned their own personal badge of honor by living the same sort of lifestyle, this post IS for you. I don’t like network-vendor-religion, but CCIE is the shortest title I can use to describe the cream of the crop of network experts.

Ignoring for a moment the wide variety of project related activities that fill most of the day (yawn), a CCIE basically does 3 things:

• Breaks the network
• Fixes the network
• Implements Moves, Adds, and Changes

(hopefully not all three every day!)

The Episode Where He Breaks the Network

I have definitely broken networks. If you’re a CCIE for any amount of time, you have too. It’s ok to nod right now, we’ve all done it. Sometimes it wasn’t our fault, sometimes it was. Over time, these painful experiences transform into lessons that we learn from, like a scar. Some people have a LOT of scars. Here are a few of mine.

One time I took an entire major news office off the air because I misconfigured a route-map to try to distribute traffic across two unequal bandwidth links. The business owners were cheap and bought a 5MB link and a “backup” 1MB link. At some point in time they decided they wanted to use the backup for a certain type of traffic so they asked me to policy route packets across the 1MB. In networking, every bit counts, and one IP range that doesn’t align with the bitmask can bring you a world of hurt. In my case it caused all of the traffic on the 5MB link to go towards the 1MB link. 5MB > 1MB obviously and the link was saturated in a second and voila, goodbye D.C. news office. Lesson: Don’t let management force you to put a major system at risk because they are cheap. Redundancy means EQUAL COST.

Another time I caused a pretty big hiccup on a large e-commerce website with my early attempts at automation. I was using PERL (yeah I’m that old) to connect to a Cisco 5k via telnet (SSH was not supported on the 5k) to gather some basic telemetry from a show command. If the script couldn’t connect it would simply retry again until the connection was established, because who needs connection throttling anyhow? The 5k was built before anyone cared about security, so if you hit the mgmt vty with 10 connection attempts in a row very quickly you caused the device to have a kernel panic. Joy. Naturally this campus access switch was connected to the core via L2, and the core was also connected to the front end of the e-commerce site. Spanning Tree can be a really annoying thing when it’s doing its job. There have been a lot of times when I have cursed Radia Perlman’s name while the network was down (Sorry Radia, I have no way of counting all the times that your invention magically saved my ass). Lesson: Never test something new, no matter how harmless it may seem, in production.

That last one is what some people refer to as a “Resume Generating Event”.

I’ve also borked some rather simple things, like unplugging a cable and putting it back. Sometimes if you’re working in a crowded wiring closet the cable goes into the next port to the right, which is of course in a different VLAN. And that cable wasn’t going to a user’s desktop, it just had to go to a Microsoft Exchange server, and here we go again, goodbye foreign news office. News organizations don’t need 24/7 email access right? Lesson: Don’t let anyone force you to take a personal risk because of their poor planning. Lesson 2: Don’t plug/unplug cables if you cannot view them easily.

If this post seems like a Freudian therapy session, well sorry about that but over 20 years you definitely get the opportunity to really screw things up. Hell, that was only 3 mistakes, imagine how long this post could theoretically be! Admittedly these mistakes happened in a different age, and some of these companies were actually respected for how loose (read: fast) they operated their networks.

What’s common about all of these events? Aside from me having to spend the rest of the day hiding in a one of the wiring closets, in each situation I was dealing with an environment created by someone else. At the CLI, or looking at a mess of cabling, It was like I was time traveling back to see the exact design decisions and thinking of someone years before me. I’m not blaming someone else, I certainly made the mistakes, but none of these networks were created by me. These are what network engineers refer to as snowflakes. Each one of them is individualized and unique, with a unique set of behaviors and problems. CCIEs hate other people’s snowflakes.

Buying a CCIE a Birthday Gift

But networking is all about order, organization, and determinism. Networking is a lot like Legos. Everyone in the world knows how to use Legos. They feel a certain way in your hand, and they snap together with a specific click that allows you to know if the connection is sound. Legos interconnect in very defined ways, but the connectivity options are so wide that you can stack them in an infinite number of configurations.

CCIEs don’t like picking up somebody else’s set of problems because there is never enough documentation to make them comfortable. But with Legos, you don’t need documentation. The bricks are well defined parts that interconnect in a very prescribed manner. And when you are building a big set, the manual shows you exactly how to assemble it to achieve the outcome advertised on the front of the box. So don’t buy the CCIE in your life a snowflake for their birthday.

Lots of networking companies sell Legos. Some still want you to use the CLI to assemble their Legos. Does that sound very Lego-ish to you? We (everyone in the networking industry) need to do a much better job delivering a more friendly and consumerized experience for network builders. We need to produce polished, perfectly fitting, functional building blocks so more people are able to assemble them, move them, change them, and interconnect them to the rest of our Lego City. We have to deliver better products, with more intelligence baked in, so end users have more deterministic outcomes.

And hopefully far fewer incidents of innocent engineers breaking the network.

P.S. Next time I’m going to talk about things I did that saved the day. Yippee