Dave Winer is inspiring me to write smaller posts on things that come to mind. I think it’s good for the soul.
One of the most telling things about changing jobs to a completely different career, from IT Operations to Marketing, is that there are some people who, while well paid, really get treated like shit. All of you IT Operations professionals out there, I am truly sorry for how your management and employers mistreat you. And for you individual contributors out there, I’m going to go one step further, and I’m going to sincerely apologize to your bosses, probably up to Director level, although some VPs probably get the shaft too. Your middle management’s life sucks, and, assuming they do right by you, you should be thankful they’re willing to work 24 hours a day 7 days a week with no on-call rotation.
IT Operations people’s lives are constantly getting disrupted. This is the case in many operational jobs, but none more so than in IT Operations. Sales Vice Presidents don’t get called every night a time or two to approve something or to be informed of something, and they certainly don’t work many weekends or miss vacations because some system decides out of the blue to go on the fritz.
Many times during my career people would tell me that there were other jobs out there that didn’t have the kind of time commitments, constant interruptions, and general downsides that IT Operations had. I always laughed them off, told them I was sure they were right, but now that I’m on the other side I have to say that life is truly better when your phone doesn’t ring every night with a problem. It’s like when you’re young and poor and you’re afraid to answer the phone because of some bill collector on the other end of the line. Once you get older and start paying your bills on time, the phone ringing no longer evokes a visceral fear reaction, and it’s the same after you change jobs. There is life after Operations, and while it isn’t for everyone, after 15 years of it, it was for me.
Three years with barely an update, now I’m at least cranking out content on a once a week basis. I only know of a couple of people who still read my rantings, but it feels good to express opinions after 4 years of attempting to claw my way to middle management by biting my tongue and playing nicely. What a perfect segue into my thoughts on culture.
If virtual ink were a tradable commodity, I’m sure there’d be a playboy trading magnate who made his entire family fortune from selling virtual ink to people who like to write about what makes one company’s culture so successful versus another. So I suppose I’m walking a well-trodden path, but after spending much time analyzing why culture can go from good to bad, there are a couple of things to do (or not do):
Hire Slow, but Make Hiring Easy
Choose your employees very carefully. Only hire the best. One bad apple will certainly spoil the bunch, but make it easy to bring people on. If it takes more than 2 weeks from when the candidate is identified, after the interviews have completed, from the day they could start, your HR processes are broken and you need to fix HR.
As I said earlier, a bad apple spoils the bunch. If you have a low performer, move them out quickly. Give them a month or two of severance, apologize for it not working out, and wish them the best with a good reference. Just because they’re not working out at your company does not mean they will add no value anywhere.
Accountability, Accountability, Accountability
Failure happens, even at the best companies. Adjust, move on, but hold people accountable. This doesn’t mean your first or second failure means you should exit, it simply means if you’re not being successful then it needs to be analyzed and corrected. Maybe you’re in the wrong seat on the bus. Nothing frustrates your workforce more than seeing someone still in the same position, still failing, a year after it was obvious to everyone they weren’t succeeding. Your personal relationships can be difficult to untangle from your business relationships, but the phrase “it’s not personal, it’s just business” exists for a reason. Everyone should be measured, graded, and held responsible for the areas for which they are accountable.
“What Gets Measured Gets Managed”
Everyone’s heard this quote. S.M.A.R.T. goals, etc. You would think it’s a bunch of management school bullshit that they teach to people to allow them to become HR Generalists. You would think that, until you’ve been somewhere it isn’t practiced. The fish out of water analogy is a very graphic way of describing an organization where the rank and file are flapping around without direction. Goals should be clear, everyone should be marching in lock step towards achieving them. Giving your teams freedom to work on their own is awesome, in moderation. An organization where everyone is marching to their own drum breeds frustration and malicious compliance.
Ok, so it’s not a word. It conveys the point though. Being wishy-washy, constant direction changing, and a lack of consistency will drive morale through the floor. Nothing will demoralize a group of people who have been working hard towards a goal than to see that goal shift, morph, move, etc, on a regular basis. This doesn’t mean you shouldn’t guide the ship, correct course, and be proactive about hitting a moving target. However, large scale about-faces will certainly destroy productivity.
Most of the points were written on what not to do, but these points are very binary. Each has two sides of the coin, the right way to function and the wrong way. Avoid the wrong way, the right way finds itself. Create a performance culture, foster a meritocracy, and smart and motivated people will find their way to you. An organization full of the best people has no option but to succeed.
Amy Hoy (@amyhoy) is an old friend of mine. We have spent untold amounts of hours on IRC together. We’ve just recently reconnected on Twitter, where yesterday she tweeted a link to “Don’t bite the shit sandwich” on her blog. I’m an avid follower of the startup scene, especially a number of the bullshit peddlers she warns against in her post. I’ve been actively involved in that scene as well, and while I don’t have a huge success story, I think the mistakes I’ve made lead me to some slightly different conclusions.
First of all, I’d get a lot more viewers here if I came out and said “Eric Ries is wrong, lean startups are bad,” or even “Amy Hoy doesn’t know what the hell she’s talking about,” but I won’t. The truth is in the middle. I agree with several of Amy’s conclusions. Startup methodology, like development methodology, exhibits a lot of characteristics of a religion. I believe a lot of great startup companies fail because of the idea that they need to give a product away for free to attract eyeballs, gain critical mass, create shareholder value and then exit. I was part of one of those.
I’ve started an Internet company and I’ve started more traditional small businesses. What I’ll say about the small business world is that it’s steeped in reality. No income coming through the door? Speculative business idea? Don’t look for help from your local bank. More money flowing out than flowing in? Hope you have a rich uncle to fund your “speculation.” Most of the country works this way, real businesses, exchanging dollars for good and services from each other.
If you want freedom from Corporate America (which I’m seriously considering at this point), you can follow Amy’s model (and Charlie’s too). I’d like to see a lot more like those two as well. However, it’s not really new. Plenty of people have been out selling themselves for years. Artists, writers, consultants, you name it, there’s plenty of small businesses built around a person.
The problem with her concept of not taking funding or building to sell is if you have an idea, a great idea, a world changing idea, or maybe even just an idea that’ll fill a great niche, you’re going to get pushed into a service based model. Maybe you have to do consulting on the side to start your business, which is a major distraction. If you don’t, your business is going to be pushed into compromise to make revenue early and be forced early on to attempt to extract value from your idea when it may not be the time to charge for it yet. You’re going to be forced into overhead for customer service, general administration, and key early stage development cycles are going to be consumed attempting to respond to customer demands when your vision for where the product should be going should be the right one.
There is no right answer. If you’re looking to create a product business and not a service business though, don’t discount financing your business with an early stage capital investment. Google doesn’t get built if Sergey and Larry turn their search engine into a product for a specific customer and lose their long term vision of indexing the world’s information. Facebook, recent controversy aside, does not get to be a household product if Zuckerberg spends his days trying to find a way to bring in revenue three months after Facebook’s founding. Nothing wrong with creating a brand around yourself and spending your days free from bullshit either, though.
I was listening to a video from Startup Lessons Learned, where a brief comment was made regarding customer centricity at IMVU, which reminded me of a nagging thought in my head in recent months. Customer Centricity is evil, in IT. That’s a bold statement, so let me explain. There are a lot of mechanisms of providing the customer what they need, and perhaps even want, without placing them at the center of your activities.
Being customer centric, as implemented most places I’ve seen, means an unwavering attitude internal to the organization that it will bend over backwards to provide the customer what they want. This leads to some unintended consequences, such as a continual focus on the functional, rather than the non-functional. For those who spend little time doing software project, program or requirements management, functional requirements are things like “I want the software to be able to calculate taxes and print it on a receipt” and non-functional requirements are things like “I want the software to always work.” Bending towards customer centricity and an attitude that the customer is always right, especially in the world of custom software, means the resources of the organization are constantly allocated to the functional while often largely ignoring the non-functional until it’s too late.
This attitude can and will, at a point in the future, leave you in a situation where the scale and complexity of the software the IT organization is producing will begin to collapse under it’s own weight. The nimbleness once felt, perhaps as recently as a few years back, will begin to become cumbersome, as the continuous iterations of “get it done, the customer needs it right away” lead to band-aids, patchwork engineering, and layer upon layer of complexity. Time for things like application rationalization activities, engineering and architecture improvements, and infrastructure improvements are pushed aside in order to make room in the capacity model for more functional requirements for the customers.
The solution? Simple IT. A culture of simplicity is required around every change to drive compromise between what the customer wants and needs and what IT can deliver and maintain. Constant reinforcement of a simplicity culture is required to drive every individual contributor to ask “is this additional complexity required?” It will drive from monolithic to modular, from n to n-1 layers of abstraction, from customization to off the shelf. No organization is perfect, so additional iterations of software should continually simplify and rationalize the changes that in hindsight complicated rather than simplified. The customer demands the non-functional without asking for it, and the customer gets what they’re asking for through simplicity.
What does the mean for the customer? The customer asks for 30 reports, they get 5, because it’s actually in their benefit. The customer wants business rules that force the agents to jump through hoops to try to upsell customers, which leads to ridiculous clicking and wasted agent time, IT pushes towards training, because the complexity hurts the users’ perception of the software and impacts their behavior negatively. The customer asks for a different pricing model for every market, IT refuses to support it. Will this lead to significant conflict and accusations that IT is not responsive to the business? Absolutely! Perhaps the company will go through three IT leadership teams before they realize their behavior is the problem, or perhaps they will never realize. Either way, IT has a duty to the business to draw a line between what is possible and what is right, and IT is in the unique position to provide the business with facts and data about how decisions driving complexity hurt not only IT, but hurt the business. Simplicity is, in reality, complicated to deliver; it’s the enemy of mediocrity; it’s the champion of forethought and design; and a culture centered around simplicity is the only way to deliver years of iterative software without being forced to spend significant amounts of money, time, and resources, redoing what customer centricity, and by extension, complexity, create.
A couple of things have happened recently which has made it to where I believe it’s time to start writing thoughts, publicly, again. First, I’m relatively certain people do not read this, at least regularly. Secondly, even though I have advanced significantly in my career and I am now a very competent middle manager, I believe my entire happiness in my work life is suffering simply because we seem to defer to political correctness. Making sure no one’s feelings are hurt rather than running an organization as a meritocracy where the best ideas win seems to be lay of the land, at least in American business these days. I used to not be afraid of putting my thoughts out there, now I seem to be in constant fear that being liked is the most important aspect to career advancement. I have decided that while I have come very very far in the last few years attempting to ensure that I am effective at change in a large organization due to being well liked and respected, I will no longer place being liked ahead of stating what I think will be in the best interest of our organization. That does not mean I intended to go back to being a daft prick like I have been in former years, but I will be firm on my thoughts on the direction our organization should be taking. I want these ideas to stand on their own, so this will be immediately followed by another post on being too customer centric.
I just got back from spending three days at Cisco’s corporate headerquarters. I’ve spent a good portion of my career dealing with vendors. I’ve spent days in meetings, evaluating products, specing out hardware, leased lines, and other ancillary pieces for building out networks and datacenters, but rarely have I left a set of meetings so jazzed about the future.
The first day was spent in design meetings and demo sessions. We opened the day by outlining where we’re currently at with our network. Currently, it’s a fairly well designed network with quite a few well known warts, but for the most part it functions very well. We’re not really hitting any capacity issues anywhere, but for the first time we’re really trying to plan out 2 to 5 years in advance. We know our current backbone, which is OC12 at the core nationwide, won’t last us for probably more than the next 12 to 18 months. We know this, because every time we improve our network, we find new ways to utilize it and we blow away all our traffic projections. However, we know that while our current design is working really well, we don’t want to upgrade 10 core locations to OC48 since half the sites sitting on that core won’t need that kind of connectivity. So we spent the first part of the morning drawing up a new core architecture with fewer sites and ideas for making it more highly available, including using DPR and/or MPLS fast reroute to provide sub-50 millisecond or sub-100 millisecond failover time in the event of a network failure. I think we have a promising set of ideas to go into our meetings next month with Level 3, who we purchase the majority of our core connectivity from.
The second half of the day was spent in demos. We saw some interesting new products on the Unified Communications front, as well a demo of Telepresence. Cisco makes this point often, but it can’t be stressed enough. Telepresence cannot be compared to video conferencing. We have an existing video conference system, which Telepresence has inspired me to improve, but it’s clunky and awkward by comparison. Telepresence really makes you feel like you’re sitting across the room from the other participants. Our current video conference system is used maybe once a month nationwide, but I guarantee Telepresence would be booked 100% in our company. I’d like to try to make our existing video conference system more like Telepresence, because a lot of the ideas Telepresence puts forward in terms of controlling the environment, using high quality video, and directional audio for the partipants can be adapted to our existing system. It won’t be Telepresence, but it certainly would improve what we currently have.
We finished up the day with a demo of products Cisco is pushing for retail and a discussion about their Service Control Engine product. The retail demo inspired a discussion about using EVDO as a backhaul for our retail stores, which is so obvious as to be a real slap in the face that we’re not already testing it out. We already own that last mile connectivity, why are we continuing to pay our competitors for connectivity to our retail outlets? Also, the Service Control Engine discussion was fascinating. We went from discussing a product we didn’t even know we owned to talking about all the things it could do for us. If you hear about providers shaping traffic down for Peer to Peer traffic and dunning customers for high bandwidth utilization, this is the product that’s doing it. It can shape traffic at wire speeds including multiple gigabit connections. Really really impressive.
The next two days were spent in Cisco’s Customer Proof of Concept lab. We went there to prove that the 7600 platform which we’re currently basing our core network architecture on will scale to meet our demands over the next 2 to 5 years. As I expected it did. I also had an excellent discussion with a Cisco executive over in the Unified Communications area about how we can leverage our investment in our IP Telephony infrastructure to really improve communications and productivity.
Cisco really went out of their way to get us access to people who would could answer our questions, give us good ideas for the future and give us access to equipment to test our their claims for our own assurance. They’re really the gold standard that I’m holding everyone else up to in terms of how they should treat their customers. We could have spent the same, dollar for dollar, with many other vendors and barely gotten a sales engineer to give us a call back, but Cisco seems to understand from the top down that people sell products, the products don’t sell themselves. They understand that we’re looking for a vendor to partner with us to help us get to our next level. Most often in this business there are two or more products that will do any given job, and while technically one might be superior, in the end both would work for a given task. The decision comes down to more than what exactly the product does, and while having a superior product helps, having a superior organization that stands behind that product and offers it at competitive prices is in my book the best way to get your product into customers hands. Cisco has a product for nearly anything you need in the networking world, and while we won’t ever be exclusively a Cisco shop, they’re much more likely to sell us a product on a given day due to the fact that they spend so much time focusing on helping their customers implement their solutions and spend so much time focusing on helping their customers support their existing installations. At the end of the day, the most important question I have to ask when making a purchasing decision usually isn’t what is the best product, it’s which is the best company. Right now, Cisco’s going a long way to convincing me they are.
Over on Bram Cohen’s blog (of BitTorrent fame), Mark Cuban made a comment that reminded me of a pretty common misconception about the ISP business. Mark said, in reference to Bram and attempting to make money from BitTorrent:
Unfortunately for you,ISPs crack down on heavy bandwidth users, particularly uploaders and enforce their TOS.
By definition, seeders create upstream bandwidth. The ISPs dont want to see more upstream usage Bram, i know its a tough concept for you, but in the mind of the ISP, upstream use = bad. MOre upstream b/w use = more bad. Which in turn pushes them not to increase the bandwidth available to end users, but to evaluate where the upstream use is coming from and look at shutting it off and throttling it. Call me crazy, but that equates to a challenge for the BT universe.
This couldn’t be further from the truth. The upstream bandwidth is not a concern for most ISPs, especially your standard Cable or DSL provider. Broadband ISPs have to order bandwidth synchronously, in various speed increments, because all high capacity circuits come in synchronous form. This means if you order a Gigabit Ethernet connection to an upstream provider, you’re actually buying 2 Gigabits per second worth of bandwidth, one from your provider and another towards your provider. Broadband ISPs sell bandwidth asynchoronously (the A in ADSL), which means they’re selling you something like 6 megabits down and 784k or less upstream. Obviously, it’s well known that ISPs oversubscribe their upstream links which is why peak times can see serious difficulties achieving the full downstream. However, no matter how you do the math of an ISP which serves mostly bandwidth consumers, this still leaves large amounts of bandwidth on the upstream on the ISPs egress connections. In fact, most Broadband ISPs don’t provision large WANs to haul traffic back to central egress points, because it makes more sense to dump the traffic off at a peering point in the market, assuming you’re in a large metro where you can cheap bandwidth. Because ISPs providing broadband to homes and businesses also have such a one way usage pattern, it means that they cannot negotiate peering arrangements with other ISPs for an even trade of bandwidth, meaning they’re always going to be paying for that bandwidth.
In the business, this is generally called Tier 2 bandwidth. Broadband ISPs will sell large hicap circuits to businesses largely interested in serving content much cheaper than you can buy Tier 1 bandwidth from a major internet provider (something like UUNet, etc). This also becomes very evident if you attempt to buy bandwidth from a Tier 1 provider and a Tier 2 or Tier 3 provider, because it becomes very difficult to load balance your traffic over to the Tier 2 providers because of the way BGP routing works. There are methods for overcoming this (AS padding), but it still never works out quite the way you want.
So, in summary, ISPs which are selling bandwidth to users which are using BitTorrent don’t give a rats ass about the upstream. They’ve got loads to spare. What they’re concerned with is the users who are using large amounts of upstream bandwidth are generally also heavy consumers of downstream, and those people are causing oversubscription issues.