When a customer isn’t just a customer

I just got back from spending three days at Cisco’s corporate headerquarters.   I’ve spent a good portion of my career dealing with vendors.   I’ve spent days in meetings, evaluating products, specing out hardware, leased lines, and other ancillary pieces for building out networks and datacenters, but rarely have I left a set of meetings so jazzed about the future.

CPOCThe first day was spent in design meetings and demo sessions.   We opened the day by outlining where we’re currently at with our network.   Currently, it’s a fairly well designed network with quite a few well known warts, but for the most part it functions very well.   We’re not really hitting any capacity issues anywhere, but for the first time we’re really trying to plan out 2 to 5 years in advance.   We know our current backbone, which is OC12 at the core nationwide, won’t last us for probably more than the next 12 to 18 months.   We know this, because every time we improve our network, we find new ways to utilize it and we blow away all our traffic projections.   However, we know that while our current design is working really well, we don’t want to upgrade 10 core locations to OC48 since half the sites sitting on that core won’t need that kind of connectivity.   So we spent the first part of the morning drawing up a new core architecture with fewer sites and ideas for making it more highly available, including using DPR and/or MPLS fast reroute to provide sub-50 millisecond or sub-100 millisecond failover time in the event of a network failure.   I think we have a promising set of ideas to go into our meetings next month with Level 3, who we purchase the majority of our core connectivity from.

The second half of the day was spent in demos.   We saw some interesting new products on the Unified Communications front, as well a demo of Telepresence.   Cisco makes this point often, but it can’t be stressed enough.   Telepresence cannot be compared to video conferencing.   We have an existing video conference system, which Telepresence has inspired me to improve, but it’s clunky and awkward by comparison.   Telepresence really makes you feel like you’re sitting across the room from the other participants.   Our current video conference system is used maybe once a month nationwide, but I guarantee Telepresence would be booked 100% in our company.   I’d like to try to make our existing video conference system more like Telepresence, because a lot of the ideas Telepresence puts forward in terms of controlling the environment, using high quality video, and directional audio for the partipants can be adapted to our existing system.   It won’t be Telepresence, but it certainly would improve what we currently have.

We finished up the day with a demo of products Cisco is pushing for retail and a discussion about their Service Control Engine product.   The retail demo inspired a discussion about using EVDO as a backhaul for our retail stores, which is so obvious as to be a real slap in the face that we’re not already testing it out.   We already own that last mile connectivity, why are we continuing to pay our competitors for connectivity to our retail outlets?    Also, the Service Control Engine discussion was fascinating.   We went from discussing a product we didn’t even know we owned to talking about all the things it could do for  us.   If you hear about providers shaping traffic down for Peer to Peer traffic and dunning customers for high bandwidth utilization, this is the product that’s doing it.   It can shape traffic at wire speeds including multiple gigabit connections.   Really really impressive.

The next two days were spent in Cisco’s Customer Proof of Concept lab.   We went there to prove that the 7600 platform which we’re currently basing our core network architecture on will scale to meet our demands over the next 2 to 5 years.   As I expected it did.   I also had an excellent discussion with a Cisco executive over in the Unified Communications area about how we can leverage our investment in our IP Telephony infrastructure to really improve communications and productivity.

Cisco really went out of their way to get us access to people who would could answer our questions, give us good ideas for the future and give us access to equipment to test our their claims for our own assurance.   They’re really the gold standard that I’m holding everyone else up to in terms of how they should treat their customers.   We could have spent the same, dollar for dollar, with many other vendors and barely gotten a sales engineer to give us a call back, but Cisco seems to understand from the top down that people sell products, the products don’t sell themselves.   They understand that we’re looking for a vendor to partner with us to help us get to our next level.   Most often in this business there are two or more products that will do any given job, and while technically one might be superior, in the end both would work for a given task.   The decision comes down to more than what exactly the product does, and while having a superior product helps, having a superior organization that stands behind that product and offers it at competitive prices is in my book the best way to get your product into customers hands.   Cisco has a product for nearly anything you need in the networking world, and while we won’t ever be exclusively a Cisco shop, they’re much more likely to sell us a product on a given day due to the fact that they spend so much time focusing on helping their customers implement their solutions and spend so much time focusing on helping their customers support their existing installations.   At the end of the day, the most important question I have to ask when making a purchasing decision usually isn’t what is the best product, it’s which is the best company.   Right now, Cisco’s going a long way to convincing me they are.

New Pictures from New Camera

We recently purchased a Canon SD1000 for my 27th birthday. It’s an awesome, incredibly compact camera, unlike our old Nikon which was a total brick and never got used because it was inconvenient to tote around. I’m hoping we’ll be taking a lot more pictures since the camera will be a lot more convenient to have with us. In the mean time, we’ve added a gallery from today’s testing to show off the new camera. Keep checking back, Katie will be featured often. Here’s a sample image of the most beautiful girl in the world:

PHP Weathermap and Cacti

Once again, open source proves its usefulness. Currently we’re implementing a rather large network monitoring setup from several large companies, mostly with TLAs as their company names. These cost millions of dollars to buy, hundreds of thousands of dollars to customize and equally as much to get the basics working in your environment. I used to use a tool called tkined back in the day to monitor our ISP network via bandwidth graphs overlayed on a network diagram. This was real-time and was the ultimate in network monitoring at the time. Strangely, the tools have all advanced, but I had yet to find something to do an equivalant graphical view of the network. That was until I found PHP Weathermap, which can be installed as a Cacti plugin. We already had a small Cacti installation that just needed to be expanded to include the majority of our core routers (it was mainly being used for ancillary routers and switches), and then I drew the map on top of an existing Visio I exported to PNG. The result:


This gives a near real time view of network bandwidth utilization. It was exactly what I was looking for. When I discussed this with the team doing the implementation from the TLA company, they said it might be possible, but I have a feeling it’s going to cost more than it would take a programmer to rewrite this from the ground up. Thus is enterprise software: the more expensive the package, the less you get out of the box and the more likely you are to end of spending hundreds of thousands if not millions only to replace it with something simpler.

The Big Bandwidth Misconception

Over on Bram Cohen’s blog (of BitTorrent fame), Mark Cuban made a comment that reminded me of a pretty common misconception about the ISP business. Mark said, in reference to Bram and attempting to make money from BitTorrent:

Unfortunately for you,ISPs crack down on heavy bandwidth users, particularly uploaders and enforce their TOS.

By definition, seeders create upstream bandwidth. The ISPs dont want to see more upstream usage Bram, i know its a tough concept for you, but in the mind of the ISP, upstream use = bad. MOre upstream b/w use = more bad. Which in turn pushes them not to increase the bandwidth available to end users, but to evaluate where the upstream use is coming from and look at shutting it off and throttling it. Call me crazy, but that equates to a challenge for the BT universe.

This couldn’t be further from the truth. The upstream bandwidth is not a concern for most ISPs, especially your standard Cable or DSL provider. Broadband ISPs have to order bandwidth synchronously, in various speed increments, because all high capacity circuits come in synchronous form. This means if you order a Gigabit Ethernet connection to an upstream provider, you’re actually buying 2 Gigabits per second worth of bandwidth, one from your provider and another towards your provider. Broadband ISPs sell bandwidth asynchoronously (the A in ADSL), which means they’re selling you something like 6 megabits down and 784k or less upstream. Obviously, it’s well known that ISPs oversubscribe their upstream links which is why peak times can see serious difficulties achieving the full downstream. However, no matter how you do the math of an ISP which serves mostly bandwidth consumers, this still leaves large amounts of bandwidth on the upstream on the ISPs egress connections. In fact, most Broadband ISPs don’t provision large WANs to haul traffic back to central egress points, because it makes more sense to dump the traffic off at a peering point in the market, assuming you’re in a large metro where you can cheap bandwidth. Because ISPs providing broadband to homes and businesses also have such a one way usage pattern, it means that they cannot negotiate peering arrangements with other ISPs for an even trade of bandwidth, meaning they’re always going to be paying for that bandwidth.

In the business, this is generally called Tier 2 bandwidth. Broadband ISPs will sell large hicap circuits to businesses largely interested in serving content much cheaper than you can buy Tier 1 bandwidth from a major internet provider (something like UUNet, etc). This also becomes very evident if you attempt to buy bandwidth from a Tier 1 provider and a Tier 2 or Tier 3 provider, because it becomes very difficult to load balance your traffic over to the Tier 2 providers because of the way BGP routing works. There are methods for overcoming this (AS padding), but it still never works out quite the way you want.

So, in summary, ISPs which are selling bandwidth to users which are using BitTorrent don’t give a rats ass about the upstream. They’ve got loads to spare. What they’re concerned with is the users who are using large amounts of upstream bandwidth are generally also heavy consumers of downstream, and those people are causing oversubscription issues.

VMWare GSX Now Free

Joel Spolsky points me to a post by Mike Gunderloy about VMWare releasing their GSX Server product for free. This is outstanding news, and I plan to test it out at the office this week. I guess VMWare listened to my advice to Microsoft, which I wrote in May of last year. Virtualization technology is already free with Xen and Linux systems, it’s about time one of the commercial vendors released one of their lower-end products for people to use for free. Way to go VMWare!

Why would you want to kill exclusives?

I like Steve Rubel. I like Robert Scoble too. Robert’s dead wrong on this one though. For some reason, bloggers seem to think that just because there’s more of us and anyone can contribute the conversation, that somehow everything has to change. Not so.

Perfect example, we just did a major release about 3 weeks ago of FireAnt. We spent a lot of time on the product. The directory was over 4 months in development. There were test sites available to the public about a month prior to release. We seeded the release out to trusted videobloggers and our users groups for the product to get feedback, but we asked all of them to remain quiet. They did. The reason? We wanted to give someone who had traction the exclusive to write about the new release such that we’d get a bit of a bang with our release instead of a gradual dull thud. That exclusive fell to Mike Arrington of TechCruch, and we were not disappointed. He got the exclusive, he was happy, his readers got the scoop the day it was released, and we got extended coverage in the blogosphere echo chamber because we gave a high-profile blogger the exclusive.

Steve groks it. I’m not sure why Chris and Robert seem to think everything has changed. I could have had the exclusive or given it to someone like my good friend Steve Garfield (whose readership/viewership is nothing to sneeze at), but why would I want to release something to my 200 readers and wait for it to maybe disseminate throughout the blogosphere when I can seed it to someone with a much larger and more influential readership? If we had given it to everyone all at once, we would have ended up with that dull thud I was talking about earlier. Somebody has to help control the noise, and a little bit of PR and marketing savvy can go a long way to doing that.

Dumbest quote of the year

This came across an email list I’m on recently:

Ning co-founder Marc Andreessen recently said…

Ideally we’ll never meet any of our customers. We actually had to take the sign down from our front door because one of our customers actually stopped in, uninvited, and said, “Hi, I love your service.? And we’re like, “why are you here?? And so down came the sign.

Drop-bys like that should only happen in sitcoms as far as I’m concerned… The consumer internet businesses in a sense are ideal businesses from the standpoint of never meeting your customers.

Only in the technology business would anyone be caught dead uttering such an utterly stupid statement, and even then it doesn’t make it any less of a moronic comment. Your customers are your bread and butter. You should jump up and down if someone takes the time to stop by your office just to tell you how much they like you, and you should be just as excited if someone takes the time to tell you what you’re doing poorly, because it’s a chance to save a customer and make an advocate. This is something I’d expect to see on Rick Segal’s blog, in one of his infamous (at least to me) overheard dumb business conversations. I can’t honestly see anyone with this kind of attitude being successful in any business in the long run, technology or not.