All through their stealth mode it was abundantly clear to me what the geniuses who invented SDN and OpenFlow would be doing at Nicira. They would, of course, be building a commercial controller that fulfilled the Scott Shenker vision of the abstracted, high level development platform that the openflow controller could become, and then they would use  openflow to implement a massively scalable L2 network; the killer app for the biggest  data center operators. Wrong and Wrong. I wasn’t even close. Along the way they decided that instead of selling their controller as “The Platform” to an ecosystem of developers, they would bake it into their own solutions.   And the killer app they have chosen is a layer-2 over layer-3 solution that stuffs all the cost and complexity of a legacy, heirarchical, routed L3 cloud, underneath their overlay network. They have done this, even though their solutions include  a Hypervisor switch that neatly implements the exact type of encapsulation technology one would need to implement a layer 2 solution instead. Guido Appenzeller from Big Switch Networks, who fell off the same branch of the Stanford SDN tree as the Nicira leadership, recently reminded me that in the early days Nicira was positioned as a fabric switch company, and their early OpenFlow/SDN papers talked specifically about enabling this. So what changed?   It wouldn’t be shocking to learn that they headed out to do exactly what I anticipated and “pivoted” when the biggest data center operators told them that they are just now learning to trust layer-3 ECMP technology to introduce multipath “fabrics” into their networks. Nicira’s migration from programmable hardware switch to programmable VSwitch was likely also a recognition that OpenFlow availability on mature hardware platforms won’t be widely available for the current data center spending blitz. The already-started (2012-2014) DC server refresh will deploy 10G server interfaces and edge switches, but conservative operators are choosing L3 with ECMP as the way to increase their fabric-ness.  Moving from uni-path to multipath in the DC network will radically increase bisectional bandwidth and improve network performance, but it is dangerous and scary because it steps away from the known and familiar. Network engineers must answer the question “How the heck do you debug packet loss when there are multiple paths from A to B and path selection is by non-obvious algorithms?” One might argue that the overlay solution providers could bring out the pure-L2 solutions later, after folks get comfortable with the smaller step to L2-over-L3, but the benefits of all-L2 are seriously diminished if you’ve already paid CAPEX for the L3 solution (although the TCO hit of continuing to run it might argue for switchover). The question here isn’t really what kind of overlay do you prefer (where Nicira fits in), but rather, what is your multipath technology in the physical network that your overlay rides on. Nicira has their own protocol proposal that they can run over an L3 cloud, called STT. VMware’s contribution to overlay protocols, VXLan, is similarly an L2-over-L3 technology, as is Microsoft’s NV-GRE, and they all assume that the best fabric to transport overlay packets is based on standard dynamic routing protocols.  The combined VMware/Nicira doesn’t get us any closer to a next generation L2 solution in the physical network. All of the major multipath solutions are based on statistically load balancing across all of the N paths at each hop. Who’s providing the far superior implementation that intelligently assigns traffic to paths based on policy or workload specifics? ECMP is a step in the right direction, but it’s theoretical baisis is “divide it randomly and hope that on average you get what you want”. We can do better.