Where does a network keep its brain?

In the sample output of a lab handout, one computer sends the same small packet to its neighbour twice. The first reply arrives in 2.76 milliseconds. The second in 0.052 milliseconds. Same cable, same two machines, same packet; yet the second attempt is more than fifty times faster.

The difference comes from a question the first packet has to ask on its way: "what are you going to do with me?" The answer does not come from the devices inside the network but from software running outside it. That is the heart of what is called a software-defined network.

The one who decides and the one who obeys

In a classic network every switch thinks for itself. It learns which address sits behind which port, and it decides on its own where to send each packet. The brain is spread across every device.

A software-defined network pulls those two jobs apart. The switch now only looks up a table: "if a packet like this arrives, do that." The table is filled by a controller at the centre, which is simply a program running on an ordinary server. The common language between them is called OpenFlow.

Each row of the table has three parts: what to look at (which port the packet came in on, which address it is heading for), what to do (send it out of a given port, drop it, or ask the controller), and a priority. If several rows match, the one with the higher priority wins.

Empty table, silent network

You do not need real hardware to see this. A tool called Mininet builds switches, cables and servers virtually, inside a single computer.

Build a network of three computers and one switch without starting any controller, and a ping from one computer to another gets no reply. The switch is not broken. Its table is empty and it has nobody to ask.

Then two rows are written into the table by hand: whatever comes in on port one goes out of port two, and whatever comes in on port two goes out of port one. The same ping now gets an answer. The switch learned nothing; it simply did what it was told.

Rules have a lifespan too. In the tool used in the lab, hand-written rows are deleted by default once they have seen no packet for sixty seconds, and the ping falls silent again. OpenFlow defines two timers here: the idle timeout, a clock that runs only while the rule goes unused, and the hard timeout, which removes the rule when it expires however much it has been used.

Why is the first packet slow?

Now the controller is running. When a computer pings its neighbour for the first time, the switch has no row matching that traffic. The table holds only a fallback rule with the lowest priority: "if nothing matches, ask the controller."

The switch sends the head of the packet up to the controller. The controller decides where the packet should go, writes a new rule into the switch, and tells it to send the waiting packet out of the right port. When the second packet arrives, the rule is already in the table. The packet passes straight through the switch without ever reaching the controller.

The controller does not see every packet. It only hears the first question of a flow and leaves the rest to the switch.

Most of those 2.76 milliseconds are the price of that round trip. In fact another question is asked even before the first ping: the computer does not know its neighbour's hardware address, so it first sends an ARP request that calls out to everyone "whose address is this?" That request travels the same route to the controller.

How do a switch and a controller get acquainted?

When a switch starts, it connects to the controller and the two go through a short introduction. Messages captured with Wireshark show the conversation step by step.

One curious detail: the switch never acknowledges a FLOW_MOD message. Silence means success; an error message comes back only when something goes wrong. There is also a difference between versions: in OpenFlow 1.0 the port list came inside the first reply of the introduction, while in 1.3 it is fetched with a separate request.

How does the controller draw the map?

To make decisions, the controller needs a map of the network: which switch is connected to which, and through which port. Nobody hands it this map. It draws the map itself, with a simple method.

The controller asks every switch to send a small frame out of each of its ports. The frame carries a label: "I left this switch through this port." The switch at the other end of the cable does not recognise the frame, finds no rule for it, and passes it to the controller with its own information added: "this arrived on my port such-and-such."

The controller puts the two pieces side by side and finds a cable. Repeat that for every port and the whole map appears. These frames speak a language called LLDP. They carry the value 0x88CC in the Ethernet type field and are addressed to 01:80:C2:00:00:0E. Standard switches never forward frames sent to that address, so the label travels only one cable's length before the switch on the far side hands it to the controller.

This discovery is not done once and forgotten; it is repeated at regular intervals, every three seconds by default in ONOS. A shorter interval means a broken cable is noticed sooner, but it also puts more control traffic on the network. A longer one lightens the load, at the cost of a map that can stay out of date for longer.

How does a network break?

The two most instructive experiments in the lab break the network on purpose.

In the first, cheap switches are wired together into a ring, and none of them runs the protocol that prevents loops. Setting it up shows no error at all. The trouble starts with the first ping.

The ARP request before the ping is a broadcast sent to everyone. Every switch sends it out of every port except the one it came in on. In a ring the frame goes round, returns to where it started and is broadcast again, multiplying on every lap. This is called a broadcast storm: links fill up and pings go unanswered.

An IP packet loses one unit of life at every router it crosses and eventually dies. An Ethernet frame has no such counter; once it enters a loop, it circulates forever.

In real networks the spanning tree protocol prevents this. Described by Radia Perlman in 1985, it switches off some of the redundant links and turns the ring into a tree; the spare cables stay in place but are not used until one breaks.

The second experiment imitates the layout of a data centre. At the top sits a single core switch, below it aggregation switches, below those the top-of-rack switches, and at the bottom the servers. Each layer branches n times; with n equal to two the network has seven switches and eight servers, with four it has twenty-one switches and sixty-four servers.

While everything runs, every server reaches every other. Then the core switch is stopped. The network splits in two: the four servers under the same aggregation switch can still talk to each other, because their paths never pass through the core. But no packet can cross from one half to the other. Of the fifty-six possible directions between eight servers, thirty-two are cut.

That is exactly the weak spot of a tree layout: a single device sits at the root, and if it goes, the network falls apart into islands. Large data centres are built with several cores and spare paths between layers for this reason.

Is one brain at the centre a good idea?

Gathering decisions at the centre has clear benefits. The controller sees the whole network from one place. When new behaviour is wanted, a program is changed instead of logging into devices one by one. Traffic can be steered with the entire network in view.

The same centralisation raises a new question. The single point of failure seen in the core switch experiment reappears on the software side: if the controller cannot be reached, nobody can decide what to do with new flows. That is why controllers in real deployments run on several servers that back each other up.

Scale has its limits too. A network built inside one computer shows the concepts faithfully, but not the latency, traffic and failures of a real data centre. The timings here measure how the mechanism works, not what will happen on real cables.

Zhipeng Wang

Founder and writer 2 pieces

Studies Computer Network Systems at Paris-Saclay, on the Systems and Networks track. Speaks Chinese, French and English.

See all their articles