Commit f9cfc571 authored by Manuel Kieweg's avatar Manuel Kieweg 🤷
Browse files

Merge branch 'develop' into 'master'


See merge request cocsn/gosdn!130
parents 2e2349b8 4f5e8d6e
authors = ["Martin Stiemerling", "Manuel Kieweg", "Malte Bauch"]
language = "en"
multilingual = false
src = "."
git-repository-url = ""
git-repository-icon = "fa-gitlab"
site-url = "/public/"
\ No newline at end of file
title: "goSDN"
subtitle: "A networking drama"
- "Malte Bauch"
- "Manuel Kieweg"
- "Martin Stiemerling"
bibliography: [./bibliography.bib]
- \usepackage{graphicx}
\ No newline at end of file
# Introduction
Data networks consists out of a variety of different network elements, link types, end hosts, services and requirements of such services. Further data networks consists not only of a single plane, but have different (logical) networking planes that have different tasks within any data network, i.e., the control plane, data plane and the network management plane. Keeping track of the different elements, links, hosts, services, their interactions, their runtime behavior on the 3 networking planes is a non-trivial tasks that is usually subsumed under the very broad term of network operations.
There are different approaches for network operations that are not only divided by their logical distinction but also how an implementer, typically network equipment vendor, is implementing the network elements and the particular operations.
We outline two basic approaches to network operation:
1. fully integrated network operations of all networking planes, i.e., usually called the traditional approach.
2. separation of control- and data planes, i.e., usually called the Software Defined Networking (SDN) approch, though there have been implementations of this concept earlier than SDN with other names, e.g., Forwarding and Control Separation (ForCeS) and others.
## Motivation
## Overarching Project Goals
* Keep It Simple, Stupid (KISS)
* Reuse existing technologies bits wherever possible if those are stable, i.e., documented, maintained etc, on a long time scale
* Integrate state-of-the-art technologies and methodologies
* Automate almost everything right from the beginning
* Document, Document, Document
* be an excellent citizen: test, test, test
* no hacks!
Some unsorted thoughts the be ordered yet:
* Storage of information in some external database, make use of database features, e.g., for state replication
* eye towards web-technologies/features, such as externalisation of state
* one API to "talk" administratively to the network controller
* core of controller should be kept simple/lean, i.e., just the basic primitives
* core should be extended by modules to add functionality beyond basics
* modules should be loaded (or unloaded) during runtime of the controller core
## Use Cases to be considered
The development of a general purpose SDN controller is not the primary goal at this early stage of the project.
Instead there are two use cases to be considered in the implemenation works that are currently ongoing:
* Primary: optical domain SDN-controller for the CoCSN project
* Secondary: SDN-controller for our local labs to manage an Ethernet-based lab environment
### Primary: optical domain SDN-controller for the CoCSN project
For this use case we initally do not consider the direct control of optical network elements, e.g., Optical Add-Drop Multiplexer (OADM) but we focus on optical network domains managed by another (SDN) controllers. The goSDN controller communicates with this domain controller and can request information about the optical network elements, the links between them and the optical and logical configuration of the network domain.
In a second step, the goSDN controller has to communicate with multiple domain controllers and has to find potential interchange points between these multiple domains. This is the preparation for a later step in this use case, when the goSDN controller has to find a network path between two end-points across multiple optical domains, including backup paths.
The intention here is to use an existing SDN southbound interface, very likely based on RESTCONF.
### Secondary: SDN-controller for our local labs to manage an Ethernet-based lab environment
For this use case we consider one of our local labs, e.g., either the telecommunications or networking lab, and how this lab with all its networking parts can be managed by the goSDN controller. In this case, the controller has to learn about all (network) elements, the links and the topology by obtaining all the required information and its own topology computation. This will require an interface between goSDN and the network components that is potentially beyond the typical SDN southbound interfaces.
## Structure of this Memo
This memo starts with this introduction that sets the stage for the theoretical underpinings of the SDN-controller
and the acutal implementation (and the various choice for this). Chapter 2 discusses the related work and chapter 3
outlines the theoretical foundations related to the control of networks and their relation to SDN. Chapter 4 uses
the output of Chapter 3 to define the conceptual design of the goSDN controller and some discussions about the pro
and cons of conceptual design. Chapter 5 describes the actual design of the current goSDN implementation and is
meant to be a compendium for the source code.
# Related Work
## This
- Other earlier works on SDN controllers, such as, Floodlight, ONOS, OpendayLight, POX
- Discription of the SB-Protocols
### And that
\ No newline at end of file
# Theoretical Underpinnings for SDN controllers
## What is a network at all?
Some loose thoughts for this chapter, as data networks consists out of:
* interfaces
* links connecting interfaces
* (network elements, or short elements. This comprises all types of devices attached to a network, such as hosts and (optical) switches
* hosts having at least or n interfaces
* hosts with ability to do forwarding decisions, e.g., an IP router on layer 3, an Ethernet-switch on layer 2, or an optical switch on layer 1 (i.e., wavelength)
* differentiation between physical and logical links is needed, as
* physical links cannot be changed by a software, as some entity has to change the physical connection, i.e., either a human or by robots (see [@de-cix:robots] as one possible example).
* logical links can be changed by a software, but the ways of doing so depend on [network changes](#can-the-network-be-changed-during-operation).
* a logical link is tight to a physical link or to an underlying logical link (a case of recursion)
* basic properties of links: directionality (uni or bi)
* directionality (uni or bi)
* point-to-point
* point-to-multipoint (multicast or broadcast)
* multi-point-to-multipoint (multicast or broadcast) (XXX is this really true?)
* configuration of interfaces
* configuration of hosts
* configuration of services on hosts
* configuration of global parameters, i.e., the network
### Can the network be changed during operation?
Changes of device or service configurations -- this is intended for semi-static configurations:
* classical way: physical links can be changed by a software via the command line interface (CLI), e.g., a router can create a new IP~subnetwork on an interface via the CLI.
* SDN way: physical links can be changed by a software via an Application Programming Interface (API), e.g., a router can create a new IP~subnetwork via an API
Changes of forwarding behavior -- this is intended for dynamic configurations that require to keep state in the network:
* classical way: not possible, as human interaction is just too slow for any reaction.
* network build-in: the control plane of a network device autonomously decides to change the flow forwarding behavior. Examples are: IP routing, firewalls and Network Address Translators (NATs).
* SDN way: forwarding behavior of the network can be changed via an API
### Network Management}
ISO/IEC ISO/IEC 7498-4, FCAPS, ITIL, Assets, configuration, etc
Configuration: Set, Query, Change etc. of configuration parameters
## Requirements
Some requirements to be filed here.
### Basic properties of networks
* recursion
* abstraction
# Conceptual Design of a SDN Controller as Network Supervisor
## Conceptual Structures
This section discusses the conceptual organizationak forms of data networks, as this seems not to be clear in many contexts. The main purpose is to clarify what in a network has to be managed, how it has to be managed and by what entity it has to be managed.
### Principal Network Domain (PND)
#### Definition
Any network consists out of basic components that are the collection of (network) elements used to form such particular network. These components, let it be any device attached to this network and the (physical) links, with their control-, data-, and management planes, form the Principal Network Domain (PND). Such a PND wil be typically under a single control, e.g., a University faculty, a company, or a data center owner.
The term PND does not express anything about logical network abstractions, such as VLANS, IP addresses, forwarding capabilities, etc. It is solely a construct to give all elements in a network a home.
For our purpose of managing a network from a network controller, we define that such a controller is
* directly in charge of the devices in the PND and thus be able to manage these network elements directly or,
* connected to a different lower network controller. The network controller would be only able to indirectly communicate with the network elements via the lower network controller or even, in case of a recursion of network controllers, only be able to talk to an even lower level network controller.
The differentiation between the PND and the following definitions of network domain, e.g., IP network, etc is important for the design of a network controller that is supervising a network in its whole.
A single controller may be in charge of multiple PNDs at the same time, though there may be multiple instances of this controller at the same time.
Any network element or host can only be part of a single PND. In case a device resides on the edge between two PNDs, e.g., a router, such a device will be under the control of only one of the PNDs, but it will be designated as a gateway device between these two (or more) PNDs.
An example for a PND:
* Ethernet switches with Ethernet links
* DWDM optical switches with fibre connections
#### Information to handle
* **principal element inventory**
This contains all known elements (such as end-hosts or network element as optical switches), independent of their relationship, of the network. This includes their hardware configuration, such as, interfaces attached to a particular host.
* **principal topology inventory**
This contains all known links and their connection to interfaces of elements out of the principal element inventory.
* ** Southbound Interface (SBI) **
The SBI to be used for this PND. The intention is to have one SBI for any given PND in order to configure the members of the PND.
### Logical Network Domain (LND)
#### Definition
A logical network domain (LND) is the implementation of a network domain within a PND or within a LND. Such a LND defines what protocols are to be used and builds on top of the structure of the underlying PND or LND. Two exemplary examples:
* a LND on top a PND: a VLAN-enabled Ethernet switch where the VLANs from a logical topology on top of the physical infrastructure.
* a LND on top of an underlying PND: an IP subnetwork based on a VLAN LND.
LNDs specify the operational parameters, such as forwarding capabilities (e.g., Ethernet, VLAN Ethernet, IP, etc), which elements of a PND are part of the LND, etc.
A logical network domain is bound to a single PND. Logical network domains can be part of logical network domains, i.e., this is a case of recursion.
#### Information to handle
* **domain element inventory**
contains the elements part of a particular network domain and it has to be a (sub)-set of the elements of the principal element inventory or a logical abstraction, such as a container or a virtual machine.
* **domain topology inventory**
This contains all known logical links and their connection to interfaces of elements out of the domain element inventory.
* ** Southbound Interface (SBI) **
The SBI to be used for this LND. This may be different from the SBI of the PND and allow so to span logical networks with their own SBI-implementation. One possible example is that the PND may use Ansible-scripts to setup the basic configuration of the devices, e.g., switch-on interfaces, enable Openflow etc and allow the LND to use Openflow for its operation. A second LND may use a RESTCONF-approach to configure its devices.
## Different Types of Southbound Interfaces
The communication with the network elements is facilitated via the Southbound Interface (SBI). The SBI allows an entity external to the network element (NE) to control all or certain behaviors of such NE. We differentiate between three basic SBI types for our further discussion:
* Flow-based: a controller can instruct the NE to alter the forwarding of particular flows based on a fixed set of forwarding instructions and filters.
* Topology-based: a controller can instruct the NE to alter the topology of a network, e.g., by switching interface on or off, but it may not be able to alter the forwarding of flows
* Programmability-based: a controller can load the forwarding logic as a program on the NE and can include the forwarding rules as part of the program code.
It is important to note that the basic types of SBI can stand on their own, but they can also possibly appear in combinations, e.g., a combined flow- and programmability-based SBI sounds possible.
## Core functions
The short description of core functionality of goSDN: Keep track of everything and be in control of everything.
An half-unsorted list of core functions:
* manage SBIs
* Load SBIs
* Use predefined SBI-Interfaces
* Configure SBI instance , e.g., IP/port of SBI server, certificates, etc
* SBI generic error handling, such as, failure of connection to SBI server
* Link between SBI errors and "SBI-content" (e.g., topology information), i.e., what to do if a SBI instance fails.
* Denote PND to SBI relationsship
* read-only access to network information (SBI-content)
* learn about NEs, interfaces, links and hosts in a PND via the SBI
* learn about NEs, interfaces, links and hosts in zero, one or more LND via the SBI
* learn about changes with respect to NEs, interfaces, links and hosts in PNDs and LNDs
* store information about NEs, interfaces, links and hosts in PNDs and LNDs
* related this information to each other, such as, what interface is attached to what NE or what link is connecting which interfaces
* write access to network information
* use all functions of the "read-only" part above in order to learn about what's happening in the network
* create a LND
* use the topology information to calculate possible paths between parts of the network or between attached hosts in a PND or LND
* write information to SBI
* topology information
* flow forwarding information
* general configuration information, e.g., interface up (?)
* handle feedback from SBI (success, failure, issues)
* received this type of information
* process it in a meaningful way
* SBI-Interface things
* PND/LND differentiation needed
* Exchange of validity information, such as timer for information entries (i.e., NEs, interfaces, links and hosts )
* Representation of network information (NEs, interfaces, links and hosts ) in the core
* Representation should be independent of SBI but requires specific information provided by or via the SBI
* Readling list for Representation
* our current shoot-through-approach (TAPI JSON dump to DB)
* TAPI as information model in the core
* No use for us: ACTN:
* Storage in the graph database
## Parking Lot for pending Stuff
* **host configuration**
This is based on the information provided by the host inventory and contains the actual operational configuration of the hosts. This will probably contain only the configuration of the network devices, such as, switches and routers, potentially also servers, but not end-hosts.
* **Network**
* **Network Configuration)**
* **Southbound Interface (SBI)**
* **Northbound Interface (SBI)**
* **East-West-bound Interface (SBI)**
## Applying Changes to What Plane?
Some basic thoughts to dissect how different approaches are applying changes to the various planes.
### Changes to the Control Plane
### Changes to the Data Plane
This is the use case for the SDN approach: A so-called SDN-controller applies policy rules to the data plane. These policy rules are defining the handling of the flows in the networks on a larger scale or to be more precise the handling of more less specified packets.
A change to the data plane will not directly trigger a change to other planes. Though the flow of packets on the data plane can be observed by the control plane and the control plane can take action depending on the data packets.
### Changes to the Management Plane
# Implementation Aspects of the goSDN Controller
## Why we do this in go
Because it rocks, but let's see afterwards what can be written here.
## Storing Information
Section XXX (Conceptual Design of a SDN Controller as Network Supervisor)
discusses the need to store information about for element inventories and
topology inventories.
### Element Inventories
Storing information about network elements and their properties is a relative
static process, at least when one considers potential changes over time.
Typically such network elements are added to a network and they will remain in
the network for a longer time, i.e., multiple minutes or even longer.
### Topology Inventory
Every network has one given physical topology (G<sub>physical</sub> ) and on
top of this at least one logical topology (G<sub>logical1</sub>). There may be
multiple logical topologies (G<sub>n+1</sub>) on top logical topologies
(G<sub>n</sub>), i.e., a recursion. Such logical topologies (G<sub>n+1</sub>)
can again have other logical topologies as recursion or other logical topologies
in parallel.
A topology consists out of interfaces, which are attached to their respective
network elements, and links between these interfaces.
Mathematically, such a topology can be described as a directed graph, whereas
the interfaces of the network elements are the nodes and the links are
the edges.
G<sub>physical</sub> ist a superset of G<sub>logical1</sub>.
The topology inventory has to store the particular graph for any topology and
also the connections between the different levels of topologies. For instance,
the G<sub>logical1</sub> is linked to G<sub>physical</sub>. (needs to be clear
if changes in n-1 graph has impact on n graph).
For further study at this point: Which type of database and implementation of
databases should be used to store the different topology graphs and their
pontential dependencies? How should the interface between gosdn and this
database look like?
Here is an attempt to describe the above text in a graphical reprensetation (kinda of...not perfect yet):
graph TB
SubGraph1 --> SubGraph1Flow
subgraph "G_logical1"
SubGraph1Flow(Logical Net)
Node1_l1[Node1_l1] <--> Node2_l1[Node2_l1] <--> Node3_l1[Node3_l1] <--> Node4_l1[Node4_l1] <--> Node5_l1[Node5_l1] <--> Node1_l1[Node1_l1]
subgraph "G_physical"
Node1[Node 1] <--> Node2[Node 2] <--> Node3[Node 3]
Node4[Node 4] <--> Node2[Node 2] <--> Node5[Node 5]
Net_physical[Net_physical] --> SubGraph1[Reference to G_logical1]
### Potential other Inventories
There may be the potential need to store information beyond pure topologies,
actually about network flows, i.e., information about a group of packets
belonging together.
## Database
A database will be used for the management and persistence of network
topologies and their associated elements within goSDN.
Since network topologies are often depicted as graphs, it was obvious to stick
to this concept and, also due to their increasing popularity, to use a graph
database. After a more intensive examination of graph databases it was found
that they (with their labels, nodes, relations and properties) are well suited
for a representation of network topologies.
The first basic idea was to create different single graphs representing the
different network topologies and label each node and edge to ensure a clear
assignment to a topology.
This would mean that nodes and edges of a graph have 1...n labels.
Therefore if you want to display a simple network topology in a graph, you can
display the different network elements as individual nodes and the edges between
network elements as their respective connections, such as Ethernet.
This works with both physical and logical topologies, which are described in
more detail [here](#topology-inventory).
So a simple topology in a graph database could look like shown below.
graph TD
A[Node 1 - Label: 'Host,physical'] -->|Ethernet - Label: 'physical'| B[Node 2 - Label: 'Hub,physical']
C[Node 3 - Label: 'Host,physical'] -->|Ethernet - Label: 'physical'| B
B -->|Ethernet - Label: 'physical'| D[Node 4 - Label: 'Host,physical']
B -->|Ethernet - Label: 'physical'| E[Node 5 - Label: 'Host,physical']
For this purpose some experiments with the [Redis](
module [`RedisGraph`]( were carried out
first. The basic implementation was possible, but the function of assigning
several labels to one node/edge is missing (originally we considered this to be
indispensable especially to map different topologies).
For this reason we looked around for an alternative and with
[neo4j]( we found a graph database, which gives us the
possibility to label nodes and edges with a multitude of labels and offers a
wide range of additional plugins such as [apoc](
### neo4j
TODO: add a little description for neo4j in general
#### Implementation With neo4j
The current implementation offers the possibility to persist different network
elements (e.g. devices, interfaces...) and their physical topology and mainly
serves to represent the prototypical dataflow of goSDN to the database.
The following figure shows our first idea of a persistence of network
topologies with neo4j (to save space, only the labels were included).
graph TD
A --> |belongs to| PND
B --> |belongs to| PND
C --> |belongs to| PND
D --> |belongs to| PND
E --> |belongs to| PND
A[Label: 'Host,physical,logical1'] --> |Label: 'physical'| B[Label: 'Hub,physical,logical1']
D[Label: 'Host,physical,logical1'] --> |Label: 'physical'| B
B --> |Label: 'physical'| C[Label: 'Host,physical,logical1']
B --> |Label: 'physical'| E[Label: 'Host,physical,logical1']
A --> |Label: 'logical1'| B
B --> |Label: 'logical1'| C
C --> |Label: 'logical1'| D
D --> |Label: 'logical1'| E
E --> |Label: 'logical1'| A
The basic idea is to assign the different network elements to a specific
Principal Network Domain (PND). The different topologies are represented by a
neo4j relationship between the network elements that are stored as neo4j nodes.
However, with this current variant it is not possible, as required in
[Topology Inventory](#topology-inventory), to represent topologies that are hierarchically
interdependent, since neo4j does not allow relations to be stored as properties
(as described [here](
Furthermore, multiple links between the same nodes which belong to the same
topology are difficult to represent, since this model only provides a single
link between nodes of a certain topology.
For the reason mentioned above, a more complex idea for persistence is available
for the further development, which hopefully allows us to persist and map
network elements, PNDs and topologies with all their hirarchical dependencies.
The following figure tries to visualize this idea.
graph TD
subgraph "dependencies of topologies"
logical1 -->|related_to| physical
logical5 -->|related_to| physical
logical3 -->|related_to| logical1
subgraph "every node belongs to a specific PND"
Node1 -->|belongs_to| PND
Node2 -->|belongs_to| PND
Node3 -->|belongs_to| PND
Node4 -->|belongs_to| PND
Node5 -->|belongs_to| PND
subgraph "relationship between nodes (nodes can be linked by 0...n links)"
lp2 --> |connects| Node4
lp2 --> |connects| Node2
lp3 --> |connects| Node2
lp3 --> |connects| Node3
lp4 --> |connects| Node2
lp4 --> |connects| Node5
lp5 --> |connects| Node1
lp5 --> |connects| Node2
subgraph "links are part of a topology"
lp1 --> |connects| Node1
lp1 --> |connects| Node2
lp1 --> |part_of| physical
subgraph "links can contain 1...n layers"
lp2 --> |contains| ODUH
lp2 --> |contains| OTUCN
lp2 --> |contains| ODUCN
The basic structure explained in the upper part remains the same.
However, the relations, which previously served as links between the respective
nodes, now become **separate nodes**. These nodes now act as links between the
respective network elements and are part of a network topology (which itself
is represented as a separate node in the graph). By this change, network
topologies can now be interdependent. Furthermore, as can be seen in the figure
above, you can add additional nodes to the link nodes by using this scheme.
So a physical link between two nodes could e.g. **contain** several cables.
All other information can be stored in the properties of the respective nodes/edges.
The above idea is not yet approved and there are still open questions.
- Is there a better solution for the assumption that there are several different physical connections between the same nodes than separate link nodes between them?
- Can topologies run over different PNDs -> membership to different PNDs?
- Where can we benefit from using different layers? (e.g. possible saving of unnecessary relations between nodes)
- Do the sdn controllers provide us with the necessary information to map the topologies in this way?
- ...
## YANG to code
The base of the development of goSDN are YANG modules. The RESTful API used for RESTCONF is defined in an OpenAPI 2.0 file. This API documentation is generated from the YANG module. The YANG module description is also used to generate code stubs for the goSDN RESTCONF client.
### YANG
YANG defines an abstract network interface. It is the foundation of the RESTCONF protocol. Several code generators exist to generate code stubs from a given definition.
### OpenAPI
OpenAPI - formerly known as Swagger - is a framework that defines RESTful APIs. We use OenAPI documentations to define the RESTCONF server implementation of the cocsn YANG modules.
### Toolchain
We use 3 different tools for the code generation workflow. For the RESTCONF server `yanger` is used to generate the OpenAPI documentation from the YANG file. `go-swagger` is used to generate a RESTCONF server with stubs for the REST calls.
The RESTCONF client stubs used by goSDN are generated from YANG files using YGOT.
### Dependencies
For now we can only use the OpenAPI 2.0 standard. This is because `go-swagger` does not support OpenAPI 3.0 specifications yet.
## Storing Information
This section keeps by now some loose thoughts about what information has to be stored how and where.
There seem to be two classes of information to be stored in the controller:
* short-living information, such as, current configured network flows or obtained network configuration out of use case #1 (CoCSN)
* long-time information, such as, information about principle network domains, elements in such a domain if directly learned from SBI, etc
Long-time information should be persistenly stored in the database and survive reboots of goSDN etc. Short-Living information doesn't have to survive reboots of goSDN
### Some more details for implementation for the database(s)
We define the principle network domain (PND) and each piece of information of any PND has to be stored in relation the particular PND.
Specification of a PND:
* Human readable name of PND
* Textual description for further information
* Set of supported Southbound-Interfaces, e.g., RESTCONF, TAPI, OpenFlow etc
* Physical Inventory Network Elements, hosts and links, pontentially only the SBI SDN controller
A PND entry must be explicitly generated, though some information can be automatically be generated, e.g., the physical inventory for use-case #1 (CoCSN) would mean that the information about the SBI domain specific SDN controller is entered.
<?xml version="1.0" encoding="utf-8"?>
<style xmlns="" class="in-text" version="1.0" demote-non-dropping-particle="sort-only" default-locale="en-US">
<title>ACM SIG Proceedings ("et al." for 3+ authors)</title>
<link href="" rel="self"/>
<link href="" rel="documentation"/>
<name>Naeem Esfahani</name>
<name>Chris Horn</name>
<name>Patrick O'Brien</name>
<category citation-format="numeric"/>
<category field="science"/>
<category field="engineering"/>
<rights license="">This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License</rights>
<macro name="author">
<if type="webpage">
<text variable="title" suffix=":"/>
<names variable="author">
<name name-as-sort-order="all" and="text" sort-separator=", " initialize-with="." delimiter-precedes-last="never" delimiter=", "/>
<label form="short" prefix=" "/>
<names variable="editor"/>
<names variable="translator"/>
<macro name="editor">
<names variable="editor">
<name initialize-with="." delimiter=", " and="text"/>
<label form="short" prefix=", "/>