Legacy Cisco IOS

Cisco IOS (Internetwork Operating System) was Unix-based and originally designed in 1984 to implement routing functionality on Cisco hardware.  It was based upon a monolithic architecture which meant that any processes ran by the OS were stacked and interrelated.  This had two main issues with it:

1. Memory was shared across all processes.  This meant that any process could modify or even  corrupt the memory of another process, causing more than just an issue with the process which initiated the issue.
2. Run-to-completion scheduler.  All calls must go through the kernel, which will not interrupt any existing processes that have the CPU.  These processes must report back to the kernel when they are done before another process is allocated to the CPU.

Forwarding (Data) and Control plane functionality were thus combined into a single failure domain.  If an SNMP bug caused a buffer-overwrite condition which happened to overwrite the EIGRP process memory allocation, you are now affecting not only affecting the control plane, but also your forwarding plane, thus crashing the router and causing an outage.  In this same use case, if the SNMP process is being walked by a monitoring server while the EIGRP process needs to update the RIB, this process could be slowed down dramatically until the process finishes.

The original OS design worked without too many issues, as there weren’t many platforms that Cisco supported.  Each IOS version had to be written specifically for each platform to support the drivers and features necessary for the platform to function.  As Cisco acquired more companies, and customers requested additional features and functionality, maintaining the plethora of IOS trains became unmanageable.

Flexibility to deploy a new feature to market also became a problem.  Any new feature required a revision of the entire IOS binary rather than just making it an install-able package that utilized the underlying kernel for functionality.

IOS XE Features and Improvements

Cisco’s IOS XE resolves these issues from its predecessor through multiple changes.  The first main hurdle was to create a base OS image which would not only provide the capability of splitting up the forwarding and control plane, but it would also provide a standards based approach for installing any increased functionality in a timely fashion in the future.  Linux was determined to be the underlying platform of choice to create this capability.  Linux is now the underlying common infrastructure which makes the previous IOS features just software addons to the underlying OS.  Drivers could now be written with knowledgeable programmers from the Linux community which decreased hardware release schedules.  IOS was split up into the following packages:

1. RPBase – Provides the operating system software for the route processor
2. RPControl – Provides the control-plane processes that interface between Cisco IOS Software and the rest of the platform
3. RPIOS – Provides the Cisco IOS Software kernel, which is where Cisco IOS Software features are stored and run; each consolidated image variant has a different RPIOS sub-package. This has recently come back to a single image standard, with licensing controlling the features available and not the IOS image in this package.
4. RPAccess – Provides components to manage enhanced router access functionality (SSH, SNMP, HTTP as examples)
5. SIPBase – Share interface processor (SIP) carrier card operating system and control processes
6. SIPSPA – Provides the shared port adapter (SPA) driver and associated field-programmable device (FPD) images
7. ESPBase – Provides the ESP operating system and control processes for the ESP software

Each of these can be restarted and upgraded as needed without causing the data plane to restart.  IOS XE contains two functions which finalize the separation of the control and data plane.  The Forwarding and Feature Manager (FFM) provides an API to the control plane processes and translates those instructions into meaningful changes in the control plane.  The FFM then utilizes the Forwarding Engine Driver (FED) to update the data plane to reflect the desired changes from the original API call to the FFM.

Application and process state have also been moved from memory to a database structure.  This allows for the process data to be synchronized more easily across processes since memory does not have to be shared nor do inter-process calls need to be executed.  Each process requiring the information can call to the centralized database and see the information stored within.  If a process must restart, it can be done so without losing the state of the crashed process.  The information can be reintroduced after the restart, thus increasing the high availability of the system.

Cisco Live 2018 – IOS XE Architecture for Programmability – BRKSDN-2666 – Jeff McLaughlin

IOS XE also provides the capability of running hosted applications in either an LXC or VM configuration.   Since IOS XE is just a Linux machine with software packages running on top of it, any Linux supported application can be implemented with little development effort.

Programmability is also improved upon with this structure.  With each feature set being implemented as its own process, and a database storing its state in a standardized format, API calls can be created for readily update configurations and report on the status of a process through standards based data exporting functionalities.