Build DSCP to ToS conversion table with Python

by Przemek

Contents

Introduction

In this post we're going to write Python program that generates DSCP to ToS conversion table while avoiding hardcoding values as much as possible. We will then save the final table to csv file with pre-defined column headers.

I got the idea for this blog article from the tweet posted the other day by Nick Russo. I thought it is an interesting problem to tackle as similar ones pop up all the time during early stages of Network Automation journey. What makes this challenge great is that it requires us to carry out tasks that apply to writing larger programs.

  • We need to understand the problem and possibly do some research.
  • We have to come up with plan of action.
  • We need to break down larger tasks into smaller pieces.
  • We need to implement all of the pieces we identified.
  • Finally we have to put pieces back together and make sure final product works.

Before you continue reading this post I encourage you to try to implement solution to this problem yourself. Programming really is one of those skills that you learn best by doing. Once you've got something in place you can come back and see how I tackled this.

Full disclosure, I posted my hastily put together initial solution on Twitter but I made a lot of tweaks since then. It was lunch time and your first solution will rarely be the final one :) You can find it on Github, with few revisions due to several mistakes, and see how it evolved into solution I'm presenting here.

Problem description.

Our goal is to produce DSCP to ToS conversion table, additionally we need to provide multiple representations and bit values for each DSCP and ToS code. The end result should resemble the following CSV:

Discovery and research phase

There is quite a lot going on here but few patterns become apparent quite quickly.

We can see that some columns show the same value with the only change being number base. DSCP and ToS codes are shown in binary, decimal and hexadecimal bases. This is something we should be able to tackle even without understanding much about meaning of DSCP and ToS codes.

Looking at the entries it also appears that ToS = DSCP x 4. But we should not take this for granted, we should do research and refer to documents defining DSCP and ToS. Humans have tendency to see patterns everywhere and this can sometimes make us come up with good solution to a wrong problem.

With that being said I searched for RFC that can help us with understanding what DSCP and ToS are.

Below ToS definition is taken from RFC795:

The IP Type of Service has the following fields:

   Bits 0-2:  Precedence.
   Bit    3:  0 = Normal Delay,      1 = Low Delay.
   Bits   4:  0 = Normal Throughput, 1 = High Throughput.
   Bits   5:  0 = Normal Relibility, 1 = High Relibility.
   Bit  6-7:  Reserved for Future Use.

      0     1     2     3     4     5     6     7
   +-----+-----+-----+-----+-----+-----+-----+-----+
   |                 |     |     |     |     |     |
   |   PRECEDENCE    |  D  |  T  |  R  |  0  |  0  |
   |                 |     |     |     |     |     |
   +-----+-----+-----+-----+-----+-----+-----+-----+

   111 - Network Control
   110 - Internetwork Control
   101 - CRITIC/ECP
   100 - Flash Override
   011 - Flash
   010 - Immediate
   001 - Priority
   000 - Routine

You might have noticed that some items here match elements in our table. Precedence bits 0-2 match binary and decimal ToS Precedence columns as well as the ToS String Format. Bits 3, 4 and 5 match ToS Delay, Throughput and Reliability.

So we figured quite a lot already by noticing that same value is represented in different base number bases and then by looking at one RFC describing ToS.

We now need to figure out DSCP class codepoints (1st column) and understand how does DSCP map to ToS.

And to do that we'll look at more RFCs.

RFC2474 says this about DSCP:

   A replacement header field, called the DS field, is defined, which is
   intended to supersede the existing definitions of the IPv4 TOS octet
   [RFC791] and the IPv6 Traffic Class octet [IPv6].

   Six bits of the DS field are used as a codepoint (DSCP) to select the
   PHB a packet experiences at each node.  A two-bit currently unused
   (CU) field is reserved and its definition and interpretation are
   outside the scope of this document.  The value of the CU bits are
   ignored by differentiated services-compliant nodes when determining
   the per-hop behavior to apply to a received packet.

   The DS field structure is presented below:


        0   1   2   3   4   5   6   7
      +---+---+---+---+---+---+---+---+
      |         DSCP          |  CU   |
      +---+---+---+---+---+---+---+---+

        DSCP: differentiated services codepoint
        CU:   currently unused

Aha! So we now know that ToS uses 8 bits and DSCP uses just first 6 bits of the same byte in the IP header. From that follows that when converting from DSCP to ToS we will have to shift our number by two bits to the left.

But what does that mean? Well, each bit we shift our value by equals to multiplying by 2. In example below we shift binary number 0010 (decimal 2) to the left by 2 bits with resulting number being binary 1000 (decimal 8).

bin 0010 - dec 2

shift by 2 bits to the left

bin 1000 - dec 8

Hopefully now you can see that our initial hunch was correct, ToS = DSCP x 4, but now you also know why, which is much more important.

Great, so we have pretty much all pieces of the puzzle now. Let's find out what are the names of DSCP codepoints.

RFC4594 has this to say on the topic:

    ------------------------------------------------------------------
   |   Service     |  DSCP   |    DSCP     |       Application        |
   |  Class Name   |  Name   |    Value    |        Examples          |
   |===============+=========+=============+==========================|
   |Network Control|  CS6    |   110000    | Network routing          |
   |---------------+---------+-------------+--------------------------|
   | Telephony     |   EF    |   101110    | IP Telephony bearer      |
   |---------------+---------+-------------+--------------------------|
   |  Signaling    |  CS5    |   101000    | IP Telephony signaling   |
   |---------------+---------+-------------+--------------------------|
   | Multimedia    |AF41,AF42|100010,100100|   H.323/V2 video         |
   | Conferencing  |  AF43   |   100110    |  conferencing (adaptive) |
   |---------------+---------+-------------+--------------------------|
   |  Real-Time    |  CS4    |   100000    | Video conferencing and   |
   |  Interactive  |         |             | Interactive gaming       |
   |---------------+---------+-------------+--------------------------|
   | Multimedia    |AF31,AF32|011010,011100| Streaming video and      |
   | Streaming     |  AF33   |   011110    |   audio on demand        |
   |---------------+---------+-------------+--------------------------|
   |Broadcast Video|  CS3    |   011000    |Broadcast TV & live events|
   |---------------+---------+-------------+--------------------------|
   | Low-Latency   |AF21,AF22|010010,010100|Client/server transactions|
   |   Data        |  AF23   |   010110    | Web-based ordering       |
   |---------------+---------+-------------+--------------------------|
   |     OAM       |  CS2    |   010000    |         OAM&P            |
   |---------------+---------+-------------+--------------------------|
   |High-Throughput|AF11,AF12|001010,001100|  Store and forward       |
   |    Data       |  AF13   |   001110    |     applications         |
   |---------------+---------+-------------+--------------------------|
   |    Standard   | DF (CS0)|   000000    | Undifferentiated         |
   |               |         |             | applications             |
   |---------------+---------+-------------+--------------------------|
   | Low-Priority  |  CS1    |   001000    | Any flow that has no BW  |
   |     Data      |         |             | assurance                |
    ------------------------------------------------------------------

                Figure 3. DSCP to Service Class Mapping

Just what we needed! All of the DSCP names and corresponding values in one place.

This has been great but notice that we didn't write a single line of code! And this is how it should be. Often you will feel like jumping right in and getting those Python statements flowing. I would however advise you to spend some time to understand the problem first, you might find out that once you have better grasp of the task it will make it easier to write your code.

With that said, I think we're ready to get our hands dirty :)

Plan of action

I decided to split this problem by treating value generation for each column as a separate task to implement. I'm also separating generation of values from rendering of the whole table and from writing the table to the CSV file.

Here's my high level plan of action.

  • Decide on what are the initial values.
  • Generate each of the resulting values from initial values.
  • Put values together to form table row.
  • Generate final conversion table.
  • Write conversion table to CSV file.

Initial values

I'm going to start with choosing initial values from which we will derive all of the other items. Natural choice here will be DSCP or ToS codes. You can't go wrong with either but I'll pick DSCP because numbers are smaller and they look more familiar to me.

Looking at DSCP codes in the table from the beginning of the post we have 0, then jump to 8, and then numbers increase by 2 all the way to 40. So that's a nice pattern. Finally we have numbers 46, 48 and 56.

From that we could define a list and type all of the numbers by hand, like so:

dscps_dec = [0, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 46, 48, 56]

This is a bit verbose though and our original problem asked to use logic where possible, within reason as we want this code to be easy to understand.

Below is what I ended up doing:

dscps_dec = (0, *range(8, 41, 2), 46, 48, 56)

I created a tuple, with 0, 46, 48, 56 typed manually and the numbers between 8 and 40 are unpacked from sequence returned by range().

DSCP to ToS conversion and different number bases

Now that we have our DSCP values we can start looking into generating remaining elements.

First let's get on with DSCP to ToS conversion and representations in different number bases. This should be a good warm up before more involving tasks.

For converting to binary and hexadecimal we'll employ Python's string format() method:

"{:06b}".format(dscp)
"{:#04x}".format(dscp)

And a little test:

>>> dscp = 22
>>> print("{:06b}".format(dscp))
010110
>>> print("{:#04x}".format(dscp))
0x16

That's looking good. Let's look more closely at what's happening here.

  • "{:06b}" - b means take provided number and display it in binary base, 6 makes field 6 characters wide and 0 adds 0s in the front if the resulting number has less than 6 digits.

  • {:#04x} - x means display number in hexadecimal base, 4 makes field 4 characters wide, 0 prepends 0s if needed and # asks for display of base indicator 0x.

Not too bad. We've got 3 columns sorted out, now we'll tackle ToS numbers.

We already found out that we need to shift DSCP number by two bits to arrive at ToS value. We could multiply DSCP by 4 but to emphasize our intent here we'll use Python's bitwise operator <<.

tos_dec = dscp << 2

And some tests:

>>> dscp = 16
>>> tos_dec = dscp << 2
>>> tos_dec
64
>>> dscp = 28
>>> tos_dec = dscp << 2
>>> tos_dec
112

With that in hand we can now get bin and hex values, we'll use similar string formatting as we did for DSCP but we'll make binary string field 8 characters wide as ToS values use all 8 bits.

"{:#04x}".format(tos_dec)
"{:08b}".format(tos_dec)

>>> tos_dec = 112
>>> print("{:08b}".format(tos_dec))
01110000
>>> print("{:#04x}".format(tos_dec))
0x70

Perfect, two more columns ticked off.

ToS Precedence, Delay, Throughput and Reliability

Let's move onto ToS Precedence and Delay, Throughput and Reliability bits. We know that Precedence is encoded by bits 0-2. And we just computed binary number so perhaps we can reuse that?

Here's what I did:

tos_bin = "{:08b}".format(tos_dec)
tos_prec_bin = tos_bin[:3]

And a little test:

>>> tos_dec = 112
>>> tos_bin = "{:08b}".format(tos_dec)
>>> tos_prec_bin = tos_bin[:3]
>>> tos_prec_bin
'011'

Since we'll be using binary ToS value on it's own and to help us get ToS Precedence value, I assigned result of binary base conversion to a variable. That variable now holds a reference to a string from which we need first 3 digits for ToS Precedence. We can use list slicing here, and assign the slice to a new variable.

We got ToS precedence in binary but we also need it in decimal form. We can take advantage of built-in int() function that can take string representation of number with optional base, and will give use back decimal.

int(tos_prec_bin, 2)

We feed it result from our previous assignment of tos_prec_bin:

>>> int(tos_prec_bin, 2)
3

Hey, it works! This is going quite well so far.

We still got 3 single bits to work out. We can see in RFC795 that we need to check if bit at given position is set or not. We will also need to check bits in 3 different places, so it makes sense to encapsulate this logic in the separate function.

To check if bit is set we should look at bitwise AND which in Python is done with & operator.

Ok, but what do we AND with what? On one side we will have our ToS number but we need to figure the other side.

Looking again at the diagram taken from RFC we see that bits are counted from left to right, starting with bit 0.

      0     1     2     3     4     5     6     7
   +-----+-----+-----+-----+-----+-----+-----+-----+
   |                 |     |     |     |     |     |
   |   PRECEDENCE    |  D  |  T  |  R  |  0  |  0  |
   |                 |     |     |     |     |     |
   +-----+-----+-----+-----+-----+-----+-----+-----+

To check bits 3, 4 and 5 we could use corresponding binary numbers:

0b00010000
0b00001000
0b00000100

If we apply bitwise AND to first binary number 0b00010000 and whatever value we pass, we will get 0b00010000 only if passed number has bit 3 set to 1. Otherwise end result will be 0.

Here's an example of how that would work:

>>> tos_bin = 0b0011_0000
>>> bw_and_res = tos_bin & 0b0001_0000
>>> "{:#010b}".format(bw_and_res)
'0b00010000'

Hopefully now you can see that bitwise AND operator can help us check if given bit is set or not.

There's one slight problem though, we want to get back number 1 if bit is set or 0 if it is not. Right now we get back 8-bit binary number. We'll now fix it and encapsulate the logic in the function named kth_bit8_val.

def kth_bit8_val(byte, k):
    """
    Returns value of k-th bit

    Most Significant Bit, bit-0, on the Left

    :param byte: 8 bit integer to check
    :param k: bit value to return
    :return: 1 if bit is set, 0 if not
    """
    return 1 if byte & (0b1000_0000 >> k) else 0

Our function is essentially one liner that does the following:

  • Takes provided number in byte argument and k-th bit to check in k argument.

  • Shifts binary number 0b10000000 to right by k bits with (0b1000_0000 >> k). This prepares our mask for bitwise AND operation. If we wanted to check bit 3 we'd get 0b00010000. Remember we count from 0.

  • With mask in place we can AND our number with the mask (byte & (0b1000_0000 >> k). Result will be either binary number 0 or binary number equal to our mask if the required bit is set, e.g. 0b00010000.

  • Because we don't want the 8-bit binary number but just 1 or 0, we write short if..else statement that returns 0 if binary number is 0 or 1 if it's anything else.

Here's an example of our function in action:

>>> tos_val = 152
>>> kth_bit8_val(tos_val, 3)
1
>>> kth_bit8_val(tos_val, 4)
1
>>> kth_bit8_val(tos_val, 5)
0

With our function in place we can now get values for ToS Delay, Throughput and Reliability bits:

kth_bit8_val(tos_dec, 3)
kth_bit8_val(tos_dec, 4)
kth_bit8_val(tos_dec, 5)

Let's try these out:

>>> tos_dec = 88
>>> kth_bit8_val(tos_dec, 3)
1
>>> kth_bit8_val(tos_dec, 4)
1
>>> kth_bit8_val(tos_dec, 5)
0
>>> f"{tos_dec:08b}"
'01011000'

As you can see we got correct values for bits 3, 4 and 5. Another task completed :).

ToS string formats

We now only have 2 tasks left, getting DSCP class and ToS string format. These will require some manual typing but we can still have decent amount of logic.

First we'll get ToS string format out of the way.

If you look at RFC795 excerpt again you should see that ToS strings depend on the value of ToS Precedence field.

      0     1     2     3     4     5     6     7
   +-----+-----+-----+-----+-----+-----+-----+-----+
   |                 |     |     |     |     |     |
   |   PRECEDENCE    |  D  |  T  |  R  |  0  |  0  |
   |                 |     |     |     |     |     |
   +-----+-----+-----+-----+-----+-----+-----+-----+

   111 - Network Control
   110 - Internetwork Control
   101 - CRITIC/ECP
   100 - Flash Override
   011 - Flash
   010 - Immediate
   001 - Priority
   000 - Routine

And we already have that, we used int(tos_prec_bin, 2) before to get value of ToS Precedence field in decimal. Let's assign this to a variable.

Next we'll create a dictionary with keys being all possible ToS numbers and values in dictionary being corresponding strings.

tos_prec_dec = int(tos_prec_bin, 2)

TOS_STRING_TBL = {
    0: "Routine",
    1: "Priority",
    2: "Immediate",
    3: "Flash",
    4: "FlashOverride",
    5: "Critical",
    6: "Internetwork Control",
    7: "Network Control",
}

With tos_prec_dec variable in place we can use its value to get from dictionary corresponding string format.

TOS_STRING_TBL[tos_prec_dec]

Example use:

>>> tos_prec_bin = "110"
>>> tos_prec_dec = int(tos_prec_bin, 2)
>>> TOS_STRING_TBL[tos_prec_dec]
'Internetwork Control'
>>> tos_prec_bin = "010"
>>> tos_prec_dec = int(tos_prec_bin, 2)
>>> TOS_STRING_TBL[tos_prec_dec]
'Immediate'

That seems to be working fine, awesome! Only one more task left. Getting DSCP class.

DSCP classes

Let's have a look again at what RFC2579 and RFC4594 say about DSCP classes.

RFC2597:


   The RECOMMENDED values of the AF codepoints are as follows: AF11 = '
   001010', AF12 = '001100', AF13 = '001110', AF21 = '010010', AF22 = '
   010100', AF23 = '010110', AF31 = '011010', AF32 = '011100', AF33 = '
   011110', AF41 = '100010', AF42 = '100100', and AF43 = '100110'.  The
   table below summarizes the recommended AF codepoint values.

                        Class 1    Class 2    Class 3    Class 4
                      +----------+----------+----------+----------+
     Low Drop Prec    |  001010  |  010010  |  011010  |  100010  |
     Medium Drop Prec |  001100  |  010100  |  011100  |  100100  |
     High Drop Prec   |  001110  |  010110  |  011110  |  100110  |
                      +----------+----------+----------+----------+

RFC4594 has this to say on the topic:

    ------------------------------------------------------------------
   |   Service     |  DSCP   |    DSCP     |       Application        |
   |  Class Name   |  Name   |    Value    |        Examples          |
   |===============+=========+=============+==========================|
   |Network Control|  CS6    |   110000    | Network routing          |
   |---------------+---------+-------------+--------------------------|
   | Telephony     |   EF    |   101110    | IP Telephony bearer      |
   |---------------+---------+-------------+--------------------------|
   |  Signaling    |  CS5    |   101000    | IP Telephony signaling   |
   |---------------+---------+-------------+--------------------------|
   | Multimedia    |AF41,AF42|100010,100100|   H.323/V2 video         |
   | Conferencing  |  AF43   |   100110    |  conferencing (adaptive) |
   |---------------+---------+-------------+--------------------------|
   |  Real-Time    |  CS4    |   100000    | Video conferencing and   |
   |  Interactive  |         |             | Interactive gaming       |
   |---------------+---------+-------------+--------------------------|
   | Multimedia    |AF31,AF32|011010,011100| Streaming video and      |
   | Streaming     |  AF33   |   011110    |   audio on demand        |
   |---------------+---------+-------------+--------------------------|
   |Broadcast Video|  CS3    |   011000    |Broadcast TV & live events|
   |---------------+---------+-------------+--------------------------|
   | Low-Latency   |AF21,AF22|010010,010100|Client/server transactions|
   |   Data        |  AF23   |   010110    | Web-based ordering       |
   |---------------+---------+-------------+--------------------------|
   |     OAM       |  CS2    |   010000    |         OAM&P            |
   |---------------+---------+-------------+--------------------------|
   |High-Throughput|AF11,AF12|001010,001100|  Store and forward       |
   |    Data       |  AF13   |   001110    |     applications         |
   |---------------+---------+-------------+--------------------------|
   |    Standard   | DF (CS0)|   000000    | Undifferentiated         |
   |               |         |             | applications             |
   |---------------+---------+-------------+--------------------------|
   | Low-Priority  |  CS1    |   001000    | Any flow that has no BW  |
   |     Data      |         |             | assurance                |
    ------------------------------------------------------------------

                Figure 3. DSCP to Service Class Mapping

So it looks like we can build our logic by looking at first 3 bits and then 2 bits after that. Last bit is always set to 0 so we can ignore it.

Now, if you remember, bits 0-2 correspond to ToS Precedence value, and we already worked that out. Bits 3 and 4 are ToS Delay and Throughput bits, which we know already as well.

It looks like we can pass the values we previously computed to a function and then inside of the function we'll work out the DSCP name.

Below is what I came up with:

def dscp_class(bits_0_2, bit_3, bit_4):
    """
    Takes values of DSCP bits and computes dscp class

    Bits 0-2 decide major class
    Bit 3-4 decide drop precedence

    :param bits_0_2: int: decimal value of bits 0-2
    :param bit_3: int: value of bit 3
    :param bit_4: int: value of bit 4
    :return: DSCP class name
    """
    bits_3_4 = (bit_3 << 1) + bit_4
    if bits_3_4 == 0:
        dscp_cl = "cs{}".format(bits_0_2)
    elif (bits_0_2, bits_3_4) == (5, 3):
        dscp_cl = "ef"
    else:
        dscp_cl = "af{}{}".format(bits_0_2, bits_3_4)

    return dscp_cl

We take 3 arguments since we will have those available from earlier computations. So we get separately bits 0-2, bit 3 and bit 4.

First we compute decimal value held by bits 3 and 4 as this decides precedence drop.

bits_3_4 = (bit_3 << 1) + bit_4

With that in hand we can start building if..else logic.

We know that if bits 3 and 4 are equal to 0 we're dealing with cs class. The actual number of the cs class is decided by bits 0-2, and we know value of those. So we can insert that value into the string:

if bits_3_4 == 0:
    dscp_cl = "cs{}".format(bits_0_2)

Next off we're dealing with an exception, ef class. Here we need to check if bits 0-2 equal to 5 and bits 3-4 equal to 3 as per RFC table:

| Telephony     |   EF    |   101110    | IP Telephony bearer      |

elif (bits_0_2, bits_3_4) == (5, 3):
    dscp_cl = "ef"

Finally to build string for af class we append value held in bits 0-2 followed by value held in bits 3-4:

else:
    dscp_cl = "af{}{}".format(bits_0_2, bits_3_4)

Let's see the completed function in action:

>>> tos_prec = 0b001
>>> tos_del = 0
>>> tos_thr = 0
>>> dscp_class(tos_prec, tos_del, tos_thr)
'cs1'
>>> tos_prec = 0b010
>>> tos_del = 1
>>> tos_thr = 1
>>> dscp_class(tos_prec, tos_del, tos_thr)
'af23'
>>> tos_prec = 0b101
>>> tos_del = 1
>>> tos_thr = 1
>>> dscp_class(tos_prec, tos_del, tos_thr)
'ef'

Things are looking good indeed.

And that's it, we've got all of the components in place. Now we need to put them together.

Generating conversion table row

I decided to move building of all of the values for given DSCP code into its own function. It's easier for me to reason about this code, as well as test it later, when it is separated. I named this function dscp_conv_tbl_row:

def dscp_conv_tbl_row(dscp):
    """
    Generates DSCP to ToS conversion values as well as different representations

    :param dscp: int: decimal DSCP code
    :return: dict with each value assigned to value name
    """
    tos_dec = dscp << 2
    tos_bin = "{:08b}".format(tos_dec)
    tos_prec_bin = tos_bin[0:3]
    tos_prec_dec = int(tos_prec_bin, 2)
    tos_del_fl = kth_bit8_val(tos_dec, 3)
    tos_thr_fl = kth_bit8_val(tos_dec, 4)
    tos_rel_fl = kth_bit8_val(tos_dec, 5)
    dscp_cl = dscp_class(tos_prec_dec, tos_del_fl, tos_thr_fl)

    tbl_row_vals = (
        dscp_cl,
        "{:06b}".format(dscp),
        "{:#04x}".format(dscp),
        dscp,
        tos_dec,
        "{:#04x}".format(tos_dec),
        tos_bin,
        tos_prec_bin,
        tos_prec_dec,
        tos_del_fl,
        tos_thr_fl,
        tos_rel_fl,
        TOS_STRING_TBL[tos_prec_dec],
    )

    return tbl_row_vals

We've already seen most of this code, it's just now put together. We're taking DSCP value and computing values for different components that we need in our conversion table. We finally return all of the items in a tuple.

This is how it looks like when we run it:

>>> dscp_conv_tbl_row(28)
('af32', '011100', '0x1c', 28, 112, '0x70', '01110000', '011', 3, 1, 0, 0, 'Flash')

All of the values for one table row. Now we need to write code that computes these tuples for each of DSCP codes we require. Our end goal is to have full table that is saved to CSV. This is a good place to think about how to store the resulting rows.

Building final table

We know we want to save final product in CSV format so generating one big string is probably not a good idea. Either list of dictionary would work best here.

Python comes with built-in library for working with CSV files, named appropriately enough csv. We can use either of its methods writer() or DictWriter() to write our data to file. I opted for DictWriter() as this allows me to automatically map my data structure onto columns in resulting file. I suggest you try out both and see what works better for you.

Now that I know how I want to write to CSV file I know that my rows will be recorded as list of dictionaries. Time to write function that encapsulates this logic:

def gen_dscp_conversion_table():
    """
    Generates DSCP to TOS conversion table with rows for selected DSCP codes

    Final table is a list of dictionaries to make writing CSV easier

    :return: list(dict): final DSCP to TOS conversion table
    """
    column_names = (
        "DSCP Class",
        "DSCP (bin)",
        "DSCP (hex)",
        "DSCP (dec)",
        "ToS (dec)",
        "ToS (hex)",
        "ToS (bin)",
        "ToS Prec. (bin)",
        "ToS Prec. (dec)",
        "ToS Delay Flag",
        "ToS Throughput Flag",
        "ToS Reliability Flag",
        "TOS String Format",
    )
    # These are the DSCP codes we're interested in
    dscps_dec = (0, *range(8, 41, 2), 46, 48, 56)

    conv_tbl = [dict(zip(column_names, dscp_conv_tbl_row(dscp))) for dscp in dscps_dec]

    return conv_tbl

As you can see we first define tuple with column names, these match what will end up in the the CSV file.

Then we create tuple with DSCP values and finally we create list comprehension inside of which we create our dictionaries.

This list comprehension is composed of few elements so I'm going to break it down for you.

  • for dscp in dscps_dec - provides decimal dscp values for which we generate table rows.

  • dscp_conv_tbl_row(dscp) - returns tuple with all values for our table row for given dscp.

  • zip(column_names, dscp_conv_tbl_row(dscp)) - goes over both column names and table row values and pairs 1st item from one with 1st item from second, then it moves to 2nd pair and so on. Result is a sequence of tuples in (column_name, dscp_tbl_row) format.

  • dict(zip(column_names, dscp_conv_tbl_row(dscp))) - takes sequence of tuples and turns them into dictionary with 1st element being turned into key and 2nd element becoming value.

The end result of this list comprehension is a list of dictionaries, each dictionary having column names as keys with corresponding values mapped to them.

Writing table to CSV file

We're almost there. All that's left for us to do is to write fruits of our hard labor into the file.

def main():
    dscp_conversion_table = gen_dscp_conversion_table()

    with Path("dscp_tos_conv_table.csv").open(mode="w", newline="") as fout:
        dict_writer = csv.DictWriter(f=fout, fieldnames=dscp_conversion_table[0].keys())

        dict_writer.writeheader()
        dict_writer.writerows(dscp_conversion_table)

  • We're generating the full table first. Then we open file we want to write to, we specify write mode and newline is set to "" as required by csv library.

  • Next we create csv.DictWriter() object giving it our file. We set fieldnames argument to keys from the first dictionary in our list. All dictionaries have the same keys so it doesn't matter which one we take.

  • Finally we ask DictWriter to write header to the file, and then we write to the same file all of the entries in one go. We gave DictWriter dictionary keys so it's able to map all of the row values automatically.

And with that we came to an end. Or have we?

Yes and no. We built program generating DSCP to ToS conversion table and we wrote the table to CSV file. But, do we know if this works correctly? Do we know if the values it produces are correct? And even if they are correct now, can we be certain that they will stay correct if we modify the program?

The answer is: we don't really know if any of this is correct. Maybe we checked the wrong bit somewhere or maybe there's a typo in a DSCP class name. Even a program that can fit on two pages of text can introduce a fair number of bugs.

Writing tests

You might already think what I'm thinking: testing. We need testing.

So testing we shall have.

In the course of writing our program we created 4 functions:

  • dscp_class
  • kth_bit8_val
  • dscp_conv_tbl_row
  • gen_dscp_conversion_table

I want to write tests for 3 of these but not gen_dscp_conversion_table. This one mostly relies on values generated by other 3 and I don't feel like we need to have coverage here.

So, how do we test?

We should look at each of these functions and think of values that will give us good confidence in our code being correct. For example dscp_class has if..else statement so it would be a good idea to test it with values that will make code take all possible paths. On top of that we want to add few cases that follow the most common path.

For my tests I'm using pytest and pytest.mark.parametrize() decorator. This allows me to define multiple test cases and pass them as parameters to our test function. Without it we'd have to write multiple assert statements.

Testing dscp_class

For testing dscp_class I chose values that should result in csX, afXY and ef DSCP classes. This accounts for all branches of if..else statement. To make sure our logic is correct I got few different examples for each of the major classes.

Here are the final test values and test code that I came up with:

@pytest.mark.parametrize(
    "bits_0_2, bit_3, bit_4, res_class",
    [
        (0b000, 0, 0, "cs0"),
        (0b001, 0, 1, "af11"),
        (0b011, 0, 0, "cs3"),
        (0b011, 1, 0, "af32"),
        (0b100, 1, 1, "af43"),
        (0b101, 1, 1, "ef"),
        (0b111, 0, 0, "cs7"),
    ]
)
def test_dscp_class(bits_0_2, bit_3, bit_4, res_class):
    assert dscp_class(bits_0_2, bit_3, bit_4) == res_class

Testing kth_bit8_val

For kth_bit8_val I decided to choose just few values. Corner case here is checking first and last bits, so we need to make sure these are handled correctly. Also each bit can be either set or unset (0 or 1) so we test if both 0 and 1 bit values are picked up.

Here is our test code:

@pytest.mark.parametrize(
    "byte, k_bit, bit_val",
    [
        (0b1000_0000, 0, 1),
        (0b1110_1111, 3, 0),
        (0b0100_0110, 5, 1),
        (0b0000_0001, 7, 1),
    ]
)
def test_kth_bit8_val(byte, k_bit, bit_val):
    assert kth_bit8_val(byte, k_bit) == bit_val

Testing test_gen_table_row

Lastly, we have test_gen_table_row. Here I chose values for 7 different ToS strings as well as values corresponding to different DSCP classes. We also want to test if values are returned in the correct order. If my function passes all of these test I will be fairly confident that it is working correctly.

Testing code:

@pytest.mark.parametrize(
    "dscp_code, conv_row_values",
    [
        (0, ("cs0", "000000", "0x00", 0, 0, "0x00", "00000000", "000", 0, 0, 0, 0, "Routine")),
        (8, ("cs1", "001000", "0x08", 8, 32, "0x20", "00100000", "001", 1, 0, 0, 0, "Priority")),
        (12, ("af12", "001100", "0x0c", 12, 48, "0x30", "00110000", "001", 1, 1, 0, 0, "Priority")),
        (22, ("af23", "010110", "0x16", 22, 88, "0x58", "01011000", "010", 2, 1, 1, 0, "Immediate")),
        (26, ("af31", "011010", "0x1a", 26, 104, "0x68", "01101000", "011", 3, 0, 1, 0, "Flash")),
        (40, ("cs5", "101000", "0x28", 40, 160, "0xa0", "10100000", "101", 5, 0, 0, 0, "Critical")),
        (46, ("ef", "101110", "0x2e", 46, 184, "0xb8", "10111000", "101", 5, 1, 1, 0, "Critical")),
        (48, ("cs6", "110000", "0x30", 48, 192, "0xc0", "11000000", "110", 6, 0, 0, 0, "Internetwork Control")),
        (56, ("cs7", "111000", "0x38", 56, 224, "0xe0", "11100000", "111", 7, 0, 0, 0, "Network Control")),
    ]
)
def test_gen_table_row(dscp_code, conv_row_values):
    assert dscp_conv_tbl_row(dscp_code) == conv_row_values

Running tests

Right, we have test functions, we have test values, time to actually run these tests against our codebase:

F:\projects\dscp_tos_conv
(venv) λ pytest -v
=================== test session starts ====================== 
platform win32 -- Python 3.8.2, pytest-6.0.1, py-1.9.0, \
pluggy-0.13.1 -- f:\projects\dscp_tos_conv\venv\scripts\python.exe
cachedir: .pytest_cache
rootdir: F:\projects\dscp_tos_conv
collected 20 items

test_dscp_tos_table.py::test_dscp_class[0-0-0-cs0] PASSED
[  5%] test_dscp_tos_table.py::test_dscp_class[1-0-1-af11] PASSED
[ 10%] test_dscp_tos_table.py::test_dscp_class[3-0-0-cs3] PASSED
[ 15%] test_dscp_tos_table.py::test_dscp_class[3-1-0-af32] PASSED
[ 20%] test_dscp_tos_table.py::test_dscp_class[4-1-1-af43] PASSED
[ 25%] test_dscp_tos_table.py::test_dscp_class[5-1-1-ef] PASSED
[ 30%] test_dscp_tos_table.py::test_dscp_class[7-0-0-cs7] PASSED
[ 35%] test_dscp_tos_table.py::test_kth_bit8_val[128-0-1] PASSED
[ 40%] test_dscp_tos_table.py::test_kth_bit8_val[239-3-0] PASSED
[ 45%] test_dscp_tos_table.py::test_kth_bit8_val[70-5-1] PASSED
[ 50%] test_dscp_tos_table.py::test_kth_bit8_val[1-7-1] PASSED
[ 55%] test_dscp_tos_table.py::test_gen_table_row[0-conv_row_values0] PASSED
[ 60%] test_dscp_tos_table.py::test_gen_table_row[8-conv_row_values1] PASSED
[ 65%] test_dscp_tos_table.py::test_gen_table_row[12-conv_row_values2] PASSED
[ 70%] test_dscp_tos_table.py::test_gen_table_row[22-conv_row_values3] PASSED
[ 75%] test_dscp_tos_table.py::test_gen_table_row[26-conv_row_values4] PASSED
[ 80%] test_dscp_tos_table.py::test_gen_table_row[40-conv_row_values5] PASSED
[ 85%] test_dscp_tos_table.py::test_gen_table_row[46-conv_row_values6] PASSED
[ 90%] test_dscp_tos_table.py::test_gen_table_row[48-conv_row_values7] PASSED
[ 95%] test_dscp_tos_table.py::test_gen_table_row[56-conv_row_values8] PASSED
[100%]

=================== 20 passed in 0.07s ================================

Awesome, all of the tests passed!

You might think this looks great but I did make some mistakes in my code before showing you the final run, that's the whole idea of testing, to give you confidence that your code works they way you expect it to. And when it doesn't you go back and fix it :)

Without tests you could miss bugs lurking somewhere so it really is a good idea to get used to creating them alongside your main program. Hopefully I showed you here that no code is too small to have a few tests thrown in for good measure, the future you will be thankful!

Closing thoughts

I had fun working on this challenge. It really is a good use case for problem decomposition, learning string formatting and bit-wise operators. And despite not looking like a tough one at first it did require me to dig a bit deeper and make sure that I understand the problem I'm trying to solve.

Finally after making few silly mistakes during refactoring I was reminded that testing is rarely, if ever, optional. You can get away with not writing tests for long time but sooner or later you'll wish you'd learned it earlier!

I hope you learned something new and enjoyed this post as much as I did writing it :)

References