Jinja2 Tutorial - Part 4 - Template filters

by Przemek

This is part 4 of Jinja2 tutorial where we continue looking at the language features, specifically we'll be discussing template filters. We'll see what filters are and how we can use them in our templates. I'll also show you how you can write your own custom filters.

Jinja2 Tutorial series

Contents

Overview of Jinja2 filters

Let's jump straight in. Jinja2 filter is something we use to transform data held in variables. We apply filters by placing pipe symbol | after the variable followed by name of the filter.

Filters can change the look and format of the source data, or even generate new data derived from the input values. What's important is that the original data is replaced by the result of transformations and that's what ends up in rendered templates.

Here's an example showing a simple filter in action:

Template:

First name: {{ first_name | capitalize }}

Data:

first_name: przemek

Result:

First name: Przemek

We passed first_name variable to capitalize filter. As the name of the filter suggests, string held by variable will end up capitalized. And this is exactly what we can see happened. Pretty cool, right?

It might help to think of filters as functions that take Jinja2 variable as argument, the only difference to standard Python functions is syntax that we use.

Python equivalent of capitalize would look like this:

def capitalize(word):
    return word.capitalize()

first_name = "przemek"

print("First name: {}".format(capitalize(first_name)))

Great, you say. But how did I know that capitalize is a filter? Where did it come from?

There's no magic here. Someone had to code all of those filters and make them available to us. Jinja2 comes with a number of useful filters, capitalize is one of them.

All of the built-in filters are documented in official Jinja2 docs. I'm including link in references and later in this post I'll show examples of some of the more useful, in my opinion, filters.

Multiple arguments

We're not limited to simple filters like capitalize. Some filters can take extra arguments in parentheses. These can be either keyword or positional arguments.

Below is an example of a filter taking extra argument.

Template:

ip name-server {{ name_servers | join(" ") }}

Data:

name_servers:
  - 1.1.1.1
  - 8.8.8.8
  - 9.9.9.9
  - 8.8.4.4

Result:

ip name-server 1.1.1.1 8.8.8.8 9.9.9.9 8.8.4.4

Filter join took list stored in name_servers and created a string by gluing together elements of the list with space as a separator. Separator is the argument we supplied in parenthesis and we could use different one depending on our needs.

You should refer to documentation to find out what arguments, if any, are available for given filter. Most filters use reasonable defaults and don't require all of the arguments to be explicitly specified.

Chaining filters

We've seen basic filter usage but we can do more. We can chain filters together. This means that multiple filters can be used at once, each separated by pipe |.

Jinja applies chained filters from left to right. Value that comes out of leftmost filter is fed into the next one, and the process is repeated until there are no more filters. Only the final result will end up in rendered template.

Let's have a look at how it works.

Data:

scraped_acl:
  - "   10 permit ip 10.0.0.0/24 10.1.0.0/24"
  - "   20 deny ip any any"

Template

{{ scraped_acl | first | trim }}

Result

10 permit ip 10.0.0.0/24 10.1.0.0/24

We passed list containing two items to first filter. This returned first element from the list and handed it over to trim filter which removed leading spaces.

The end result is line 10 permit ip 10.0.0.0/24 10.1.0.0/24.

Filter chaining is a powerful feature that allows us to perform multiple transformations in one go. The alternative would be to store intermediate results which would decrease readability and wouldn't be as elegant.

Additional filters and custom filters

Great as they are, built-in filters are very generic and many use cases call for more specific ones. This is why automation frameworks like Ansible or Salt provide many extra filters that cover wide range of scenarios.

In these frameworks you will find filters that can transform IP objects, display data in YAML/Json, or even apply regex, just to name a few. In references you can find links to docs for filters available in each framework.

Finally, you can create new filters yourself! Jinja2 provides hooks for adding custom filters. These are just Python functions, so if you wrote Python function before you will be able to write your own filter as well!

Aforementioned automation frameworks also support custom filters and the process of writing them is similar to vanilla Jinja2. You again need to write a Python function and then documentation for given tool will show you steps needed to register your module as a filter.

Why use filters?

No tool is a good fit for every problem. And some tools are solutions in search of a problem. So, why use Jinja2 filters?

Jinja, like most of the templating languages, was created with web content in mind. While data is stored in the standardized format in the database, we often need to transform it when displaying documents to the users. This is where language like Jinja with its filters enables on the go modification to the way data is presented, without having to touch back-end. That's the selling point of filters.

Below is my personal view on why I think Jinja2 filters are a good addition to the language:

1. They allow non-programmers to perform simple data transformations.

This applies to vanilla filters as well as extra filters provided by automation frameworks. For example, network engineers know their IP addresses and they might want to operate on them in templates without having any programming knowledge. Filters to the rescue!

2. You get predictable results.

If you use generally available filters anyone with some Jinja2 experience will know what they do. This allows people to get up to speed when reviewing templates written by others.

3. Filters are well maintained and tested.

Built-in filters as well as filters provided by automation frameworks are widely used by a lot people. This gives you high confidence that they give correct results and don't have many bugs.

4. The best code is no code at all.

The moment you add data transformation operations to your program, or create a new filter, you become responsible for the code, forever. Any bugs, feature requests, and tests will come your way for the lifetime of the solution. Write as much stuff as you want when learning but use already available solutions in production, whenever possible.

When not to use filters?

Filters can be really powerful and save us a lot of time. But with great power comes great responsibility. Overuse filters and you can end up with templates that are difficult to understand and maintain.

You know those clever one liners that no one, including yourself, can understand few months down the line? It's very easy to get into those situations with chaining a lot of filters, especially ones accepting multiple arguments.

I use the below heuristics to help me decide if what I did is too complicated:

  • Is what I wrote at the limit of my understanding?
  • Do I feel like what I just wrote is really clever?
  • Did I use many chained filters in a way that didn't seem obvious at first?

If you answer yes to at least one of the above, you might be dealing with the case of too clever for your own good. It is possible that there's no good simpler solution for your use case, but chances are you need to do refactoring. If you're unsure if that's the case it's best to ask your colleagues or check with community.

To show you how bad things can get, here's an example of Jinja2 lines I wrote a few years ago. These use filters provided by Ansible and it got so complicated that I had to define intermediate variables.

Have a look at it and try to figure what does it do, and more importantly, how does it do it.

Template, cut down for brevity:

{% for p in ibgp %}
{%  set jq = "[?name=='" + p.port + "'].{ myip: ip, peer: peer }" %}
{%  set el = ports | json_query(jq) %}
{%  set peer_ip = hostvars[el.0.peer] | json_query('ports[*].ip') | ipaddr(el.0.myip) %}
...
{%  endfor %}

Example data used with the template:

ibgp:
  - { port: Ethernet1 }
  - { port: Ethernet2 }
..
ports:
  - { name: Ethernet1, ip: "10.0.12.1/24", speed: 1000full, desc: "vEOS-02

There's so much to unpack here. In the first line I assign query string to a variable as a workaround for character escaping issues. In line two I apply json_query filter with argument coming from variable in line one, the result is stored in another helper variable. Finally, in the line three I apply two chained filters json_query and ipaddr.

The end result of these three lines should be IP address of BGP peer found on given interface.

I'm sure you will agree with me that this is terrible. Does this solution get any ticks next to heuristics I mentioned earlier? Yes! Three of them! This is a prime candidate for refactoring.

There are generally two things we can do in cases such as this:

  • Pre-process data in the upper layer that calls rendering, e.g. Python, Ansible, etc.
  • Write a custom filter.
  • Revise the data model to see if it can be simplified.

In this case I went with option 2, I wrote my own filter, which luckily is the next topic on our list.

Writing your own filters

As I already mentioned, to write a custom filter you need to get your hands dirty and write some Python code. Have no fear however! If you ever wrote a function taking an argument you've got all it takes. That's right, we don't need to do anything too fancy, any regular Python function can become a filter. It just needs to take at least one argument and it must return something.

Here's an example of function that we will register with Jinja2 engine as a filter:

# hash_filter.py
import hashlib


def j2_hash_filter(value, hash_type="sha1"):
    """
    Example filter providing custom Jinja2 filter - hash

    Hash type defaults to 'sha1' if one is not specified

    :param value: value to be hashed
    :param hash_type: valid hash type
    :return: computed hash as a hexadecimal string
    """
    hash_func = getattr(hashlib, hash_type, None)

    if hash_func:
        computed_hash = hash_func(value.encode("utf-8")).hexdigest()
    else:
        raise AttributeError(
            "No hashing function named {hname}".format(hname=hash_type)
        )

    return computed_hash

In Python this is how we tell Jinja2 about our filter:

# hash_filter_test.py
import jinja2
from hash_filter import j2_hash_filter

env = jinja2.Environment()
env.filters["hash"] = j2_hash_filter

tmpl_string = """MD5 hash of '$3cr3tP44$': {{ '$3cr3tP44$' | hash('md5') }}"""

tmpl = env.from_string(tmpl_string)

print(tmpl.render())

Result of rendering:

MD5 hash of '$3cr3tP44$': ec362248c05ae421533dd86d86b6e5ff

Look at that! Our very own filter! It looks and feels just like built-in Jinja filters, right?

And what does it do? It exposes hashing functions from Python's hashlib library to allow for direct use of hashes in Jinja2 templates. Pretty neat if you ask me.

To put it in words, below are the steps needed to create custom filter:

  1. Create a function taking at least one argument, that returns a value. First argument is always the Jinja variable preceding | symbol. Subsequent arguments are provided in parentheses (...).
  2. Register the function with Jinja2 Environment. In Python insert your function into filters dictionary, which is an attribute of Environment object. Key name is what you want your filter to be called, here hash, and value is your function.
  3. You can now use your filter same as any other Jinja filter.

Fixing "too clever" solution with Ansible custom filter

We know how to write custom filters, so now I can show you how I replaced part of my template where I went too far with clever tricks.

Here is my custom filter in its fully glory:

# get_peer_info.py
import ipaddress


def get_peer_info(our_port_info, hostvars):
    peer_info = {"name": our_port_info["peer"]}
    our_net = ipaddress.IPv4Interface(our_port_info["ip"]).network
    peer_vars = hostvars[peer_info["name"]]

    for _, peer_port_info in peer_vars["ports"].items():
        if not peer_port_info["ip"]:
            continue
        peer_net_obj = ipaddress.IPv4Interface(peer_port_info["ip"])
        if our_net == peer_net_obj.network:
            peer_info["ip"] = peer_net_obj.ip
            break
    return peer_info


class FilterModule(object):

    def filters(self):
        return {
            'get_peer_info': get_peer_info
        }

First part is something you've seen before, it's a Python function taking two arguments and returning one value. Sure, it's longer than "clever" three-liner, but it's so much more readable.

There's more structure here, variables have meaningful names and I can tell what it's doing pretty much right away. More importantly, I know how it's doing it, process is broken down into many individual steps that are easy to follow.

Custom filters in Ansible

The second part of my solution is a bit different than vanilla Python example:

class FilterModule(object):

    def filters(self):
        return {
            'get_peer_info': get_peer_info
        }

This is how you tell Ansible that you want get_peer_info to be registered as a Jinja2 filter.

You create class named FilterModule with one method called filters. This method must return dictionary with your filters. Keys in dictionary are names of the filters and values are functions. I say filter(s) and not filter, because you can register multiple filters in one file. Optionally you can have one filter per file if you prefer.

Once all is done you need to drop your Python module in filter_plugins directory, which should be located in the root of your repository. With that in place you can use your filters in Ansible Playbooks as well as Jinja2 templates.

Below you can see structure of the directory where my playbook deploy_base.yml is located in relation to the get_peer_info.py module.

.
├── ansible.cfg
├── deploy_base.yml
├── filter_plugins
│   └── get_peer_info.py
├── group_vars
    ...
├── hosts
├── host_vars
    ...
└── roles
    └── base

Jinja2 Filters - Usage examples

All of the Jinja2 filters are well documented in officials docs but I felt that some of them could use some more examples. Below you will find my subjective selection with some comments and explanations.

batch

batch(value, linecount, fill_with=None) - Allows you to group list elements into multiple buckets, each containing up to n elements, where n is number we specify. Optionally we can also ask batch to pad bucket with default entries to make all of the buckets exactly n in length. Result is list of lists.

I find it handy for splitting items into groups of fixed size.

Template:

{% for i in  sflow_boxes|batch(2) %}
Sflow group{{ loop.index }}: {{ i | join(', ') }}
{% endfor %}

Data:

sflow_boxes:
 - 10.180.0.1
 - 10.180.0.2
 - 10.180.0.3
 - 10.180.0.4
 - 10.180.0.5

Result:

Sflow group1: 10.180.0.1, 10.180.0.2
Sflow group2: 10.180.0.3, 10.180.0.4
Sflow group3: 10.180.0.5

center

center(value, width=80)- Centers value in a field of given width by adding space padding. Handy when adding formatting to reporting.

Template:

{{ '-- Discovered hosts --' | center }}
{{ hosts | join('\n') }}

Data:

hosts:
 - 10.160.0.7
 - 10.160.0.9
 - 10.160.0.3

Result:

                             -- Discovered hosts --                             
10.160.0.7
10.160.0.9
10.160.0.15

default

default(value, default_value='', boolean=False) - Returns default value if passed variable is not specified. Useful for guarding against undefined variables. Can also be used for optional attribute that we want to set to sane value as a default.

In below example we place interfaces in their configured vlans, or if no vlan is specified we assign them to vlan 10 by default.

Template:

{% for intf in interfaces %}
interface {{ intf.name }}
  switchport mode access
  switchport access vlan {{ intf.vlan | default('10') }}
{% endfor %}

Data:

interfaces:
  - name: Ethernet1
    vlan: 50
  - name: Ethernet2
    vlan: 50
  - name: Ethernet3
  - name: Ethernet4

Result:

interface Ethernet1
  switchport mode access
  switchport access vlan 50
interface Ethernet2
  switchport mode access
  switchport access vlan 50
interface Ethernet3
  switchport mode access
  switchport access vlan 10
interface Ethernet4
  switchport mode access
  switchport access vlan 10

dictsort

dictsort(value, case_sensitive=False, by='key', reverse=False) - Allows us to sort dictionaries as they are not sorted by default in Python. Sorting is done by key by default but you can request sorting by value using attribute by='value'.

In below example we sort prefix-lists by their name (dict key):

Template:

{% for pl_name, pl_lines in prefix_lists | dictsort %}
ip prefix list {{ pl_name }}
  {{ pl_lines | join('\n') }}
{% endfor %}

Data:

prefix_lists:
  pl-ntt-out:
    - permit 10.0.0.0/23
  pl-zayo-out:
    - permit 10.0.1.0/24
  pl-cogent-out:
    - permit 10.0.0.0/24

Result:

ip prefix list pl-cogent-out
  permit 10.0.0.0/24
ip prefix list pl-ntt-out
  permit 10.0.0.0/23
ip prefix list pl-zayo-out
  permit 10.0.1.0/24

And here we order some peer list by priority (dict value), with higher values being more preferred, hence use of reverse=true:

Template:

BGP peers by priority

{% for peer, priority in peer_priority | dictsort(by='value', reverse=true) %}
Peer: {{ peer }}; priority: {{ priority }}
{% endfor %}

Data:

peer_priority:
  ntt: 200
  zayo: 300
  cogent: 100

Result:

BGP peers by priority

Peer: zayo; priority: 300
Peer: ntt; priority: 200
Peer: cogent; priority: 100

float

float(value, default=0.0) - Converts the value to float number. Numeric values in API responses sometimes come as strings. With float we can make sure string is converted before making comparison.

Here's an example of software version checking that uses float.

Template:

{% if eos_ver | float >= 4.22 %}
Detected EOS ver {{ eos_ver }}, using new command syntax.
{% else %}
Detected EOS ver {{ eos_ver }}, using old command syntax.
{% endif %}

Data:

eos_ver: "4.10"

Result

Detected EOS ver 4.10, using old command syntax.

groupby

groupby(value, attribute) - Used to group objects based on one of the attributes. You can choose to group by nested attribute using dot notation. This filter can be used for reporting based on feature value or selecting items for an operation that is only applicable to a subset of objects.

In the below example we group interfaces based on the vlan they're assigned to:

Template:

{% for vid, members in  interfaces | groupby(attribute='vlan') %}
Interfaces in vlan {{ vid }}: {{ members | map(attribute='name') | join(', ') }}
{% endfor %}

Data:

interfaces:
  - name: Ethernet1
    vlan: 50
  - name: Ethernet2
    vlan: 50
  - name: Ethernet3
    vlan: 50
  - name: Ethernet4
    vlan: 60

Result:

Interfaces in vlan 50: Ethernet1, Ethernet2, Ethernet3
Interfaces in vlan 60: Ethernet4

int

int(value, default=0, base=10) - Same as float but here we convert value to integer. Can be also used for converting other bases into decimal base:

Example below shows hexadecimal to decimal conversion.

Template:

LLDP Ethertype
hex: {{ lldp_ethertype }} 
dec: {{ lldp_ethertype | int(base=16) }}

Data:

lldp_ethertype: 88CC

Result:

LLDP Ethertype
hex: 88CC 
dec: 35020

join

join(value, d='', attribute=None) - Very, very useful filter. Takes elements of the sequence and returns concatenated elements as a string.

For cases when you just want to display items, without applying any operations, it can replace for loop. I find join version more readable in these cases.

Template:

ip name-server {{ name_servers | join(" ") }}

Data:

name_servers:
  - 1.1.1.1
  - 8.8.8.8
  - 9.9.9.9
  - 8.8.4.4

Result:

ip name-server 1.1.1.1 8.8.8.8 9.9.9.9 8.8.4.4

map

map(*args, **kwargs) - Can be used to look up an attribute or apply filter on all objects in the sequence.

For instance if you want to normalize letter casing across device names you could apply filter in one go.

Template:

Name-normalized device list:
{{ devices | map('lower') | join('\n') }}

Data:

devices:
 - Core-rtr-warsaw-01
 - DIST-Rtr-Prague-01
 - iNET-rtR-berlin-01

Result:

Name-normalized device list:
core-rtr-warsaw-01
dist-rtr-prague-01
Inet-rtr-berlin-01

Personally I find it most useful for retrieving attributes and their values across a large number of objects. Here we're only interested in values of name attribute:

Template:

Interfaces found:
{{ interfaces | map(attribute='name') | join('\n') }}

Data:

interfaces:
  - name: Ethernet1
    mode: switched
  - name: Ethernet2
    mode: switched
  - name: Ethernet3
    mode: routed
  - name: Ethernet4
    mode: switched

Result:

Interfaces found:
Ethernet1
Ethernet2
Ethernet3
Ethernet4

reject

reject(*args, **kwargs) - Filters sequence of items by applying a Jinja2 test and rejecting objects succeeding the test. That is item will be removed from the final list if result of the test is true.

Here we want to display only public BGP AS numbers.

Template:

Public BGP AS numbers:
{% for as_no in as_numbers| reject('gt', 64495) %}
{{ as_no }}
{% endfor %}

Data:

as_numbers:
 - 1794
 - 28910
 - 65203
 - 64981
 - 65099

Result:

Public BGP AS numbers:
1794
28910

rejectattr

rejectattr(*args, **kwargs) - Same as reject filter but test is applied to the selected attribute of the object.

If your chosen test takes arguments, provide them after test name, separated by commas.

In this example we want to remove 'switched' interfaces from the list by applying test to the 'mode' attribute.

Template:

Routed interfaces:

{% for intf in interfaces | rejectattr('mode', 'eq', 'switched') %}
{{ intf.name }}
{% endfor %}

Data:

interfaces:
  - name: Ethernet1
    mode: switched
  - name: Ethernet2
    mode: switched
  - name: Ethernet3
    mode: routed
  - name: Ethernet4
    mode: switched

Result:

Routed interfaces:

Ethernet3

select

select(*args, **kwargs) - Filters the sequence by retaining only the elements passing the Jinja2 test. This filter is the opposite of reject. You can use either of those depending on what feels more natural in given scenario.

Similarly to reject there's also selectattr filter that works the same as select but is applied to the attribute of each object.

Below we want to report on private BGP AS numbers found on our device.

Template:

Private BGP AS numbers:
{% for as_no in as_numbers| select('gt', 64495) %}
{{ as_no }}
{% endfor %}

Data:

as_numbers:
 - 1794
 - 28910
 - 65203
 - 64981
 - 65099

Result:

Private BGP AS numbers:
65203
64981
65099

tojson

tojson(value, indent=None) - Dumps data structure in JSON format. Useful when rendered template is consumed by application expecting JSON. Can be also used as an alternative to pprint for prettifying variable debug output.

Template:

{{ interfaces | tojson(indent=2) }}

Data:

interfaces:
  - name: Ethernet1
    vlan: 50
  - name: Ethernet2
    vlan: 50
  - name: Ethernet3
    vlan: 50
  - name: Ethernet4
    vlan: 60

Result:

[
  {
    "name": "Ethernet1",
    "vlan": 50
  },
  {
    "name": "Ethernet2",
    "vlan": 50
  },
  {
    "name": "Ethernet3",
    "vlan": 50
  },
  {
    "name": "Ethernet4",
    "vlan": 60
  }
]

unique

unique(value, case_sensitive=False, attribute=None) - Returns list of unique values in given collection. Pairs well with map filter for finding set of values used for given attribute.

Here we're finding which access vlans we use across our interfaces.

Template:

Access vlans in use: {{ interfaces | map(attribute='vlan') | unique | join(', ') }}

Data:

interfaces:
  - name: Ethernet1
    vlan: 50
  - name: Ethernet2
    vlan: 50
  - name: Ethernet3
    vlan: 50
  - name: Ethernet4
    vlan: 60

Result:

Access vlans in use: 50, 60

Conclusion

And with this fairly long list of examples we came to the end of this part of the tutorial. Jinja2 filters can be a very powerful tool in right hands and I hope that my explanations helped you in seeing their potential.

You do need to remember to use them judiciously, if it starts looking unwieldy and doesn't feel right, look at alternatives. See if you can move complexity outside of the template, revise your data model, or if that's not possible, write your own filter.

That's all from me. As always, I look forward to seeing you again, more Jinja2 posts are coming soon!

References