Jinja2 Tutorial - Part 5 - Macros

by Przemek

Welcome to the part 5 of Jinja2 Tutorial where we learn all about macros. We'll talk about what macros are, why we would use them and we'll see some examples to help us appreciate this feature better.

Jinja2 Tutorial series

Contents

What are macros?

Macros are similar to functions in many programming languages. We use them to encapsulate logic used to perform repeatable actions. Macros can take arguments or be used without them.

Inside of macros we can use any of the Jinja features and constructs. Result of running macro is some text. You can essentially treat macro as one big evaluation statement that also allows parametrization.

Why and how of macros

Macros are great for creating reusable components when we find ourselves copy pasting around same lines of text and code. You might benefit from macro even when all it does is rendering static text.

Take for example device banners, these tend to be static but are used over and over again. Instead of copy pasting text of the banner across your templates you can create macro and have it render the banner.

Not only will you reduce mistakes that can happen during copying but you also make future updates to the banner much easier. Now you have only one place where the banner needs to be changed and anything else using this macro will reflect the changes automatically.

{% macro banner() -%}
banner motd ^
===========================================
|   This device is property of BigCorpCo  |
|   Unauthorized access is unauthorized   |
|  Unless you're authorized to access it  |
|  In which case play nice and stay safe  |
===========================================
^
{% endmacro -%}

{{ banner() }}

banner motd ^
===========================================
|   This device is property of BigCorpCo  |
|   Unauthorized access is unauthorized   |
|  Unless you're authorized to access it  |
|  In which case play nice and stay safe  |
===========================================
^

So that's our first macro right there!

As you can see above we start macro with {% macro macro_name(arg1, arg2) %} and we end it with {% endmacro %}. Arguments are optional.

Anything you put in between opening and closing tags will be processed and rendered at a location where you called the macro.

Once we defined macro we can use it anywhere in our template. We can directly insert results by using {{ macro_name() }} substitution syntax. We can also use it inside other constructs like if..else blocks or for loops. You can even pass macros to other macros!

Adding parameters

Real fun begins when you start using parameters in your macros. That's when their show their true potential.

Oure next macro renders default interface description. We assign different roles to our ports and we want the default description to reflect that. We could achieve this by writing macro taking interface role as argument.

Data:

interfaces:
 - name: Ethernet10
   role: desktop
 - name: Ethernet11
   role: desktop
 - name: Ethernet15
   role: printer
 - name: Ethernet22
   role: voice

Template with macro:

{% macro def_if_desc(if_role) -%}
Unused port, dedicated to {{ if_role }} devices
{%- endmacro -%}

{% for intf in interfaces -%}
interface {{ intf.name }}
  description {{ def_if_desc(intf.role) }}
{% endfor -%}

Rendered text:

interface Ethernet10
  description Unused port, dedicated to desktop devices
  ip address
interface Ethernet11
  description Unused port, dedicated to desktop devices
  ip address
interface Ethernet15
  description Unused port, dedicated to printer devices
  ip address
interface Ethernet22
  description Unused port, dedicated to voice devices
  ip address

It might not be immediately apparent if macro is useful here since we only have one line in the body. We could've just written this line inside of the for loop. Downside of that is that our intent is not clearly conveyed.

{% for intf in interfaces -%}
interface {{ intf.name }}
  description Unused port, dedicated to {{ intf.role }} devices
{% endfor -%}

This works but it's not immediately obvious that this is description we want to be used as a default. Things will get even worse if we start adding more processing here.

If we use macro however, the name of the macro tells us clearly that a default interface description will be applied. That is, it is clear what our intent was here.

And there's the real kicker. Macros can be moved to separate files and included in templates that need them. Which means you only need to maintain this one macro that then can be used by hundreds of templates! And number of places you have to update your default description? One, just one.

Macros for deeply nested structures

Another good use case for macros is accessing values in deeply nested data structures.

Modern APIs can return results with many levels of dictionaries and lists making it easy to make error when writing expressions accessing values in these data structures.

The below is real-life example of output returned by Arista device for command:

sh ip bgp neighbors x.x.x.x received-routes | json

Due to size, I'm showing full result for one route entry only, out of 3:

{
  "vrfs": {
      "default": {
      "routerId": "10.3.0.2",
      "vrf": "default",
      "bgpRouteEntries": {
          "10.1.0.1/32": {
          "bgpAdvertisedPeerGroups": {},
          "maskLength": 32,
          "bgpRoutePaths": [
              {
              "asPathEntry": {
                  "asPathType": null,
                  "asPath": "i"
              },
              "med": 0,
              "localPreference": 100,
              "weight": 0,
              "reasonNotBestpath": null,
              "nextHop": "10.2.0.0",
              "routeType": {
                  "atomicAggregator": false,
                  "suppressed": false,
                  "queued": false,
                  "valid": true,
                  "ecmpContributor": false,
                  "luRoute": false,
                  "active": true,
                  "stale": false,
                  "ecmp": false,
                  "backup": false,
                  "ecmpHead": false,
                  "ucmp": false
              }
              }
          ],
          "address": "10.1.0.1"
          },
  ...
      "asn": "65001"
      }
  }
}

There's a lot going on here and in most cases you will only need to get values for few of these attributes.

Say we wanted to access just prefix, next-hop and validity of the path.

Below is object hierarchy we need to navigate in order to access these values:

  • vrfs.default.bgpRouteEntries - prefixes are here (as keys)
  • vrfs.default.bgpRouteEntries[pfx].bgpRoutePaths.0.nextHop - next hop
  • vrfs.default.bgpRouteEntries[pfx].bgpRoutePaths.0.routeType.valid - route validity

I don't know about you but I really don't fancy copy pasting that into all places I would need to access these.

So here's what we can do to make it a bit easier, and more obvious, for ourselves.

{% macro print_route_info(sh_bgpr) -%}
{% for route, routenfo in vrfs.default.bgpRouteEntries.items() -%}
Route: {{ route }} - Next Hop: {{ 
routenfo.bgpRoutePaths.0.nextHop }} - Permitted: {{ 
routenfo.bgpRoutePaths.0.routeType.valid }}
{% endfor %}
{%- endmacro -%}

{{ print_route_info(sh_bgp_routes) }}

Route: 10.1.0.1/32 - Next Hop: 10.2.0.0 - Permitted: True
Route: 10.1.0.2/32 - Next Hop: 10.2.0.0 - Permitted: True
Route: 10.1.0.3/32 - Next Hop: 10.2.0.0 - Permitted: True

I moved the logic, and complexity, involved in accessing attributes to a macro called print_route_info. This macro takes output of our show command and then gives us back only what we need.

If we need to access more attributes we'd only have to make changes to the body of our macro.

At the place where we actually need the information we call well named macro and give it the output of the command. This makes it more obvious as to what we're trying to achieve and mechanics of navigating data structures are hidden away.

Branching out inside macro

Let's do another example, this time our macro will have if..else block to show that we can return result depending on conditional checks.

I created data model where BGP peer IP and name are not explicitly listed in the mapping I use for specifying peers. Instead we're pointing each peer entry to local interface over which we want to establish the peering.

We're also assuming here that all of our peerings use /31 mask.

interfaces:
  Ethernet1:
    ip_add: 10.1.1.1/31
    peer: spine1
    peer_intf: Ethernet1
  Ethernet2:
    ip_add: 10.1.1.9/31
    peer: spine2
    peer_intf: Ethernet1

bgp:
  as_no: 65001
  peers:
    - intf: Ethernet1
      as_no: 64512
    - intf: Ethernet2
      as_no: 64512

Using this data model we want to build config for BGP neighbors. Taking advantage of ipaddr filter we can do the following:

  • Find 1st IP address in network configured on the linked interface.
  • Check if 1st IP address equals IP address configured on the interface.
    • If it is equal then IP of BGP peer must be the 2nd IP address in this /31.
    • If not then BGP peer IP must be the 1st IP address.

Converting this to Jinja syntax we get the following:

router bgp {{ bgp.as_no }}
{%- for peer in bgp.peers -%}
{% set fst_ip = interfaces[peer.intf].ip_add | ipaddr(0) -%}
{% if our_ip == fst_ip -%}
{% set peer_ip = fst_ip | ipaddr(1) | ipaddr('address') -%}
{% else -%}
{% set peer_ip = fst_ip | ipaddr('address') -%}
{% endif %}
 neighbor {{ peer_ip }} remote-as {{ peer.as_no }}
 neighbor {{ peer_ip }} description {{ interfaces[peer.intf].peer }}
{%- endfor %}

And this is the result of rendering:

router bgp 65001
 neighbor 10.1.1.0 remote-as 64512
 neighbor 10.1.1.0 description spine1
 neighbor 10.1.1.8 remote-as 64512
 neighbor 10.1.1.8 description spine2

Job done. We got what we wanted, neighbor IP worked out automatically from IP assigned to local interface.

But, the longer I look at it the more I don't like feel of this logic and manipulation preceding the actual neighbor statements.

You also might want to use the logic of working out peer IP elsewhere in the template which means copy pasting. And if later you change mask on the interface or want to change data structure slightly you'll have to find all the places with the logic and make sure you change all of them.

I'd say this case is another good candidate for building a macro.

So I'm going to move logic for working out peer IP to the macro that I'm calling peer_ip. This macro will take one argument local_intf which is the name of the interface for which we're configuring peering.

If you compare this version with non-macro version you can see that most of the code is the same except that instead of setting final value and assigning it to variable we use substitution statements.

{% macro peer_ip(local_intf) -%}
{% set local_ip = interfaces[local_intf].ip_add -%}
{% set fst_ip = local_ip | ipaddr(0) -%}
{% if fst_ip == local_ip -%}
{{ fst_ip | ipaddr(1) | ipaddr('address') -}}
{% else -%}
{{ fst_ip | ipaddr('address') -}}
{%- endif -%}
{% endmacro -%}

router bgp {{ bgp.as_no }}
{%- for peer in bgp.peers -%}
{%- set bgp_peer_ip = peer_ip(peer.intf) %}
 neighbor {{ bgp_peer_ip }} remote-as {{ peer.as_no }}
 neighbor {{ bgp_peer_ip }} description {{ interfaces[peer.intf].peer }}
{%- endfor %}

We use this macro in exactly one place in our function, we assign value it returns to variable bgp_peer_ip. We can then use bgp_peer_ip in our neighbor statements.

{%- set bgp_peer_ip = peer_ip(peer.intf) %}

Another thing that I like about this approach is that we can move macro to its own file and then include it in the templates that use it.

We'll talk about Jinja imports and includes in more details in future posts. However this is such a useful feature that later in this post I will show you short example of macros in their own files.

Macros in macros

Now here's an interesting one. We can pass macros as arguments to other macros. This is similar to Python where functions can be passed around like any other object.

Would we want to do it though? There are certainly cases when that might be useful. I can think of need for having more generic macro producing some result and taking another macro as an argument to enable changing of format used to render result.

This means that we could have parent macro deal with rendering common part of the output we're interested in. Then macro passed as an argument would be responsible for handling difference in rendering specific bit that would be dependent on the needs of the caller.

To illustrate this and make it easier to visualize, consider case of rendering ACL entries. Different vendors could, and often do, use different format for IP source and destination objects. Some will use "net_address/pfxlen", while some will use "net_address wildcard".

We could write multiple ACL rendering macros, one for each case. Another option would be to have if..else logic in larger macro with macro argument deciding which format to use.

Or we can encapsulate logic responsible for format conversion in tiny macros. We then can have macro responsible for ACL rendering that receives format conversion macro as one of the arguments. That is, ACL macro doesn't know how to do rendering and it does not care. It just knows that it will be given macro from outside and that it can apply it where required.

Here's the actual implementation that includes 3 different formatting macros.

Data used for our example:

networks:
  - name: quant_server_net
    prefix: 10.0.0.0/24
    services:
      - computing

svc_def:
  computing:
    - {ip: 10.90.0.5/32, prot: tcp, port: 5008}
    - {ip: 10.91.4.0/255.255.255.0, prot: tcp, port: 5009}
    - {ip: 10.91.6.32/27, prot: tcp, port: 6800}

Template with macros:

{% macro ip_w_wc(ip_net) -%}
{{ ip_net|ipaddr('network') }} {{ ip_net|ipaddr('hostmask')}}
{%- endmacro -%}

{% macro ip_w_netm(ip_net) -%}
{{ ip_net|ipaddr('network') }} {{ ip_net|ipaddr('netmask') }}
{%- endmacro -%}

{% macro ip_w_pfxlen(ip_net) -%}
{{ ip_net|ipaddr('network/prefix') }}
{%- endmacro -%}

{% macro acl_lines(aclobj, src_pfx, pfx_fmt) -%}
{% for line in aclobj %}
 permit {{ line.prot }} {{ pfx_fmt(src_pfx) }} {{ pfx_fmt(line.ip) }} 
{%- if line.prot in ['udp', 'tcp'] %} eq {{ line.port }}{% endif -%}
{% endfor -%}
{%- endmacro -%}

{% for net in networks -%}
ip access-list extended al_{{ net.name }}
{%-  for svc in net.services -%}
{{ acl_lines(svc_def[svc], net.prefix, ip_w_pfxlen) }}
{%  endfor -%}
{% endfor -%}

Rendering results with ip_w_wc macro:

ip access-list extended al_quant_server_net
 permit tcp 10.0.0.0 0.0.0.255 10.90.0.5 0.0.0.0 eq 5008
 permit tcp 10.0.0.0 0.0.0.255 10.91.4.0 0.0.0.255 eq 5009
 permit tcp 10.0.0.0 0.0.0.255 10.91.6.32 0.0.0.31 eq 6800

Rendering results with ip_w_netm macro:

ip access-list extended al_quant_server_net
 permit tcp 10.0.0.0 255.255.255.0 10.90.0.5 255.255.255.255 eq 5008
 permit tcp 10.0.0.0 255.255.255.0 10.91.4.0 255.255.255.0 eq 5009
 permit tcp 10.0.0.0 255.255.255.0 10.91.6.32 255.255.255.224 eq 6800

Rendering results with ip_w_pfxlen macro:

ip access-list extended al_quant_server_net
 permit tcp 10.0.0.0/24 10.90.0.5/32 eq 5008
 permit tcp 10.0.0.0/24 10.91.4.0/24 eq 5009
 permit tcp 10.0.0.0/24 10.91.6.32/27 eq 6800

Hopefully now you can see what I'm trying to achieve here. I can use the same parent macro in templates rendering the config for different vendors by simply providing different formatter macro. To top it off, we make our intent clear, yet again.

Our formatting macros can be reused in many places and it's very easy to add new formatters that can be used in ACL macro and elsewhere.

Also by decoupling, and abstracting away, IP prefix formatting we make ACL macro more focused.

A lot of these decisions are down to individual preferences but I feel that this technique is very powerful and it's good to know that it's there when you need it.

Moving macros to a separate file

I'll now show you an example of how a macro can be moved to a separate template file. We will then import the macro and call it from template located in a completely different file.

I decided to take macros we created for displaying IP network in different formats. I'm moving 3 formatting macros to separate file and keeping ACL macro in the original template.

The result is two templates.

ip_funcs.j2:

{% macro ip_w_wc(ip_net) -%}
{{ ip_net|ipaddr('network') }} {{ ip_net|ipaddr('hostmask')}}
{%- endmacro -%}

{% macro ip_w_netm(ip_net) -%}
{{ ip_net|ipaddr('network') }} {{ ip_net|ipaddr('netmask') }}
{%- endmacro -%}

{% macro ip_w_pfxlen(ip_net) -%}
{{ ip_net|ipaddr('network/prefix') }}
{%- endmacro -%}

acl_variants.j2:

{% import 'ip_funcs.j2' as ipfn -%}

{% macro acl_lines(aclobj, src_pfx, pfx_fmt) -%}
{% for line in aclobj %}
 permit {{ line.prot }} {{ pfx_fmt(src_pfx) }} {{ pfx_fmt(line.ip) }} 
{%- if line.prot in ['udp', 'tcp'] %} eq {{ line.port }}{% endif -%}
{% endfor -%}
{%- endmacro -%}

Prefix with prefix length ACL:

{% for net in networks -%}
ip access-list extended al_{{ net.name }}
{%-  for svc in net.services -%}
{{ acl_lines(svc_def[svc], net.prefix, ipfn.ip_w_wc) }}
{%  endfor -%}
{% endfor %}

Network with Wildcard ACL:

{% for net in networks -%}
ip access-list extended al_{{ net.name }}
{%-  for svc in net.services -%}
{{ acl_lines(svc_def[svc], net.prefix, ipfn.ip_w_pfxlen) }}
{%  endfor -%}
{% endfor %}

Network with network Mask ACL:

{% for net in networks -%}
ip access-list extended al_{{ net.name }}
{%-  for svc in net.services -%}
{{ acl_lines(svc_def[svc], net.prefix, ipfn.ip_w_netm) }}
{%  endfor -%}
{% endfor -%}

First template ip_funcs.j2 contains formatter macros, and nothing else. Notice also there's no change to the code, we copied these over ad verbatim.

Something interesting happened to our original template, here called acl_variants.j2. First line {% import 'ip_funcs.j2' as ipfn -%} is new and the way we call formatter macros is different now.

Line {% import 'ip_funcs.j2' as ipfn -%} looks like import statement in Python and it works similarly. Jinja engine will look for file called ip_funcs.j2 and will make variables and macros from that file available in namespace ipfn. That is anything found in imported file can be now accessed using ipfn. notation.

And this is how we get to the way we need to call formatters now. For example macro converting IP prefix to network/wildcard form is called with ipfn.ip_w_wc syntax.

For good measure I added all formatting variants to our template and this is the final result:

Prefix with prefix length ACL:

ip access-list extended al_quant_server_net
 permit tcp 10.0.0.0 0.0.0.255 10.90.0.5 0.0.0.0 eq 5008
 permit tcp 10.0.0.0 0.0.0.255 10.91.4.0 0.0.0.255 eq 5009
 permit tcp 10.0.0.0 0.0.0.255 10.91.6.32 0.0.0.31 eq 6800


Network with Wildcard ACL:

ip access-list extended al_quant_server_net
 permit tcp 10.0.0.0/24 10.90.0.5/32 eq 5008
 permit tcp 10.0.0.0/24 10.91.4.0/24 eq 5009
 permit tcp 10.0.0.0/24 10.91.6.32/27 eq 6800


Network with network Mask ACL:

ip access-list extended al_quant_server_net
 permit tcp 10.0.0.0 255.255.255.0 10.90.0.5 255.255.255.255 eq 5008
 permit tcp 10.0.0.0 255.255.255.0 10.91.4.0 255.255.255.0 eq 5009
 permit tcp 10.0.0.0 255.255.255.0 10.91.6.32 255.255.255.224 eq 6800

Moving macros to their own files and importing them from other templates is a very powerful feature. I will be talking more about it in the future post on imports and includes.

Advanced macro usage

Varargs and kwargs

Inside of macros you can access special variables that are exposed by default. Some of them relate to internal plumbing and are not very interesting but few of them you might find use for.

  • varargs - if macro was given more positional arguments than explicitly listed in macro's definition, then Jinja will put them into special variable called varargs. You can then iterate over them and process if you feel it makes sense.

  • kwargs - similarly to varargs, any keyword arguments not matching explicitly listed ones will end up in kwargs variable. This can be iterated over using kwargs.items() syntax.

Personally I think both of these are not very useful in most of use cases. In the world of web development it might make sense to accept a number of elements for rendering tables, and other HTML items.

In the world of infrastructure automation I prefer explicit arguments and clear intent which I feel is not the case when using special variables.

I do have some contrived examples to show you how that would work if you ever feel you really can make use of this feature.

Below macro takes one explicit argument vid, which specifies VLAN ID we want assigned as access port to interfaces. Any extra positional arguments will be treated as interface names that need to be configured for the given VLAN ID.

{% macro set_access_vlan(vid) -%}
{% for intf in varargs -%}
interface {{ intf }}
  switchport
  switchport mode access
  switchport access vlan {{ vid }}
{% endfor -%}
{%- endmacro -%}

{{ set_access_vlan(10, "Ethernet10", "Ethernet20") }}

Result:

interface Ethernet10
  switchport
  switchport mode access
  switchport access vlan 10
interface Ethernet20
  switchport
  switchport mode access
  switchport access vlan 10

Below is similar macro but this time we have no explicit arguments. We will however read any passed keyword arguments and we treat key as the interface name and value as VLAN ID to assign.

{% macro set_access_vlan() -%}
{% for intf, vid in kwargs.items() -%}
interface {{ intf }}
  switchport
  switchport mode access
  switchport access vlan {{ vid }}
{% endfor -%}
{%- endmacro -%}

{{ set_access_vlan(Ethernet10=10, Ethernet15=15, Ethernet20=20) }}

Render results:

interface Ethernet10
  switchport
  switchport mode access
  switchport access vlan 10
interface Ethernet15
  switchport
  switchport mode access
  switchport access vlan 15
interface Ethernet20
  switchport
  switchport mode access
  switchport access vlan 20

Both of these examples work and even do something potentially useful. These special variables just don't feel right to me but they're there if you ever need them.

call block

Call blocks are constructs that call other macros and are themselves macros, except they have no names, so they're only used at the point they appear.

You use call bocks with {% call called_macro() %}...{% endcal %} syntax.

These work a bit like callbacks since macros they invoke in turn call back to execute call macros. You can see similarities here to our ACL macro that used different formatting macros. We can have many call macros using single named macro, with these call macros allowing variation in logic executed by named macro.

I don't know if there's a historical reason for their existence since I can't really see any advantage of using these over named macros. They are also not very intuitive to use. But again, they're here and maybe you will find need for them.

So contrived example time!

Bellow call macro calls make_acl macro which in turn calls back to execute calling macro:

{% macro make_acl(type, name) -%}
ip access-list {{ type }} {{ name }}
{{- caller() }}
{%- endmacro -%}

{% call make_acl('extended', 'al-ext-01') %}
 permit ip 10.0.0.0 0.0.0.255 10.0.0.0.255
 deny ip any any
{%- endcall %}

Result:

ip access-list extended al-ext-01
 permit ip 10.0.0.0 0.0.0.255 10.0.0.0.255
 deny ip any any

We got some sensible result here, but at what cost? Do you see how ACL lines made it to the body? The magic is in {{ caller() }} line. Here special function caller() essentially executes body of the call block that called make_acl macro.

This is what happened, step by step:

  • call launched make_acl
  • make_acl worked its way through, rendering stuff, until it encountered caller()
  • make_acl executed calling block with caller() and inserted results
  • make_acl moved on past caller() through the rest of its body

It works but again I see no advantage over using named macros and passing them explicitly around.

Fun is not over yet though, called macro can invoke caller with arguments.

{% macro acl_lines(aclobj, src_pfx) -%}
{% for line in aclobj %}
 permit {{ line.prot }} {{ caller(src_pfx) }} {{ caller(line.ip) }} 
{%- if line.prot in ['udp', 'tcp'] %} eq {{ line.port }}{% endif -%}
{% endfor -%}
{%- endmacro -%}

{% for net in networks -%}
ip access-list extended al_{{ net.name }}
{%-  for svc in net.services -%}

{% call(ip_net) acl_lines(svc_def[svc], net.prefix) -%}
{{ ip_net|ipaddr('network/prefix') }}
{%- endcall -%}

{%  endfor -%}
{% endfor -%}

This is a call block version of our ACL rendering with variable formatters. This time I included formatter inside of the call block. Our block takes ip_net argument which it expects called macro to provide when calling back.

And this is exactly what happens on the below line:

permit {{ line.prot }} {{ caller(src_pfx) }} {{ caller(line.ip) }}

So, we have call block call acl_lines with two arguments. Macro acl_lines then calls call back with caller(src_pfx) and caller(line.ip) fulfilling its contract.

Caveat here is that we cannot reuse our formatter, it's all in the unnamed macro aka call block. Once it executes, that's it, you need a new one if you want to use formatter.

Conclusion

I think that macros are one of the more powerful features of Jinja and you will benefit greatly from learning how to use them. Combined with import you will get reusable, well defined groups of snippets that can be kept separately from other templates. This allows us to extract repeatable, sometimes complex, logic, and make our templates cleaner and easier to follow.

As always, see what works for you. If your macros get unwieldy consider using custom filters. And be careful when using advanced macro features, these should really be only reserved for special cases, if used at all.

I hope you learned something from this post and that it gave you some ideas. More posts on Jinja2 are coming so do pop by every so often to see what's new :)

References