Jinja2 Tutorial - Part 6 - Include and Import

by Przemek

Welcome to another instalment in my Jinja2 Tutorial series. So far we've learned a lot about rendering, control structures and various functions. Here we'll start discussing language features that help us deal with organizing templates. First constructs we'll look at are include and import statements.

Jinja2 Tutorial series

Contents

Introduction

Include and Import statements are some of the tools that Jinja gives us to help with organizing collections of templates, especially once these grow in size.

By using these constructs we can split templates into smaller logical units, leading to files with well-defined scopes. This in turn will make it easier to modify templates when new requirements come up.

The end goal of well-structured collection of templates is increased re-usability as well as maintainability.

Purpose and syntax

'Include' statement allows you to break large templates into smaller logical units that can then be assembled in the final template.

When you use include you refer to another template and tell Jinja to render the referenced template. Jinja then inserts rendered text into the current template.

Syntax for include is:

{% include 'path_to_template_file' %}

where 'path_to_template_file' is the full path to the template which we want included.

For instance, below we have template named cfg_draft.j2 that tells Jinja to find template named users.j2, render it, and replace {% include ... %} block with rendered text.

cfg_draft.j2

{% include 'users.j2' %}

users.j2

username przemek privilege 15 secret NotSoSecret

Final result:

username przemek privilege 15 secret NotSoSecret

Using 'include' to split up large templates

If you look at typical device configuration, you will see a number of sections corresponding to given features. You might have interface configuration section, routing protocol one, access-lists, routing policies, etc. We could write single template generating this entire configuration:

device_config.j2

hostname {{ hostname }}

banner motd ^
===========================================
|   This device is property of BigCorpCo  |
|   Unauthorized access is unauthorized   |
|  Unless you're authorized to access it  |
|  In which case play nice and stay safe  |
===========================================
^

no ip domain lookup
ip domain name local.lab
ip name-server {{ name_server_pri }}
ip name-server {{ name_server_sec }}

ntp server {{ ntp_server_pri }} prefer
ntp server {{ ntp_server_sec }}

{% for iname, idata in interfaces.items() -%}
interface {{ iname }}
 {{ idata.description }}
 {{ idata.ipv4_address }}
{% endfor %}

{% for pl_name, pl_lines in prefix_lists.items() -%}
ip prefix-list {{ pl_name }}
{%- for line in pl_lines %}
 {{ line -}}
{%  endfor -%}
{% endfor %}

router bgp {{ bgp.as_no }}
{%- for peer in bgp.peers %}
 neighbor {{ peer.ip }} remote-as {{ peer.as_no }}
 neighbor {{ peer.ip }} description {{ peer.description }}
{%- endfor %}

Often we grow our templates organically and add one section after another to the single template responsible for generating configuration. Over time however this template grows too large and it becomes difficult to maintain it.

One way of dealing with the growing complexity is to identify sections that roughly correspond to single feature. Then we can move them into their own templates that will be included in the final one.

What we're aiming for is a number of smaller templates dealing with clearly defined feature configuration sections. That way it's easier to locate file to modify when you need to make changes. It's also easier to tell which template does what since we can now give them appropriate names like "bgp.j2" and "acls.j2", instead of having one big template named "device_config.j2".

Taking the previous template we can decompose it into smaller logical units:

base.j2

hostname {{ hostname }}

banner motd ^
{% include 'common/banner.j2' %}
^

dns.j2

no ip domain lookup
ip domain name local.lab
ip name-server {{ name_server_pri }}
ip name-server {{ name_server_sec }}

ntp.j2

ntp server {{ ntp_server_pri }} prefer
ntp server {{ ntp_server_sec }}

interfaces.j2

{% for iname, idata in interfaces.items() -%}
interface {{ iname }}
 {{ idata.description }}
 {{ idata.ipv4_address }}
{% endfor %}

prefix_lists.j2

{% for pl_name, pl_lines in prefix_lists.items() -%}
ip prefix-list {{ pl_name }}
{%- for line in pl_lines %}
 {{ line -}}
{%  endfor -%}
{% endfor %}

bgp.j2

router bgp {{ bgp.as_no }}
{%- for peer in bgp.peers %}
 neighbor {{ peer.ip }} remote-as {{ peer.as_no }}
 neighbor {{ peer.ip }} description {{ peer.description }}
{%- endfor %}

We now have a collection of separate templates, each with a clear name conveying its purpose. While our example doesn't have too many lines, I think you'd agree with me that the logical grouping we arrived at will be easier to work with and it'll be quicker to build mental model of what is going on here.

With features moved to individual templates we can finally use include statements to compose our final config template:

config_final.j2

{# Hostname and banner -#}
{% include 'base.j2' %}

{% include 'dns.j2' %}

{% include 'ntp.j2' %}

{% include 'interfaces.j2' %}

{% include 'prefix_lists.j2' %}

{# BGP instance and peering -#}
{% include 'bgp.j2' %}

You open this template and just from a quick glance you should be able to get an idea as to what it's trying to do. It's much cleaner, and we can easily add comments that won't get lost among hundreds of other lines.

As a bonus, you can quickly test template with one feature disabled by commenting single line out, or simply temporarily removing it.

It's now easier to make changes and it's faster to identify feature and corresponding template instead of searching through one big template with potentially hundreds of lines.

Similarly, when new section is needed we can create separate template and include it in the final template therefore fulfilling our goal of increasing modularity.

Shared template snippets with 'include'

You might have also noticed that one of the included templates had itself include statement.

base.j2

hostname {{ hostname }}

banner motd ^
{% include 'common/banner.j2' %}
^

You can use include statement at any level in the hierarchy of templates and anywhere in the template you want. This is exactly what we did here; we moved text of our banner to a separate file which we then include in base.j2 template.

We could argue that banner itself is not important enough to warrant its own template. However, there's another class of use cases where include is helpful. We can maintain library of common snippets used across many different templates.

This differs from our previous example, where we decomposed one big template into smaller logical units all tightly related to the final template. With common library we have units that are re-usable across many different templates that might not otherwise have any similarities.

Missing and alternative templates

Jinja allows us to ask for template to be included optionally by adding ignore missing argument to include.

{% include 'guest_users.j2' ignore missing %}

It essentially tells Jinja to look for guest_users.j2 template and insert rendered text if found. If template is not found this will result in blank line, but no error will be raised.

I would generally advise against using this in your templates. It's not something that's widely used so someone reading your template might not know what it's meant to do. End result also relies on the presence of the specific file which might make troubleshooting more difficult.

There are better ways of dealing with optional features, some of which rely on template inheritance that we will talk about in the next post.

Closely related to 'ignore missing' is possibility of providing list of templates to include. Jinja will check templates for existence, including the first one that exists.

In the below example, if local_users.j2 does not exist but radius_users.j2 does, then rendered radius_users.j2 will end up being inserted.

{% include ['local_users.j2', 'radius_users.j2'] %}

You can even combine list of templates with ignore missing argument:

{% include ['local_users.j2', 'radius_users.j2'] ignore missing %}

This will result in search for listed templates and no error raised if none of them are found.

Again, while tempting, I'd advise against using this feature unless you exhausted other avenues. I wouldn't enjoy having to figure out which one of the listed templates ended up being included if something didn't look right in my final render.

To summarize, you can use 'include' to:

  • split large template into smaller logical units
  • re-use snippets shared across multiple templates

Import statement

In Jinja we use import statement to access macros kept in other templates. The idea is to have often used macros in their own files that are then imported in the templates that need them.

This is different than include statement in that no rendering takes place here. Instead the way it works is very similar to import statement in Python. Imported macros are made available for use in the template that imports them.

Three ways of importing

There are three ways in which we can import macros.

All three ways will use import the below template:

macros/ip_funcs.j2

{% macro ip_w_wc(ip_net) -%}
{{ ip_net|ipaddr('network') }} {{ ip_net|ipaddr('hostmask')}}
{%- endmacro -%}

{% macro ip_w_netm(ip_net) -%}
{{ ip_net|ipaddr('network') }} {{ ip_net|ipaddr('netmask') }}
{%- endmacro -%}

{% macro ip_w_pfxlen(ip_net) -%}
{{ ip_net|ipaddr('network/prefix') }}
{%- endmacro -%}

  1. Importing the whole template and assigning it to variable. Macros are attributes of the variable.

    imp_ipfn_way1.j2

    {% import 'macros/ip_funcs.j2' as ipfn %}

{{ ipfn.ip_w_wc('10.0.0.0/24') }}
{{ ipfn.ip_w_netm('10.0.0.0/24') }}
{{ ipfn.ip_w_pfxlen('10.0.0.0/24') }}

  2. Importing specific macro into the current namespace.

    imp_ipfn_way2.j2

    {% from 'macros/ip_funcs.j2' import ip_w_wc, ip_w_pfxlen %}

{{ ip_w_wc('10.0.0.0/24') }}
{{ ip_w_pfxlen('10.0.0.0/24') }}

  3. Importing specific macro into the current namespace and giving it an alias.

    imp_ipfn_way3

    {% from 'macros/ip_funcs.j2' import ip_w_wc as ipwild %}

{{ ipwild('10.0.0.0/24') }}

You can also combine 2 with 3:

imp_ipfn_way2_3

{% from 'macros/ip_funcs.j2' import ip_w_wc as ipwild, ip_w_pfxlen %}

{{ ipwild('10.0.0.0/24') }}
{{ ip_w_pfxlen('10.0.0.0/24') }}

My recommendation is to always use 1. This forces you to access macros via explicit namespace. Methods 2 and 3 risk clashes with variables and macros defined in the current namespace. As is often the case in Jinja, explicit is better than implicit.

Caching and context variables

Imports are cached, which means that they are loaded very quickly on each subsequent use. There is a price to pay for that, namely the imported templates don't have access to variables in template that imports them.

This means that by default you can't access variables passed into the context inside of macros imported from another file.

Instead you have to build your macros so that they only rely on values passed to them explicitly.

To illustrate this I wrote two versions of macro named def_if_desc, one trying to access variables available to template importing it. The other macro relies on dictionary passed to it explicitly via value.

Both versions use the below data:

default_desc.yaml

interfaces:
 Ethernet10:
   role: desktop

  • Version accessing template variables:

    macros/def_desc_ctxvars.j2

    {% macro def_if_desc(ifname) -%}
Unused port, dedicated to {{ interfaces[ifname].role }} devices
{%- endmacro -%}

    im_defdesc_vars.j2

    {% import 'macros/def_desc_ctxvars.j2' as desc -%}

{{ desc.def_if_desc('Ethernet10') }}

    When I try to render im_defdesc_vars.j2 I get the below traceback:

    ...(cut for brevity)
  File "F:\projects\j2-tutorial\templates\macros\def_desc_ctxvars.j2", line 2, in template
    Unused port, dedicated to {{ interfaces[ifname].descri }} devices
  File "F:\projects\j2-tutorial\venv\lib\site-packages\jinja2\environment.py", line 452, in getitem
    return obj[argument]
jinja2.exceptions.UndefinedError: 'interfaces' is undefined

    You can see that Jinja complains that it cannot access interfaces. This is just as we expected.

  • Version accessing key of dictionary passed explicitly by importing template.

    default_desc.j2

    {% macro def_if_desc(intf_data) -%}
Unused port, dedicated to {{ intf_data.role }} devices
{%- endmacro -%}

    im_defdesc.j2

    {% import 'macros/default_desc.j2' as desc -%}

{{ desc.def_if_desc(interfaces['Ethernet10']) }}

    And this renders just fine:

    Unused port, dedicated to desktop devices

Hopefully now you can see what the default behavior is when importing macros. Since values of variables in the context can change at any time, Jinja engine cannot cache them and we are not allowed to access them from within macros.

Disabling macro caching

However, if for whatever reason you think it is a good idea to allow your macros to access context variables you can change the default behavior with additional argument with context which you pass to import statement.

Note: This will automatically disable caching.

For completeness this is how we can "fix" our failing macro:

macros/def_desc_ctxvars.j2

{% macro def_if_desc(ifname) -%}
Unused port, dedicated to {{ interfaces[ifname].role }} devices
{%- endmacro -%}

im_defdesc_vars_wctx.j2

{% import 'macros/def_desc_ctxvars.j2' as desc with context -%}

{{ desc.def_if_desc('Ethernet10') }}

And now it works:

Unused port, dedicated to devices

Personally, I don't think it's a good idea to use import together with context. The whole point of importing macros from separate file is to allow them to be used in other templates and leveraging caching. There could be potentially hundreds, if not thousands, of them, and as soon as you use with context the caching is gone.

I can also see some very subtle bugs creeping in in macros that rely on accessing variables from template context.

To be on the safe side, I'd say stick with standard import and always use namespaces, e.g.

{% import 'macros/ip_funcs.j2' as ipfn %}

Conclusion

We learned about two Jinja constructs that can help us in managing complexity emerging when our templates grow in size. By leveraging import and include statements we can increase re-usability and make our templates easier to maintain. I hope that included examples showed you how you can use this knowledge to make your template collection better organized and easier to understand.

Here's my quick summary of what to use when:

Import Include
Purpose Imports macros from other templates Renders other template and inserts results
Context variables Not accessible (default) Accessible
Good for Creating shared macro libraries Splitting template into logical units and common snippets

I hope you found this post useful and are looking forward to more. Next post will continue discussion of ways to organize templates by focusing on template inheritance. Stay tuned!

References