Jinja2 Tutorial - Part 3 - Whitespace control

by Przemek

Text documents are the final result of rendering templates. Depending on the end consumer of these documents whitespace placement could be significant. One of the major niggles in Jinja2, in my opinion, is the way control statements and other elements affect whitespace output in the end documents.

To put it bluntly, mastering whitespaces in Jinja2 is the only way of making sure your templates generate text exactly the way you intended.

Now we know the importance of the problem, time to understand where it originates, to do that we’ll have a look at a lot of examples. Then we'll learn how we can control rendering whitespaces in Jinja2 templates.

Jinja2 Tutorial series

Contents

Understanding whitespace rendering in Jinja2

We'll start our learning by looking at how Jinja2 renders whitespaces by looking at trivial example, template with no variables, just two lines of text and a comment:

Starting line
{# Just a comment #}
Line after comment

This is how it looks like when it’s rendered:

Starting line

Line after comment

Ok, what happened here? Did you expect an empty line to appear in place of our comment? I did not. I’d expect the comment line to just disappear into nothingness, but that’s not the case here.

So here’s a very important thing about Jinja2. All of the language blocks are removed when the template is rendered but all of the whitespaces remain in place. That is if there are spaces, tabs, or newlines, before or after, blocks, then these will be rendered.

This explains why comment block left a blank line once template was rendered. There is a newline character after the {# #} block. While the block itself was removed, newline remained.

Below is a more involving, but fairly typical, template, containing for loop and if statements:

{% for iname, idata in interfaces.items() %}
interface {{ iname }}
 description {{ idata.description }}
  {% if idata.ipv4_address is defined %}
 ip address {{ idata.ipv4_address }}
  {% endif %}
{% endfor %}

Values that we feed into the template:

interfaces:
  Ethernet1:
    description: capture-port
  Ethernet2:
    description: leaf01-eth51
    ipv4_address: 10.50.0.0/31

And this is how Jinja2 will render this, with all settings left to defaults:


interface Ethernet1
 description capture-port
  

interface Ethernet2
 description leaf01-eth51
  
 ip address 10.50.0.0/31

This doesn’t look great, does it? There are extra newlines added in few places. Also, interestingly enough, there are leading spaces on some lines, that can’t be seen on screen but could really break things for us in the future. Overall it’s difficult to figure out where all of the whitespaces came from.

To help you in better visualizing generated text, here’s the same output, but now with all of the whitespaces rendered:

Each bullet point represents a space character, and return icon represents newlines. You should now clearly see leading spaces that were left by Jinja2 block on three of the lines, as well as all of the extra newlines.

Ok, that’s all great you say, but it still is not so obvious where these came from. The real question we want to answer is:

Which template line contributed to which line in the final result?

To answer that question I rendered whitespaces in the template as well as the output text. Then I added colored, numbered, highlight blocks in the lines of interest, to allow us to match source with the end product.

You should now see very easily where each of the Jinja blocks adds whitespaces to the resulting text.

If you’re also curious why then read on for detailed explanation:

  1. Line containing {% for %} block, number 1 with blue outlines, ends with a newline. This block gets executed for each key in dictionary. We have 2 keys, so we get extra 2 newlines inserted into final text.

  2. Line containing {% if %} block, numbers 2a and 2b with green and light-green outlines, has 2 leading spaces and ends with a newline. This is where things get interesting. The actual {% if %} block is removed leaving behind 2 spaces that always get rendered. But trailing newline is inside of the block. This means that with {% if %} evaluating to false we get 2a but NOT 2b. If it evaluates to true we get both 2a AND 2b.

  3. Line containing {% endif %} block, numbers 3a and 3b with red and orange outlines, has 2 leading spaces and ends with a newline. This is again interesting and our situation here is the reverse of previous case. Two leading spaces are inside of the if block, but the newline is outside of it. So 3b, newline, is always rendered. But when {% if %} block evaluates to true we also get 3a, if it's false then we get 3b only.

It’s also worth pointing out that if your template continued after {% endfor %} block, that block would contribute one extra newline. But worry not, we’ll have some examples later on illustrating this case.

I hope you’ll agree with me that the template we used in our example wasn’t especially big or complicated, yet it resulted in a fair amount of additional whitespaces.

Luckily, and I couldn’t stress enough how useful that is, there are ways of changing Jinja2 behavior and taking back control over exact look and feel of our text.

Note. The above explanation was updated on 12 Dec 2020. Previously 1st occurence of 3b was incorrectly attributed to 2b. Many thanks to Lawrr who triple-checked me and greatly helped in getting to the bottom of this!

Finding origin of whitespaces - alternative way

We’ve talked a bit how to tame Jinja’s engine with regards to whitespace generation. You also know that tools like J2Live can help you in visualizing all the whitespaces in the produced text. But can we tell with certainty which template line, containing block, contributed these characters to the final render?

To get answer to that question we can use a little trick. I came up with the following technique, that doesn’t require any external tools, for matching whitespaces coming from template block lines with extraneous whitespaces appearing in the resulting text document.

This method is quite simple really, you just need to add unambiguous characters to each of the block lines in the template that correspond to lines in the rendered document.

I find it works especially well with template inheritance and macros, topics we will discuss in the upcoming parts of this tutorial.

Origin of whitespaces - examples

Let’s see that secret sauce in action then. We’ll place additional characters, carefully selected so that they stand out from surrounding text, in strategic places on the lines with Jinja2 blocks. I’m using the same template we already worked with so previously that you can easily compare the results.

{% for iname, idata in interfaces.items() %}(1)
interface {{ iname }}
 description {{ idata.description }}
  (2){% if idata.ipv4_address is defined %}
 ip address {{ idata.ipv4_address }}
  (3){% endif %}
{% endfor %}

Final result:

(1)
interface Ethernet1
 description capture-port
  (2)
(1)
interface Ethernet2
 description leaf01-eth51
  (2)
 ip address 10.50.0.0/31
  (3)

I added (1), (2) and (3) characters on the lines where we have Jinja2 blocks. The end result matches what we got back from J2Live with Show whitespaces option enabled.

If you don’t have access to J2Live or you need to troubleshoot whitespace placement in production templates, then I definitely recommend using this method. It’s simple but effective.

Just to get more practice, I’ve added extra characters to slightly more complex template. This one has branching if statement and some text below final endfor to allow us to see what whitespaces come from that block.

Our template:

{% for acl, acl_lines in access_lists.items() %}(1)
ip access-list extended {{ acl }}
  {% for line in acl_lines %}(2)
    (3){% if line.action == "remark" %}
    remark {{ line.text }}
    (4){% elif line.action == "permit" %}
    permit {{ line.src }} {{ line.dst }}
    (5){% endif %}
  {% endfor %}(6)
{% endfor %}(7)

# All ACLs have been generated

Data used to render it:

access_lists:
  al-hq-in:
    - action: remark
      text: Allow traffic from hq to local office
    - action: permit
      src: 10.0.0.0/22
      dst: 10.100.0.0/24

End result:

(1)
ip access-list extended al-hq-in
  (2)
    (3)
    remark Allow traffic from hq to local office
    (4)
  (2)
    (3)
    permit 10.0.0.0/22 10.100.0.0/24
    (5)
  (6)
(7)

# All ACLs have been generated

A lot is happening here but there are no mysteries anymore. You can easily match each source line with line in the final text. And knowing where the whitespaces are coming from is the first step to learning how to control them, which is what we’re going to talk about shortly.

Also, for comparison is the rendered text without using helper characters:


ip access-list extended al-hq-in
  
    
    remark Allow traffic from hq to local office
    
  
    
    permit 10.0.0.0/22 10.100.0.0/24
    
  


# All ACLs have been generated

If you’re still reading this, congratulations! Your dedication to mastering whitespace rendering is commendable. Good news is that we’re now getting to the bit where we learn how to control Jinja2 behavior.

Controlling Jinja2 whitespaces

There are broadly three ways in which we can control whitespace generation in our templates:

  1. Enable one of, or both, trim_blocks and lstrip_blocks rendering options.
  2. Manually strip whitespaces by adding a minus sign - to the start or end of the block.
  3. Apply indentation inside of Jinja2 blocks.

First, I’ll give you an easy, by far more preferable, way of taming whitespace and then we’ll dig into the more involving methods.

So here it comes:

Always render with trim_blocks and lstrip_blocks options enabled.

That’s it, the big secret is out. Save yourself trouble and tell Jinja2 to apply trimming and stripping to all of the blocks.

If you use Jinja2 as part of another framework then you might have to consult documentation to see what the default behaviour is and how it can be changed. Later in this post I will explain how we can control whitespaces when using Ansible to render Jinja2 templates.

Just a few words of explanation on what these options do. Trimming removes newline after block while stripping removes all of spaces and tabs on the lines preceding the block. Now, if you enable trimming alone, you might still get some funny output if there are any leading whitespaces on the lines containing blocks, so that’s why I recommend having both of these enabled.

Trimming and stripping in action

For example, this is what happens when we enable block trimming but leave block stripping disabled:

ip access-list extended al-hq-in
          remark Allow traffic from hq to local office
              permit 10.0.0.0/22 10.100.0.0/24
      
# All ACLs have been generated

That’s the same example we just had a look at, and I’m sure you didn’t expect this to happen at all. Let’s add some extra characters to figure out what happened:

{% for acl, acl_lines in access_lists.items() %}
ip access-list extended {{ acl }}
  {% for line in acl_lines %}
    (3){% if line.action == "remark" %}
    remark {{ line.text }}
    (4){% elif line.action == "permit" %}
    permit {{ line.src }} {{ line.dst }}
    {% endif %}
  {% endfor %}
{% endfor %}

# All ACLs have been generated

ip access-list extended al-hq-in
      (3)    remark Allow traffic from hq to local office
    (4)      (3)    permit 10.0.0.0/22 10.100.0.0/24
      
# All ACLs have been generated

Another puzzle solved, we got rid of newlines with trim_blocks enabled but leading spaces in front of if and elif blocks remained. Something that is completely undesirable.

So how would this template render if we had both trimming and stripping enabled? Have a look:

ip access-list extended al-hq-in
    remark Allow traffic from hq to local office
    permit 10.0.0.0/22 10.100.0.0/24

# All ACLs have been generated

Quite pretty right? This is what I meant when I talked about getting intended result. No surprises, no extra newlines or spaces, final text matches our expectations.

Now, I said enabling trim and lstrip options is an easy way, but if for whatever reason you can’t use it, or want to have total control over how whitespaces are generated on a per-block then we need to resort to manual control.

Manual control

Jinja2 allows us to manually control generation of whitespaces. You do it by using a minus sing - to strip whitespaces from blocks, comments or variable expressions. You need to add it to the start or end of given expression to remove whitespaces before or after the block, respectively.

As always, it’s best to learn from examples. We’ll go back to example from the beginning of the post. First we render without any - signs added:

{% for iname, idata in interfaces.items() %}
interface {{ iname }}
 description {{ idata.description }}
  {% if idata.ipv4_address is defined %}
 ip address {{ idata.ipv4_address }}
  {% endif %}
{% endfor %}

Result:

Right, some extra newlines and there are additional spaces as well. Let’s add minus sign at the end of for block:

{% for iname, idata in interfaces.items() -%}
interface {{ iname }}
 description {{ idata.description }}
  {% if idata.ipv4_address is defined %}
 ip address {{ idata.ipv4_address }}
  {% endif %}
{% endfor %}

Looks promising, we removed two of the extra newlines.

Next we look at if block. We need to get rid of the newlines this block generates so we try adding - at the end, just like we did with for block.

{% for iname, idata in interfaces.items() -%}
interface {{ iname }}
 description {{ idata.description }}
  {% if idata.ipv4_address is defined -%}
 ip address {{ idata.ipv4_address }}
  {% endif %}
{% endfor %}

Newline after line with description under Ethernet2 is gone. Oh, but wait, why do we have two spaces in the line with ip address now? Aha! These must’ve been the two spaces preceding the if block. Let’s just add - to the beginning of that block as well and we’re done!

{% for iname, idata in interfaces.items() -%}
interface {{ iname }}
 description {{ idata.description }}
  {%- if idata.ipv4_address is defined -%}
 ip address {{ idata.ipv4_address }}
  {% endif %}
{% endfor %}

Hmm, now it’s all broken! What happened here? A very good question indeed.

So here’s the thing. These magical minus signs remove all of the whitespaces before or after the block, not just whitespaces on the same line. Not sure if you expected that, I certainly did not when I first used manual whitespace control!

In our concrete case, the first - we added to the end of if block stripped newline AND one space on the next line, the one before ip address*. Because, if we now look closely, we should’ve had three spaces there not just two. One space that we placed there ourselves and two spaces that we had in front of the if block. But that space placed by us was removed by Jinja2 due to - sign placed in the if block.

Not all is lost though. You might notice that just adding - at the beginning of if and endif blocks will render text as intended. Let’s try doing that and see what happens.

{% for iname, idata in interfaces.items() -%}
interface {{ iname }}
 description {{ idata.description }}
  {%- if idata.ipv4_address is defined %}
 ip address {{ idata.ipv4_address }}
  {%- endif %}
{% endfor %}

Result:

Bingo! We got rid of all those pesky whitespaces! But was this easy and intuitive? Not really. And to be fair, this wasn’t a very involving example. Manually controlling whitespaces is certainly possible but you must remember that all whitespaces are removed, and just the ones on the same line as the block.

Indentation inside of Jinja2 blocks

There is a method of writing blocks that makes things a bit easier and predictable. We simply put opening of the block at the beginning of the line and apply indentation inside of the block. As always, easier to explain using an example:

{% for acl, acl_lines in access_lists.items() %}
ip access-list extended {{ acl }}
{%   for line in acl_lines %}
{%     if line.action == "remark" %}
  remark {{ line.text }}
{%     elif line.action == "permit" %}
  permit {{ line.src }} {{ line.dst }}
{%     endif %}
{%   endfor %}
{% endfor %}

# All ACLs have been generated

As you can see we moved block opening {% all the way to the left and then indented as appropriate inside of the block. Jinja2 doesn’t care about extra spaces inside of the if or for blocks, it will simply ignore them. It only concerns itself with whitespaces that it finds outside of blocks.

Let’s render this to see what we get:


ip access-list extended al-hq-in
  
    
  remark Allow traffic from hq to local office
    
  
    
  permit 10.0.0.0/22 10.100.0.0/24
    
  


# All ACLs have been generated

How is this any better you may ask? At first sight, not much better at all. But before I tell you why this might be a good idea, and where it is especially useful I’ll show you the same template as we had it previously.

We’ll render it with trim_blocks enabled:

{% for acl, acl_lines in access_lists.items() %}
ip access-list extended {{ acl }}
  {% for line in acl_lines %}
    {% if line.action == "remark" %}
  remark {{ line.text }}
    {% elif line.action == "permit" %}
  permit {{ line.src }} {{ line.dst }}
    {% endif %}
  {% endfor %}
{% endfor %}

# All ACLs have been generated

ip access-list extended al-hq-in
        remark Allow traffic from hq to local office
            permit 10.0.0.0/22 10.100.0.0/24
      
# All ACLs have been generated

Terrible, just terrible. Indentation got completely out of whack. But what am I trying to show you? Well, let’s now render version of this template with indentations inside of for and if blocks, again with trim_blocks turned on:

ip access-list extended al-hq-in
  remark Allow traffic from hq to local office
  permit 10.0.0.0/22 10.100.0.0/24

# All ACLs have been generated

Isn’t that nice? Remember that previously we had to enable both trim_blocks and lstrip_blocks to achieve the same effect.

So here it is:

Starting Jinja2 blocks from the beginning of the line and applying indentation inside of them is roughly equivalent to enabling lstrip_block.

I say roughly equivalent because we don't strip anything here, we just hide extra spaces inside of blocks preventing them from being picked up at all.

And there is an extra bonus to using this method, it will make your Jinja2 templates used in Ansible safer. Why? Read on!

Whitespace control in Ansible

As you probably already know Jinja2 templates are used quite heavily when doing network automation with Ansible. Most people will use Ansible’s template module to do the rendering of templates. That module by default enables trim_blocks option but lstrip_blocks is turned off and needs to be enabled manually.

We can assume that most users will use the template module with default options which means that using indentation inside of the block technique will increase safety of our templates and the rendered text.

For the above reasons I’d recommend applying this technique if you know your templates will be used in Ansible. You will greatly reduce risk of your templates having seemingly random whitespaces popping up in your configs and other documents.

I would also say that it’s not a bad idea to always stick to this way of writing your blocks if you haven’t yet mastered the arcane ways of Jinja2. There are no real downsides of writing your templates this way.

The only side effect here is how visually templates present themselves, with a lot of blocks templates looking “busy”. This could make it difficult to see lines of text between the blocks since these need to have indentation matching your intent.

Personally I always try to use indentation within blocks method in templates meant for Ansible. For other templates, when rendered with Python, I do whatever feels right in terms of readability, and I render all templates with block trimming and stripping enabled.

Example Playbooks

For completeness sake, I built two short Ansible Playbooks, one uses default setting for template module while the other enables lstrip option.

We’ll be using the same template and data we used for testing trim and lstrip options previously.

Playbook using default settings, i.e. only trim is turned on:

---
- hosts: localhost
  gather_facts: no
  connection: local

  vars_files:
    - vars/access-lists.yml


  tasks:
    - name: Show vars
      debug:
        msg: "{{ access_lists }}"

    - name: Render config for host
      template:
        src: "templates/ws-access-lists.j2"
        dest: "out/ws-default.cfg"

And rendering results:

ip access-list extended al-hq-in
          remark Allow traffic from hq to local office
              permit 10.0.0.0/22 10.100.0.0/24
      
# All ACLs have been generated

If you recall, we got exactly same result when rendering this template in Python with trim option enabled. Again, indentations are misaligned so we need to do better.

Playbook enabling lstrip:

---
- hosts: localhost
  gather_facts: no
  connection: local

  vars_files:
    - vars/access-lists.yml


  tasks:
    - name: Show vars
      debug:
        msg: "{{ access_lists }}"

    - name: Render config for host
      template:
        src: "templates/ws-access-lists.j2"
        dest: "out/ws-lstrip.txt"
        lstrip_blocks: yes

Rendered text:

ip access-list extended al-hq-in
    remark Allow traffic from hq to local office
    permit 10.0.0.0/22 10.100.0.0/24

# All ACLs have been generated

And again, same result as when you enabled trim and lstrip when rendering Jinja2 in Python.

Finally, let’s run the first Playbook, with default setting, using the template with indentation inside of blocks.

Playbook:

---
- hosts: localhost
  gather_facts: no
  connection: local

  vars_files:
    - vars/access-lists.yml


  tasks:
    - name: Show vars
      debug:
        msg: "{{ access_lists }}"

    - name: Render config for host
      template:
        src: "templates/ws-bi-access-lists.j2"
        dest: "out/ws-block-indent.txt"

Result:

ip access-list extended al-hq-in
  remark Allow traffic from hq to local office
  permit 10.0.0.0/22 10.100.0.0/24

# All ACLs have been generated

So, we didn’t have to enable lstrip option to get the same, perfect, result. Hopefully now you can see why I recommend using indentation within blocks as the default for Ansible templates. This gives you more confidence that your templates will be rendered the way you wanted them with default settings.

Closing thoughts

When I sat down to write this post I thought I knew how whitespaces in Jinja2 work. But it turns out that some behaviour was not so clear to me. It was especially true for manual stripping with - sign, I keep forgetting that all of the whitespaces before/after the block are stripped, not just the ones on the line with block.

So my advice is this: use trimming and stripping options whenever possible and generally favour indentation within blocks over indentation outside. And spend some time learning how Jinja2 generates whitespaces, that will allow you to take full control over your templates when you need it.

And that's it, I hope you found this post useful and I look forward to seeing you again!

References: