Making Linux host tuning repeatable with Ansible

This is the third post in a small series. The low-latency Linux tuning checklist covers what to change. How to measure latency without fooling yourself covers how to tell if it worked. This one covers how to make the change stick, on every host, every time.

I use Ansible for this. The examples below are a cut-down version of the pattern: one role, an inventory grouped by environment (prd, stg, dev), settings in variables, and checks that read the live state after reboot.

1. Why hand-tuning drifts

Say you tune one host by hand. You edit GRUB, set a few sysctls, pin some IRQs, measure, and it looks good. Six months later:

The host was rebuilt and half the settings are gone.
A kernel upgrade regenerated the GRUB config.
irqbalance came back after a package update and moved your IRQs.
Someone copied the settings to a new instance type with a different core count, so isolcpus now points at the wrong CPUs.

None of these fail loudly. The host boots, the app runs, and the p99.9 gets worse. You find out from a latency graph, not from an error.

The fix is boring: settings live in code, a tool applies them, and something checks the result.

2. Layout

inventory/
  hosts.yml
  group_vars/
    all.yml
    prd.yml
    stg.yml
    dev.yml
  host_vars/
    host-p01.yml            # sets the host type; little else
roles/
  lowlat_tuning/
    defaults/main.yml
    vars/
      c6i_8xl.yml           # one file per host type
    tasks/
      main.yml
      grub.yml
      sysctl.yml
      tuned.yml
      irq.yml
      hugepages.yml
      verify.yml
    handlers/main.yml
    templates/
      tuned.conf.j2
      irq-affinity.sh.j2
      irq-affinity.service.j2
site.yml

The inventory is grouped by environment: prd, stg, dev. That matches how changes move. You try a change in dev, then stg, then prd, and the inventory makes each step a single --limit.

# inventory/hosts.yml
all:
  children:
    lowlat:
      children:
        dev:
          hosts:
            host-d01:
        stg:
          hosts:
            host-s01:
            host-s02:
        prd:
          hosts:
            host-p01:
            host-p02:
            host-p03:

Per-environment group_vars hold what really differs between environments: how careful the play is, not what the tuning is.

# inventory/group_vars/dev.yml
lowlat_require_change_window: false   # dev can change any time
lowlat_allow_reboot: true             # rebooting dev is fine

# inventory/group_vars/prd.yml
lowlat_require_change_window: true    # prd only inside an agreed window
lowlat_allow_reboot: false            # reboot only when passed on the command line

stg.yml usually sits close to prd, so stg rehearses the prd procedure.

The tuning values themselves (CPU lists, HugePages) depend on the hardware, not the environment: core count, NUMA layout, SMT. So they live in the role, one vars file per host type, and each host says which type it is:

# inventory/host_vars/host-p01.yml
lowlat_host_type: c6i_8xl

# roles/lowlat_tuning/vars/c6i_8xl.yml
# Why: CPUs 0-1 (and SMT siblings) keep housekeeping; 2-15 for the app.
lowlat_housekeeping_cpus: "0-1,16-17"
lowlat_isolated_cpus: "2-15,18-31"
lowlat_hugepages_2m: 2048          # 4 GiB, sized from the app's working set
lowlat_tuned_profile: lowlat-trading
lowlat_extra_cmdline: "processor.max_cstate=1 intel_idle.max_cstate=0"

# roles/lowlat_tuning/tasks/main.yml (first task)
- name: Load host type values
  ansible.builtin.include_vars: "{{ lowlat_host_type }}.yml"
  when: lowlat_host_type is defined

The point: a dev host and a prd host of the same type get identical tuning. If they didn’t, testing in dev and stg would tell you nothing about prd. If a host needs more in host_vars than its type, ask why first.

Keep one comment per setting: what it does and why. The checklist post says the same thing. A setting nobody can explain is a setting nobody dares remove.

defaults/main.yml holds safe defaults, so the role does nothing surprising if a variable is missing:

# roles/lowlat_tuning/defaults/main.yml
lowlat_isolated_cpus: ""
lowlat_housekeeping_cpus: "0"
lowlat_hugepages_2m: 0
lowlat_extra_cmdline: ""
lowlat_allow_reboot: false
lowlat_require_change_window: true
lowlat_sysctls: {}

3. GRUB command line

Boot parameters are the hardest part, because they only take effect after a reboot.

# roles/lowlat_tuning/tasks/grub.yml
- name: Build the tuning cmdline
  ansible.builtin.set_fact:
    lowlat_cmdline: >-
      isolcpus=managed_irq,domain,{{ lowlat_isolated_cpus }}
      nohz_full={{ lowlat_isolated_cpus }}
      rcu_nocbs={{ lowlat_isolated_cpus }}
      irqaffinity={{ lowlat_housekeeping_cpus }}
      {{ lowlat_extra_cmdline }}
  when: lowlat_isolated_cpus | length > 0

- name: Write tuning params to a GRUB drop-in
  ansible.builtin.copy:
    dest: /etc/default/grub.d/99-lowlat.cfg
    content: |
      # Managed by Ansible (lowlat_tuning). Do not edit by hand.
      GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX {{ lowlat_cmdline | trim }}"
    mode: "0644"
  when: lowlat_cmdline is defined
  notify:
    - regenerate grub
    - reboot required

Why a drop-in file instead of editing /etc/default/grub with lineinfile? Because a regex against one long line is fragile, and package updates touch that file. A separate file owned by Ansible is easy to diff and easy to remove.

The catch: not every distro reads /etc/default/grub.d/. Debian and Ubuntu do. RHEL-family hosts use grubby and BLS entries instead, so there you’d run grubby --update-kernel=ALL --args=... from a command task with a check for the current args. Pick one path per distro and test it.

Handlers:

# roles/lowlat_tuning/handlers/main.yml
- name: regenerate grub
  ansible.builtin.command: update-grub
  changed_when: true

- name: reboot required
  ansible.builtin.file:
    path: /var/run/lowlat-reboot-required
    state: touch
    mode: "0644"

- name: restart irqbalance
  ansible.builtin.systemd:
    name: irqbalance
    state: restarted

- name: restart irq affinity
  ansible.builtin.systemd:
    name: irq-affinity.service
    state: restarted
    daemon_reload: true

Note the reboot handler doesn’t reboot. It drops a marker file. The actual reboot is a separate, deliberate step (section 9). A role that reboots production hosts on its own is a role someone will run at the wrong time.

4. sysctls

# inventory/group_vars/all.yml (same in every environment)
lowlat_sysctls:
  vm.swappiness: 1               # avoid swapping app memory
  kernel.numa_balancing: 0       # stop the kernel migrating pages under us
  vm.stat_interval: 10           # fewer vmstat wakeups on all CPUs

# roles/lowlat_tuning/tasks/sysctl.yml
- name: Apply sysctls
  ansible.posix.sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    sysctl_file: /etc/sysctl.d/90-lowlat.conf
    sysctl_set: true
    state: present
  loop: "{{ lowlat_sysctls | dict2items }}"

sysctl_set: true applies the value now and writes it to the file, so it survives a reboot. ansible.posix is a separate collection; pin its version in requirements.yml.

5. A custom tuned profile

tuned bundles a lot of settings (governor, C-states, some sysctls) into one profile. Make your own profile that inherits a stock one, rather than editing the stock one.

# roles/lowlat_tuning/templates/tuned.conf.j2
# Managed by Ansible
[main]
summary=Low-latency profile for {{ group_names | join(',') }}
include=network-latency

[cpu]
force_latency=cstate.id_no_zero:1
governor=performance

[sysctl]
kernel.sched_rt_runtime_us=-1

# roles/lowlat_tuning/tasks/tuned.yml
- name: Install tuned
  ansible.builtin.package:
    name: tuned
    state: present

- name: Create profile directory
  ansible.builtin.file:
    path: "/etc/tuned/{{ lowlat_tuned_profile }}"
    state: directory
    mode: "0755"

- name: Template profile
  ansible.builtin.template:
    src: tuned.conf.j2
    dest: "/etc/tuned/{{ lowlat_tuned_profile }}/tuned.conf"
    mode: "0644"
  register: tuned_profile

- name: Get active profile
  ansible.builtin.command: tuned-adm active
  register: tuned_active
  changed_when: false
  check_mode: false

- name: Activate profile
  ansible.builtin.command: "tuned-adm profile {{ lowlat_tuned_profile }}"
  when: >
    tuned_profile is changed or
    lowlat_tuned_profile not in tuned_active.stdout

The tuned-adm active read plus a when is what keeps this idempotent. A bare command task reports “changed” on every run, and then you can’t tell real changes from noise.

Watch for overlap. tuned’s cpu-partitioning profile also sets isolcpus and nohz_full on the command line. If you use that profile, don’t also set them in GRUB, or you’ll have two sources of truth that disagree.

6. irqbalance and IRQ affinity

Two options. Turn irqbalance off and pin everything yourself, or keep it and tell it to stay off the isolated CPUs. The second is less work:

# roles/lowlat_tuning/tasks/irq.yml
- name: Keep irqbalance off isolated CPUs
  ansible.builtin.lineinfile:
    path: /etc/default/irqbalance     # /etc/sysconfig/irqbalance on RHEL
    regexp: '^#?IRQBALANCE_BANNED_CPULIST='
    line: 'IRQBALANCE_BANNED_CPULIST={{ lowlat_isolated_cpus }}'
  notify: restart irqbalance

For IRQs you want on specific cores, like NIC queues, use a small script run by a systemd unit at boot:

# roles/lowlat_tuning/templates/irq-affinity.service.j2
[Unit]
Description=Pin IRQs for low-latency host (managed by Ansible)
After=network-online.target irqbalance.service
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/irq-affinity.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

# roles/lowlat_tuning/templates/irq-affinity.sh.j2
#!/bin/bash
set -euo pipefail
# Pin every IRQ for {{ lowlat_nic }} to housekeeping CPUs.
for irq in $(grep '{{ lowlat_nic }}' /proc/interrupts | awk -F: '{print $1}'); do
  echo "{{ lowlat_housekeeping_cpus }}" > "/proc/irq/${irq}/smp_affinity_list" || true
done

Why a unit and not an Ansible task that writes /proc/irq/* directly? /proc doesn’t survive a reboot. Ansible sets the desired state; the unit re-applies it on every boot, even if Ansible never runs again.

The || true is there because some IRQs can’t be moved and the write fails. The verify step (section 7) is what tells you if the ones you care about didn’t move.

7. HugePages

# roles/lowlat_tuning/tasks/hugepages.yml
- name: Reserve 2 MiB HugePages
  ansible.posix.sysctl:
    name: vm.nr_hugepages
    value: "{{ lowlat_hugepages_2m }}"
    sysctl_file: /etc/sysctl.d/91-lowlat-hugepages.conf
    sysctl_set: true
  when: lowlat_hugepages_2m | int > 0

Setting this at runtime can fall short on a host that’s been up a while, because memory is fragmented and the kernel can’t find enough contiguous pages. If you need a guaranteed count, add hugepages=N to the GRUB cmdline instead so it’s reserved at boot. Either way, check the number you actually got, not the number you asked for.

8. Verify the live state, and fail loudly

This is the part people skip, and it’s the part that matters. Files on disk say what you asked for. /proc and /sys say what you got.

# roles/lowlat_tuning/tasks/verify.yml
- name: Read live state
  ansible.builtin.slurp:
    src: "{{ item }}"
  loop:
    - /proc/cmdline
    - /sys/devices/system/cpu/isolated
    - /sys/devices/system/cpu/nohz_full
    - /proc/sys/vm/nr_hugepages
  register: live
  check_mode: false

- name: Index live state
  ansible.builtin.set_fact:
    lowlat_live: "{{ dict(live.results | map(attribute='item')
                    | zip(live.results | map(attribute='content')
                    | map('b64decode') | map('trim'))) }}"

- name: Assert boot params are active
  ansible.builtin.assert:
    that:
      - "'nohz_full=' ~ lowlat_isolated_cpus in lowlat_live['/proc/cmdline']"
      - "'rcu_nocbs=' ~ lowlat_isolated_cpus in lowlat_live['/proc/cmdline']"
    fail_msg: >-
      Tuning cmdline not active. Reboot pending?
      Got: {{ lowlat_live['/proc/cmdline'] }}

- name: Assert HugePages reserved
  ansible.builtin.assert:
    that:
      - lowlat_live['/proc/sys/vm/nr_hugepages'] | int >= lowlat_hugepages_2m | int
    fail_msg: "Wanted {{ lowlat_hugepages_2m }} HugePages, got {{ lowlat_live['/proc/sys/vm/nr_hugepages'] }}"

One trap: /sys/devices/system/cpu/isolated prints a normalised range. If you wrote 2-15,18-31 it may print the same, but 2,3,4-15 comes back as 2-15. Compare against a normalised form, or keep your variables in the normalised form to begin with.

Check IRQ affinity too:

- name: Read NIC IRQ affinity
  ansible.builtin.shell: |
    set -o pipefail
    for irq in $(grep '{{ lowlat_nic }}' /proc/interrupts | awk -F: '{print $1}'); do
      echo "$irq $(cat /proc/irq/$irq/effective_affinity_list)"
    done
  args:
    executable: /bin/bash
  register: nic_irqs
  changed_when: false
  check_mode: false

- name: Show NIC IRQ placement
  ansible.builtin.debug:
    var: nic_irqs.stdout_lines

effective_affinity_list shows where the IRQ actually lands, which can differ from smp_affinity_list (what was requested). Turning that list into an assert against the housekeeping set takes a few more lines of Jinja; worth doing once and reusing.

Also add the governor, C-state and tuned profile from the checklist post. Anything you tuned, verify.

9. Rolling it out safely

# site.yml
- name: Low-latency host tuning
  hosts: lowlat
  become: true
  serial: 1
  max_fail_percentage: 0
  pre_tasks:
    - name: Refuse to run outside the change window
      ansible.builtin.assert:
        that: >-
          not lowlat_require_change_window | bool
          or lowlat_change_window_open | default(false) | bool
        fail_msg: "Change window closed. Set lowlat_change_window_open=true."
  roles:
    - lowlat_tuning
  post_tasks:
    - name: Check for pending reboot
      ansible.builtin.stat:
        path: /var/run/lowlat-reboot-required
      register: reboot_flag

    - name: Reboot if needed and allowed
      ansible.builtin.reboot:
        reboot_timeout: 900
        post_reboot_delay: 30
      when:
        - reboot_flag.stat.exists
        - lowlat_allow_reboot | bool

    - name: Verify after reboot
      ansible.builtin.include_role:
        name: lowlat_tuning
        tasks_from: verify

What each part does:

serial: 1: one host at a time. If the first host breaks, the rest are untouched.
max_fail_percentage: 0: any failure stops the play.
The window assert: a crude but effective guard. It only bites where group_vars say so, so dev runs freely and stg and prd need the window flag.
The reboot flag: dev’s group_vars allow reboots. In stg and prd the reboot only happens if you pass -e lowlat_allow_reboot=true on purpose.
Verify after reboot: the play only moves to the next host if the live state matches.

/var/run is a tmpfs on most systems, so the reboot marker clears itself on reboot. Handy.

Promote the change one environment at a time, with the same play and the same commit:

ansible-playbook -i inventory/hosts.yml site.yml --limit dev
ansible-playbook -i inventory/hosts.yml site.yml --limit stg \
  -e lowlat_change_window_open=true -e lowlat_allow_reboot=true
ansible-playbook -i inventory/hosts.yml site.yml --limit prd \
  -e lowlat_change_window_open=true -e lowlat_allow_reboot=true

Don’t move to the next environment until the previous one has verified and been measured (section 13). If prd needs a different value than stg, that’s a sign the tuning belongs in the host type file, or that stg isn’t the same hardware as prd.

Before the first real prd host, drain it from whatever sends it traffic. That part is app-specific and lives outside this role.

10. Check mode, –diff and idempotence

Before applying anything:

ansible-playbook -i inventory/hosts.yml site.yml \
  --limit stg --check --diff

--check runs the play without changing anything. --diff shows what would change in each file. Read the diff. That’s your change review.

Two things make check mode useful:

Read-only tasks need check_mode: false. Otherwise slurp and command reads are skipped and later tasks have nothing to work with.
command tasks need changed_when. Otherwise they show “changed” every time.

The goal is idempotence: run the play twice, and the second run reports zero changes. If it doesn’t, some task is lying, and you can’t use “changed” as a drift signal.

11. Testing with Molecule or a test host

Molecule runs a role against a throwaway instance, then runs it again to check idempotence:

molecule test    # create, converge, idempotence, verify, destroy

The limit: containers share the host kernel. You can test that files get written and the second run is clean, but not that isolcpus took effect after a reboot. For that you need a real VM or a real instance of the same type as prd. That’s what the dev group is for: at least one dev host per host type, tuned with the same role, rebooted, verified, then measured.

12. Drift detection

The same play in check mode, on a schedule, is a drift detector:

ansible-playbook -i inventory/hosts.yml site.yml \
  --limit prd --check --diff -e lowlat_change_window_open=true \
  | tee /var/log/lowlat-drift-$(date +%F).log

Run it from a scheduler (cron, a CI job, AWX). Alert if the recap shows changed > 0 or failed > 0 on any host. Changed means a file drifted. Failed means the live state doesn’t match, which often means a pending reboot nobody did.

Pair it with the boot-time check from the checklist post. The scheduled run catches config drift; the boot check catches the case where the host came up wrong.

13. Tie it back to measurement

Ansible tells you the settings are in place. It doesn’t tell you they helped. After a rollout, on at least one host per type:

Run the same cyclictest or hwlat check you ran before.
Run the app benchmark the same way, as in the measurement post: same load, same duration, full histograms, compare the tail.
Keep the numbers next to the commit that changed the settings.

Then a setting has a reason, a test, and a result in one place. That’s what makes it safe to change later.

Checklist

Inventory grouped by environment (prd, stg, dev); per-environment group_vars for safety settings
CPU lists per host type in the role, with a comment per setting
GRUB params in a drop-in (or grubby), via handlers, no silent reboots
sysctls via ansible.posix.sysctl with a file under /etc/sysctl.d/
Custom tuned profile that inherits a stock one; no overlap with GRUB
irqbalance kept off isolated CPUs; IRQ pinning in a systemd unit
HugePages reserved and actual count checked
Verify tasks read /proc and /sys and fail loudly
serial: 1, explicit reboot flag, verify after reboot; promote dev → stg → prd
--check --diff before every change; second run reports zero changes
Molecule for the role, a real test instance for boot params
Scheduled check-mode run for drift
Before/after measurements stored with the change