Data … as usual

Linux redhat’s based systems (like Oracle Linux) come with systemd init system that is used to bootstrap the user space and to manage system processes after booting.

Systemd has been created to reduce boot time (and shutdown time) of your server, but sometime, you can have a process that slows down the boot process and, in order to diagnose it, you can use a very interesting command to do that stuff.

The command is systemd-analyze.

With no argument, it will help you to know in how many time your server starts-up and how many times it spent on each phase (kernel, initrd, and user space) :

[root@oel7 ~]# systemd-analyze
Startup finished in 1.942s (kernel) + 6.141s (initrd) + 1min 44.688s (userspace) = 1min 52.772s

Another interesting option is the “blame” option that will give you the list of running systemd units ordered by time to initialize (remark: I used the no-pager switch to disable results paging)

[root@oel7 ~]# systemd-analyze blame --no-pager
    1min 15.289s kdump.service
         50.119s plymouth-quit-wait.service
         37.486s vboxadd.service
          8.134s accounts-daemon.service
          6.110s tuned.service
          5.179s systemd-logind.service
          5.092s libvirtd.service
          5.026s NetworkManager-wait-online.service
          4.873s systemd-udev-settle.service
          4.870s lvm2-monitor.service
          4.863s ModemManager.service
          4.749s ksm.service
          4.742s rhel-dmesg.service
          4.316s postfix.service
          4.052s NetworkManager.service
          3.924s dev-mapper-ol\x2droot_lv.device
          3.678s abrt-ccpp.service
          3.200s rtkit-daemon.service
          3.087s polkit.service
          2.937s gssproxy.service
          1.875s lvm2-pvscan@8:2.service
          1.508s rsyslog.service
          1.505s network.service
          1.425s packagekit.service
          1.420s sysstat.service
          1.401s sshd.service
          1.374s systemd-tmpfiles-setup-dev.service
          1.214s avahi-daemon.service
          1.109s netcf-transaction.service
          1.063s proc-fs-nfsd.mount
          1.014s chronyd.service
           932ms rhel-readonly.service
           913ms rhnsd.service
           877ms systemd-udevd.service
           842ms dev-mapper-ol\x2dswap_lv.swap
           800ms auditd.service
           778ms plymouth-read-write.service
           750ms u01.mount
           746ms systemd-tmpfiles-setup.service
           709ms systemd-vconsole-setup.service
           692ms systemd-fsck@dev-mapper-ol\x2dorasoft_lv.service
           638ms systemd-fsck@dev-mapper-ol\x2doradata_lv.service
           632ms systemd-journald.service
           631ms u02.mount
.../...

With this command, you can display aa tree of every steps in the init critical chain and how many time it took:

[root@oel7 ~]# systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

graphical.target @1min 44.678s
└─multi-user.target @1min 44.678s
  └─vboxadd-service.service @50.648s +372ms
    └─vboxadd.service @13.158s +37.486s
      └─basic.target @12.969s
        └─paths.target @12.969s
          └─cups.path @12.968s
            └─sysinit.target @12.509s
              └─systemd-update-utmp.service @12.403s +105ms
                └─auditd.service @11.602s +800ms
                  └─systemd-tmpfiles-setup.service @10.719s +746ms
                    └─rhel-import-state.service @10.120s +590ms
                      └─local-fs.target @10.118s
                        └─run-user-0.mount @1min 8.832s
                          └─local-fs-pre.target @9.141s
                            └─lvm2-monitor.service @4.269s +4.870s
                              └─lvm2-lvmetad.service @5.220s
                                └─lvm2-lvmetad.socket @4.267s
                                  └─-.slice

The last but not least option offers the possibility to generate a SVG diagram (readable in a web browser) to visualize all the details of the bootstrap and daemons initialisation.

To do that you have to use the plot option:

[root@oel7 ~]# systemd-analyze plot > t.html

Then, you will get a nice plot in your browser to easily identify which process took the major part of the boot time:

For information, a legend is displayed on the page bottom.

That’s it for today 😉