Wood's Words2022-11-16T18:42:05+00:00https://www.natewoodward.org/blogNate Woodwardnate.blog@natewoodward.orgUpdate rkhunter After SUSE Package Updates2022-08-09T18:08:14+00:00https://www.natewoodward.org/blog/2021/10/15/zypper-rkhunter<p>How to suppress rkhunter false positives due to zypper package updates on openSUSE 15.4.</p>
<p><a href="https://www.getpagespeed.com/server-setup/security/sane-use-of-rkhunter-in-centos-7">Danila Vershinin’s blog post</a>
explains why one might want to do this and has instructions for CentOS 7.
The only problem I had with his setup is that I couldn’t find anything for SUSE that did what
<code class="language-plaintext highlighter-rouge">yum-plugin-post-transaction-actions</code> does, so I had to write a zypper plugin for that.</p>
<ol id="markdown-toc">
<li><a href="#setup" id="markdown-toc-setup">Setup</a></li>
<li><a href="#how-it-works" id="markdown-toc-how-it-works">How it Works</a></li>
</ol>
<h3 id="setup">Setup</h3>
<p>The daily system cron job to run rkhunter was removed in openSUSE 15.4,
so you will need to set one up yourself.
The files for the cron job that openSUSE 15.3 used are still available,
but were moved under /usr/share,
so we can simply copy those files to set up the job.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cp /usr/share/fillup-templates/sysconfig.rkhunter /etc/sysconfig/rkhunter
cp /usr/sharedoc/packages/rkhunter-1.4.6/rkhunter.cron /etc/cron.daily/suse.de-rkhunter
chmod +x /etc/cron.daily/suse.de-rkhunter
</code></pre></div></div>
<p>You may also want to edit <code class="language-plaintext highlighter-rouge">/etc/sysconfig/rkhunter</code> and adjust
<code class="language-plaintext highlighter-rouge">CRON_DB_UPDATE</code>, <code class="language-plaintext highlighter-rouge">REPORT_EMAIL</code>, etc.</p>
<p><a href="https://github.com/natewoodward/zypp-plugin-post-commit-actions">Install <code class="language-plaintext highlighter-rouge">zypp-plugin-post-commit-actions</code></a>.</p>
<p>Create <code class="language-plaintext highlighter-rouge">/etc/zypp/post-actions.d/rkhunter.action</code> with this line:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>*:any:echo $name >> /var/lib/rkhunter/changed-packages.dat
</code></pre></div></div>
<p>Create a script, <code class="language-plaintext highlighter-rouge">/etc/cron.daily/0rkhunter</code>, with these contents:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#!/bin/bash
pkglist=/var/lib/rkhunter/changed-packages.dat
touch "$pkglist"
while read pkg; do
/usr/bin/rkhunter --propupdate "$pkg" &>/dev/null
done < <(sort -u "$pkglist")
: > "$pkglist"
</code></pre></div></div>
<p>Make the script executable with <code class="language-plaintext highlighter-rouge">chmod +x /etc/cron.daily/0rkhunter</code> .</p>
<p>If all goes well, you’ll no longer get false positives like below for files that were changed due to a package being updated, installed, or removed:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Warning: The file properties have changed:
File: /bin/systemctl
Current inode: 133679 Stored inode: 71681
Warning: The file properties have changed:
File: /usr/bin/curl
Current inode: 135284 Stored inode: 72525
</code></pre></div></div>
<h3 id="how-it-works">How it Works</h3>
<p>Whenever a package is installed, updated, or removed with zypper,
we store the package name in <code class="language-plaintext highlighter-rouge">/var/lib/rkhunter/changed-packages.dat</code>
with the help of the zypper plugin.</p>
<p>Before the system cron job (<code class="language-plaintext highlighter-rouge">suse.de-rkhunter</code>) for rkhunter runs,
our <code class="language-plaintext highlighter-rouge">0rkhunter</code> cron script takes the stored package names
and runs <code class="language-plaintext highlighter-rouge">rkhunter --propupd $pkg</code> against each one.
Running rkhunter in this way updates the rkhunter database for files included in the RPM package named <code class="language-plaintext highlighter-rouge">$pkg</code>.
It does so using the file attributes from RPM’s database,
not from the the filesystem.</p>
<p>See <a href="https://www.getpagespeed.com/server-setup/security/sane-use-of-rkhunter-in-centos-7">Danila Vershinin’s blog post</a>
for a more thorough explanation of the pieces at play here.</p>
<!--
### Footnotes
[^1]: Credit goes to <user> for <whatever reasons>.
-->
Monitoring Java Memory Usage with Jstat and Check MK2020-08-11T20:13:44+00:00https://www.natewoodward.org/blog/2019/12/02/monitoring-java-memory-usage-with-jstat-and-check-mk<p>I recently wrote a script called
<a href="https://github.com/natewoodward/code-snippets/tree/master/check-mk"><code class="language-plaintext highlighter-rouge">check_jvm_memory</code></a>
to monitor JVM memory usage with <code class="language-plaintext highlighter-rouge">jstat</code> and Check MK.
Let’s go over how to use it.</p>
<ol id="markdown-toc">
<li><a href="#requirements" id="markdown-toc-requirements">Requirements</a></li>
<li><a href="#quick-setup" id="markdown-toc-quick-setup">Quick Setup</a></li>
<li><a href="#install" id="markdown-toc-install">Install</a></li>
<li><a href="#configure" id="markdown-toc-configure">Configure</a> <ol>
<li><a href="#name" id="markdown-toc-name">Name</a></li>
<li><a href="#pidcommand" id="markdown-toc-pidcommand">PidCommand</a></li>
<li><a href="#threshold" id="markdown-toc-threshold">Threshold</a></li>
<li><a href="#label" id="markdown-toc-label">Label</a></li>
</ol>
</li>
</ol>
<h3 id="requirements">Requirements</h3>
<p>The script runs as a <a href="https://checkmk.com/cms_localchecks.html">local check</a>,
so it requires the Check MK agent.</p>
<p>For obvious reasons, it requires that you have a running Java process that you want to monitor.</p>
<p>It requires <code class="language-plaintext highlighter-rouge">pidof</code> unless you configure the <a href="#pidcommand"><code class="language-plaintext highlighter-rouge">PidCommand</code></a> option.</p>
<p>It also requires <code class="language-plaintext highlighter-rouge">perl</code>,
and a version of <code class="language-plaintext highlighter-rouge">jstat</code> that’s compatible with the Java process you’re monitoring.</p>
<h3 id="quick-setup">Quick Setup</h3>
<p>Grab
<a href="https://raw.githubusercontent.com/natewoodward/code-snippets/master/check-mk/check_jvm_memory"><code class="language-plaintext highlighter-rouge">check_jvm_memory</code></a>
from GitHub and drop it in the
<a href="https://checkmk.com/cms_localchecks.html#folder">Check MK script directory</a>
on the host you want to monitor.
The path to the script directory varies depending on how the Check MK agent was set up.
On my systems, it’s <code class="language-plaintext highlighter-rouge">/usr/share/check-mk-agent/local</code>.</p>
<p>If everything’s set up right,
running the script should give you output that starts with a <code class="language-plaintext highlighter-rouge">P</code>, similar to this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>P JVM_MEMORY.SurvivorSpace1 SurvivorSpace1=0.000000% SurvivorSpace1 0.0% (0.00 / 25.56 MB)
P JVM_MEMORY.OldGen OldGen=26.603203% OldGen 26.6% (340.52 / 1280.00 MB)
P JVM_MEMORY.SurvivorSpace0 SurvivorSpace0=1.325642% SurvivorSpace0 1.3% (0.34 / 25.56 MB)
P JVM_MEMORY.EdenSpace EdenSpace=13.898290% EdenSpace 13.9% (28.47 / 204.88 MB)
P JVM_MEMORY.MetaSpace MetaSpace=98.491438% MetaSpace 98.5% (249.63 / 253.45 MB)
P JVM_MEMORY.CompressedClassSpace CompressedClassSpace=98.215967% CompressedClassSpace 98.2% (32.61 / 33.20 MB)
</code></pre></div></div>
<p>Once Check MK inventories the host,
you should be able to add the new service checks through WATO in the usual way.</p>
<h3 id="install">Install</h3>
<p>I prefer to install the script to <code class="language-plaintext highlighter-rouge">/usr/local/bin/check_jvm_memory</code>,
then symlink it from the Check MK script directory using a name to indicate the service it checks.
For example, to monitor Tomcat I symlink the script from <code class="language-plaintext highlighter-rouge">/usr/share/check-mk-agent/local/300/check_tomcat_memory</code>.
Using the <code class="language-plaintext highlighter-rouge">300</code> subdirectory causes Check MK to run the script every 300 seconds (5 minutes),
and the <code class="language-plaintext highlighter-rouge">check_tomcat_memory</code> filename makes the script produce this output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>P TOMCAT_MEMORY.SurvivorSpace1 SurvivorSpace1=0.000000% SurvivorSpace1 0.0% (0.00 / 25.56 MB)
P TOMCAT_MEMORY.OldGen OldGen=26.603203% OldGen 26.6% (340.52 / 1280.00 MB)
P TOMCAT_MEMORY.SurvivorSpace0 SurvivorSpace0=1.325642% SurvivorSpace0 1.3% (0.34 / 25.56 MB)
P TOMCAT_MEMORY.EdenSpace EdenSpace=25.776531% EdenSpace 25.8% (52.81 / 204.88 MB)
P TOMCAT_MEMORY.MetaSpace MetaSpace=98.491438% MetaSpace 98.5% (249.63 / 253.45 MB)
P TOMCAT_MEMORY.CompressedClassSpace CompressedClassSpace=98.215967% CompressedClassSpace 98.2% (32.61 / 33.20 MB)
</code></pre></div></div>
<p>Since the script is called as <code class="language-plaintext highlighter-rouge">check_tomcat_memory</code> and not <code class="language-plaintext highlighter-rouge">check_jvm_memory</code>,
the service check names start with <code class="language-plaintext highlighter-rouge">TOMCAT_MEMORY</code> instead of <code class="language-plaintext highlighter-rouge">JVM_MEMORY</code>.
This behavior can be overriden by creating a configuration file at
<code class="language-plaintext highlighter-rouge">/etc/check-mk-agent/check_tomcat_memory.cfg</code>,
and including the <a href="#name"><code class="language-plaintext highlighter-rouge">Name</code></a> option described below.</p>
<p>I recommend setting a <a href="#pidcommand"><code class="language-plaintext highlighter-rouge">PidCommand</code></a> in the configuration so that the script will always find the correct Java process to monitor if there are multiple Java processes running on the system.
You might also want to configure warning and critical thresholds for OldGen as described in the <a href="#threshold"><code class="language-plaintext highlighter-rouge">Threshold</code></a> section,
and for PermGen as well on Java 7 and under.</p>
<h3 id="configure">Configure</h3>
<p><code class="language-plaintext highlighter-rouge">check_jvm_memory</code> looks for its configuration in the <code class="language-plaintext highlighter-rouge">/etc/check-mk-agent</code> directory.
The name of the config file is the name of the script plus a <code class="language-plaintext highlighter-rouge">.cfg</code> extension.
There’s an example
<a href="https://raw.githubusercontent.com/natewoodward/code-snippets/master/check-mk/check_jvm_memory.cfg"><code class="language-plaintext highlighter-rouge">check_jvm_memory.cfg</code></a>
on GitHub.</p>
<p>Configuration options are described below.</p>
<h4 id="name">Name</h4>
<p>This lets you set a service check name that’s not derived from the script’s filename.
For instance, here’s the script’s output with <code class="language-plaintext highlighter-rouge">Name: TomcatMemory</code> set:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>P TomcatMemory.SurvivorSpace1 SurvivorSpace1=0.000000% SurvivorSpace1 0.0% (0.00 / 25.56 MB)
P TomcatMemory.OldGen OldGen=26.603203% OldGen 26.6% (340.52 / 1280.00 MB)
P TomcatMemory.SurvivorSpace0 SurvivorSpace0=1.325642% SurvivorSpace0 1.3% (0.34 / 25.56 MB)
P TomcatMemory.EdenSpace EdenSpace=45.458359% EdenSpace 45.5% (93.13 / 204.88 MB)
P TomcatMemory.MetaSpace MetaSpace=98.491438% MetaSpace 98.5% (249.63 / 253.45 MB)
P TomcatMemory.CompressedClassSpace CompressedClassSpace=98.215967% CompressedClassSpace 98.2% (32.61 / 33.20 MB)
</code></pre></div></div>
<p>Note that <code class="language-plaintext highlighter-rouge">Name</code> can’t contain spaces, or else Check MK won’t be able to parse the script’s output.</p>
<h4 id="pidcommand">PidCommand</h4>
<p><code class="language-plaintext highlighter-rouge">check_jvm_memory</code> locates the process ID (pid) of the Java process it’s monitoring using a command specified by the <code class="language-plaintext highlighter-rouge">PidCommand</code> option.
<code class="language-plaintext highlighter-rouge">check_jvm_memory</code> will look at the first line of output from this command,
find the first integer, and use that as the pid to pass to <code class="language-plaintext highlighter-rouge">jstat</code>.</p>
<p>The default <code class="language-plaintext highlighter-rouge">PidCommand</code> is <code class="language-plaintext highlighter-rouge">pidof java</code>.</p>
<p>This option is especially useful on systems that have multiple Java processes running on them.
On such systems, you can monitor each process with a differently-named <code class="language-plaintext highlighter-rouge">check_jvm_memory</code> script,
then configure a different <code class="language-plaintext highlighter-rouge">PidCommand</code> for each of those scripts.</p>
<p>Here are a few example commands you could use with this configuration option.
In every case, the script would monitor process ID <code class="language-plaintext highlighter-rouge">9170</code> since that’s the first integer on the first line of output.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@beemo:~# pidof java
9170 4978 4443
root@beemo:~# systemctl show --property MainPID tomcat
MainPID=9170
root@beemo:~# service tomcat status
tomcat (pid 9170) is running... [ OK ]
root@beemo:~# cat /var/run/tomcat.pid
9170
</code></pre></div></div>
<h4 id="threshold">Threshold</h4>
<p>This option is used to set the threshold for each service check.
The value you set is used as the <code class="language-plaintext highlighter-rouge">;warn;crit;min;max</code> values for
<a href="https://checkmk.com/cms_localchecks.html#perfdata">Check MK’s metrics</a>.
Read Check MK’s documentation for a complete description of how metrics work.</p>
<p>To explain by way of example, suppose you want Check MK to put the OldGen and PermGen service checks into Warning state if they go over 98%,
and Critical state if they go over 99%.
You can do that by configuring these thresholds:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Threshold OldGen: ;98;99
Threshold PermGen: ;98;99
</code></pre></div></div>
<h4 id="label">Label</h4>
<p>The <code class="language-plaintext highlighter-rouge">Label</code> config option lets you set a label that’s used in the service name, metrics (a.k.a. perfdata), and status detail (a.k.a. description) of the check’s output.
Most people won’t ever need to use this configuration option.</p>
<p>The <code class="language-plaintext highlighter-rouge">Label</code> option is best explained by describing how <code class="language-plaintext highlighter-rouge">check_jvm_memory</code> does its thing.
Internally, it gets its data using a <code class="language-plaintext highlighter-rouge">jstat</code> command similar to the one shown below.
The output is a bit confusing, but to use a couple examples,
the <code class="language-plaintext highlighter-rouge">S0U</code> here indicates how much of <strong>S</strong>urvivor Space #<strong>0</strong> is <strong>U</strong>sed (in KB).
Similarly, <code class="language-plaintext highlighter-rouge">S0C</code> is <strong>S</strong>urvivor Space #<strong>0</strong>’s total <strong>C</strong>apacity.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>woody@beemo:~# jstat -gc 9170
S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT
26176.0 26176.0 347.0 0.0 209792.0 178909.4 1310720.0 348693.5 259532.0 255616.8 33996.0 33389.5 84 3.658 5 2.046 5.703
</code></pre></div></div>
<p>After parsing that data, <code class="language-plaintext highlighter-rouge">check_jvm_memory</code> looks at every field header ending with a <code class="language-plaintext highlighter-rouge">U</code>,
looks up the matching header that ends in <code class="language-plaintext highlighter-rouge">C</code>,
and uses the two values for those columns to compute the percent usage of that area of memory.</p>
<p>If there were no labels set,
the script would report that <code class="language-plaintext highlighter-rouge">S0</code> has a usage of <code class="language-plaintext highlighter-rouge">1.33%</code> in the example above,
which isn’t very helpful – what the heck is <code class="language-plaintext highlighter-rouge">S0</code>?
To make the output more human friendly,
you can configure a label for <code class="language-plaintext highlighter-rouge">S0</code> so that the script reports <code class="language-plaintext highlighter-rouge">1.33%</code> usage for <code class="language-plaintext highlighter-rouge">SurvivorSpace0</code> instead:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Label S0: SurvivorSpace0
</code></pre></div></div>
<p>Internally, the script has default labels for every area of memory in Java 7 and Java 8,
so you shouldn’t need to configure any labels.
But if you don’t like the labels that <code class="language-plaintext highlighter-rouge">check_jvm_memory</code> uses,
or if other versions of Java ever add new areas of memory,
you can configure how they appear in the script’s output using the <code class="language-plaintext highlighter-rouge">Label</code> configuration option.</p>
<p>Note that any <code class="language-plaintext highlighter-rouge">Threshold</code>s you have configured need to match the <code class="language-plaintext highlighter-rouge">Label</code>s you set.
Also note that <code class="language-plaintext highlighter-rouge">Label</code>s can’t contain spaces, or else Check MK won’t be able to parse the script’s output.</p>
<!--
### Footnotes
[^1]: Credit goes to <user> for <whatever reasons>.
-->
Process Substitution And Race Conditions2019-11-25T01:07:03+00:00https://www.natewoodward.org/blog/2019/11/25/process-substitution-and-race-conditions<p>In a recent article, I showed how to
<a href="https://www.natewoodward.org/blog/2019/11/22/bash-output-patterns#duplicate-all-output-to-a-log-file">print a script’s output to both a log file and stdout</a>.
Some of you might be wondering why I used a <code class="language-plaintext highlighter-rouge">{ group command; }</code> piped into <code class="language-plaintext highlighter-rouge">tee</code> for this,
rather than <code class="language-plaintext highlighter-rouge">>(process substition)</code>. Let me explain.</p>
<p>The solution I offered in that post has a downside in that it redirects stderr to stdout.
<a href="https://stackoverflow.com/a/11886837">This StackOverflow answer</a>
tries to get around that limitation by using separate <code class="language-plaintext highlighter-rouge">tee</code> processes to handle stdout and stderr,
but it introduces a new problem.
Let’s see what happens when we run their example script:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>^_^ woody@beemo:~$ myscript
^_^ woody@beemo:~$ foo
bar
</code></pre></div></div>
<p>It might look like I typed <code class="language-plaintext highlighter-rouge">foo</code> into the terminal above,
but what actually happened is <code class="language-plaintext highlighter-rouge">myscript</code> exited,
the shell printed <code class="language-plaintext highlighter-rouge">$PS1</code> to indicate it was ready for me to type another command on stdin,
then the first <code class="language-plaintext highlighter-rouge">tee</code> process from <code class="language-plaintext highlighter-rouge">myscript</code> printed the string “foo” on stdout after the prompt,
and finally the second <code class="language-plaintext highlighter-rouge">tee</code> process printed “bar” on stderr on the next line.
This is because commands inside the <code class="language-plaintext highlighter-rouge">>( )</code> of a process substition run asynchronously,
so there’s no guarantee that our script will exit after the <code class="language-plaintext highlighter-rouge">tee</code> processes print their output.
Either event could happen first.
It’s a classic race condition.</p>
<p>We can use information from
<a href="https://unix.stackexchange.com/a/524844">another StackOverflow answer</a>
to address the race condition by forcing the script to wait for the <code class="language-plaintext highlighter-rouge">tee</code> processes to end.
Here’s what that looks like:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c"># global vars</span>
<span class="nv">prog</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">0</span><span class="p">##*/</span><span class="k">}</span><span class="s2">"</span>
<span class="nv">syncfifo</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">TMPDIR</span><span class="p">-/tmp</span><span class="k">}</span><span class="s2">/</span><span class="nv">$prog</span><span class="s2">.pipe"</span>
<span class="c"># create fifo</span>
<span class="nb">mkfifo</span> <span class="s2">"</span><span class="nv">$syncfifo</span><span class="s2">"</span> <span class="o">||</span> <span class="nb">exit </span>1
<span class="nb">trap</span> <span class="s2">"rm '</span><span class="nv">$syncfifo</span><span class="s2">'"</span> EXIT
<span class="c"># output to stdout and log file</span>
<span class="nb">exec</span> <span class="o">></span> <span class="o">>(</span><span class="nb">tee</span> <span class="nt">-ia</span> <span class="s2">"</span><span class="nv">$prog</span><span class="s2">.log"</span><span class="p">;</span> <span class="nb">echo</span> <span class="o">></span><span class="s2">"</span><span class="nv">$syncfifo</span><span class="s2">"</span><span class="o">)</span>
<span class="nb">exec </span>2> <span class="o">>(</span><span class="nb">tee</span> <span class="nt">-ia</span> <span class="s2">"</span><span class="nv">$prog</span><span class="s2">.log"</span> <span class="o">></span>&2<span class="p">;</span> <span class="nb">echo</span> <span class="o">></span><span class="s2">"</span><span class="nv">$syncfifo</span><span class="s2">"</span><span class="o">)</span>
<span class="c"># your script here</span>
<span class="nb">echo </span>foo
<span class="nb">echo </span>bar <span class="o">></span>&2
<span class="c"># close stdout and stderr so that the tee processes will exit</span>
<span class="nb">exec</span> <span class="o">></span>&-
<span class="nb">exec </span>2>&-
<span class="c"># wait for tee to exit</span>
<span class="nb">read </span>line < <span class="s2">"</span><span class="nv">$syncfifo</span><span class="s2">"</span>
<span class="nb">read </span>line < <span class="s2">"</span><span class="nv">$syncfifo</span><span class="s2">"</span>
</code></pre></div></div>
<p>Seems pretty complicated, but if it works it works, right?
Unfortunately, if you run this script enough times, you’ll see output like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>^_^ woody@beemo:~$ myscript
foo
bar
^_^ woody@beemo:~$ myscript
bar
foo
^_^ woody@beemo:~$ myscript
foo
bar
^_^ woody@beemo:~$
</code></pre></div></div>
<p>Here we’ve exposed another problem.
In addition to the race condition between our script and the <code class="language-plaintext highlighter-rouge">tee</code> processes,
there’s also a race condition between the two <code class="language-plaintext highlighter-rouge">tee</code> process substitutions,
so sometimes it prints <code class="language-plaintext highlighter-rouge">"foo\nbar"</code> and other times it prints <code class="language-plaintext highlighter-rouge">"bar\nfoo"</code>.</p>
<p><a href="https://stackoverflow.com/a/3403786">The accepted answer</a> from the first SO link only uses one <code class="language-plaintext highlighter-rouge">tee</code> process,
so it doesn’t have that problem.
But it <em>does</em> have the same race condition between the script and <code class="language-plaintext highlighter-rouge">tee</code> that we discussed before.
We can solve that problem similarly to how we solved it above for the script with two <code class="language-plaintext highlighter-rouge">tee</code> processes, like so:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c"># global vars</span>
<span class="nv">prog</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">0</span><span class="p">##*/</span><span class="k">}</span><span class="s2">"</span>
<span class="nv">syncfifo</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">TMPDIR</span><span class="p">-/tmp</span><span class="k">}</span><span class="s2">/</span><span class="nv">$prog</span><span class="s2">.pipe"</span>
<span class="c"># create fifo</span>
<span class="nb">mkfifo</span> <span class="s2">"</span><span class="nv">$syncfifo</span><span class="s2">"</span> <span class="o">||</span> <span class="nb">exit </span>1
<span class="nb">trap</span> <span class="s2">"rm '</span><span class="nv">$syncfifo</span><span class="s2">'"</span> EXIT
<span class="c"># output to stdout and log file</span>
<span class="nb">exec</span> &> <span class="o">>(</span><span class="nb">tee</span> <span class="nt">-ia</span> <span class="s2">"</span><span class="nv">$prog</span><span class="s2">.log"</span><span class="p">;</span> <span class="nb">echo</span> <span class="o">></span><span class="s2">"</span><span class="nv">$syncfifo</span><span class="s2">"</span><span class="o">)</span>
<span class="c"># your script here</span>
<span class="nb">echo </span>foo
<span class="nb">echo </span>bar <span class="o">></span>&2
<span class="c"># close stdout and stderr so that the tee process will exit</span>
<span class="nb">exec</span> <span class="o">></span>&-
<span class="nb">exec </span>2>&-
<span class="c"># wait for tee to exit</span>
<span class="nb">read </span>line < <span class="s2">"</span><span class="nv">$syncfifo</span><span class="s2">"</span>
</code></pre></div></div>
<p>But ultimately, that solution adds a lot of complexity to our script and no benefit compared to just doing this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c"># global vars</span>
<span class="nv">prog</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">0</span><span class="p">##*/</span><span class="k">}</span><span class="s2">"</span>
<span class="o">{</span>
<span class="c"># your script here</span>
<span class="nb">echo </span>foo
<span class="nb">echo </span>bar <span class="o">></span>&2
<span class="o">}</span> 2>&1 | <span class="nb">tee</span> <span class="nt">-a</span> <span class="s2">"prog.log"</span>
</code></pre></div></div>
<p>So that’s my preferred solution for duplicating a script’s output to a log file.</p>
<!--
### Footnotes
[^1]: Credit goes to <user> for <whatever reasons>.
-->
More Bash Patterns2022-05-18T00:06:06+00:00https://www.natewoodward.org/blog/2019/11/23/more-bash-patterns<p>Let’s look at a few more patterns that a <code class="language-plaintext highlighter-rouge">bash</code> shell script can use.
I’ll show how a script can drop its own privileges, run with a lower priority,
obtain a lock file<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>, and create a temp file.</p>
<ol id="markdown-toc">
<li><a href="#drop-privileges" id="markdown-toc-drop-privileges">Drop Privileges</a></li>
<li><a href="#create-a-temp-file" id="markdown-toc-create-a-temp-file">Create a Temp File</a></li>
<li><a href="#run-with-a-lower-priority" id="markdown-toc-run-with-a-lower-priority">Run With a Lower Priority</a></li>
<li><a href="#generate-a-seeded-random-number" id="markdown-toc-generate-a-seeded-random-number">Generate a Seeded Random Number</a></li>
<li><a href="#footnotes" id="markdown-toc-footnotes">Footnotes</a></li>
</ol>
<h3 id="drop-privileges">Drop Privileges</h3>
<p>Occasionally you might write a script that needs to run as a particular user.
Rather than having to remember to use <code class="language-plaintext highlighter-rouge">su</code> to switch to that user or <code class="language-plaintext highlighter-rouge">sudo -u $user</code> to run the script as that user,
you can do the following:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># global vars</span>
<span class="nv">prog</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">0</span><span class="p">##*/</span><span class="k">}</span><span class="s2">"</span>
<span class="c"># prints an error message to stderr</span>
err<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">message</span><span class="o">=</span><span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span>
<span class="nb">shift
printf</span> <span class="s2">"</span><span class="nv">$message</span><span class="se">\n</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="o">}</span>
<span class="c"># drop privileges</span>
<span class="nv">user</span><span class="o">=</span>nobody
<span class="k">if</span> <span class="o">((</span> EUID <span class="o">==</span> 0 <span class="o">))</span><span class="p">;</span> <span class="k">then
</span><span class="nb">exec </span>runuser <span class="nt">-u</span> <span class="s2">"</span><span class="nv">$user</span><span class="s2">"</span> <span class="nt">--</span> <span class="s2">"</span><span class="nv">$0</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="c"># or if runuser isn't available on your system:</span>
<span class="c">#exec su "$user" -c "$(printf %q "$0") $(printf %q "$*")"</span>
<span class="k">elif</span> <span class="o">[[</span> <span class="si">$(</span><span class="nb">id</span> <span class="nt">-un</span><span class="si">)</span> <span class="o">!=</span> <span class="s2">"</span><span class="nv">$user</span><span class="s2">"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then
</span>err <span class="s2">"This script must be run as %s or %s"</span> <span class="s1">'root'</span> <span class="s2">"</span><span class="nv">$user</span><span class="s2">"</span>
<span class="nb">exit </span>1
<span class="k">fi</span>
</code></pre></div></div>
<p>Now you can run the script from a root shell or with <code class="language-plaintext highlighter-rouge">sudo</code>,
safe in the knowledge that it will only run as the <code class="language-plaintext highlighter-rouge">nobody</code> user.</p>
<!--
### Obtain a Lock File
If you want to ensure that multiple instances of the same script can't run at the same time,
you can use the following to make each instance of a script try to obtain a lock file.
Only the first one to obtain the lock will continue running.
The others will exit.
Note that I'm using `$prog` and `err()` from the previous example without declaring them as I did in the previous example.
For the rest of this article, I'll assume that `$prog` and `err()` have already been declared.
```bash
# try to obtain lock
lockfile="${TMPDIR-/tmp}/$prog.lock"
if (set -o noclobber && echo $$>"$lockfile") 2>/dev/null; then
trap 'rm -f "$lockfile"' EXIT
else
err "Lock %s held by pid %s" "$lockfile" "$(<"$lockfile")"
exit 1
fi
```
-->
<h3 id="create-a-temp-file">Create a Temp File</h3>
<p>The <a href="https://www.owasp.org/index.php/Insecure_Temporary_File">pitfalls involved in creating a temporary file</a> are well-known.
To safely create a temp file, use <code class="language-plaintext highlighter-rouge">mktemp</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># create a temporary file</span>
<span class="nb">trap</span> <span class="s1">'rm -f "$tmpfile"'</span> EXIT
<span class="nv">tmpfile</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">mktemp</span> <span class="nt">--tmpdir</span> <span class="s2">"</span><span class="nv">$prog</span><span class="s2">.XXXXXXXXXX"</span><span class="si">)</span><span class="s2">"</span> <span class="o">||</span> <span class="nb">exit </span>1
</code></pre></div></div>
<p>This will atomically create a new temporary file in a directory defined by the <code class="language-plaintext highlighter-rouge">$TMPDIR</code> environment variable if it’s defined,
or <code class="language-plaintext highlighter-rouge">/tmp</code> if it’s not.</p>
<h3 id="run-with-a-lower-priority">Run With a Lower Priority</h3>
<p>If you have a script that uses a lot of CPU,
you can set its niceness with <code class="language-plaintext highlighter-rouge">renice</code>.
Higher niceness values give a process lower scheduling priority.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># change niceness of this script</span>
renice 10 <span class="nt">-p</span> <span class="nv">$$</span>
</code></pre></div></div>
<h3 id="generate-a-seeded-random-number">Generate a Seeded Random Number</h3>
<p>Suppose you have a script that runs daily from cron,
and it needs to perform some cleanup task once per week.
If you wanted the task to happen on a day that varies from system to system, you could do</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># clean up once per week</span>
<span class="nv">RANDOM</span><span class="o">=</span><span class="k">$((</span> <span class="m">16</span><span class="c">#$(hostname -f | md5sum | cut -c1-4) ))</span>
<span class="nv">cleanup_day</span><span class="o">=</span><span class="si">$(</span>RANDOM % 7<span class="si">)</span>
<span class="k">if</span> <span class="o">((</span> <span class="s2">"</span><span class="si">$(</span><span class="nb">date</span> +%u<span class="si">)</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"</span><span class="nv">$cleanup_day</span><span class="s2">"</span> <span class="k">))</span><span class="p">;</span> <span class="k">then</span>
<span class="c"># do cleanup task</span>
<span class="k">fi</span>
</code></pre></div></div>
<p>Note that this is only a pathological example meant to demonstrate the technique.
It would be simpler to schedule the cleanup task as a separate cron job in this scenario.
But if, for example, you needed to schedule such a cron job from a configuration management system,
you could use this technique to generate the day of the week
(or the day of the month, the hour, the minute, etc.) to schedule it on.</p>
<h3 id="footnotes">Footnotes</h3>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>I’ve removed the section about lock files after reddit user /u/vogelke <a href="https://www.reddit.com/r/commandline/comments/e1h4f0/is_stuff_like_this_useful/f8r4okk/?context=10">raised some concerns</a> about it. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Bash Output Patterns2019-11-24T21:29:14+00:00https://www.natewoodward.org/blog/2019/11/22/bash-output-patterns<p>Here’s a few patterns I use to send a <code class="language-plaintext highlighter-rouge">bash</code> script’s output to a log file,
to syslog, or as an email notification.</p>
<ol id="markdown-toc">
<li><a href="#redirect-all-output-to-a-log-file" id="markdown-toc-redirect-all-output-to-a-log-file">Redirect All Output to a Log File</a></li>
<li><a href="#conditionally-send-output-elsewhere" id="markdown-toc-conditionally-send-output-elsewhere">Conditionally Send Output Elsewhere</a></li>
<li><a href="#duplicate-all-output-to-a-log-file" id="markdown-toc-duplicate-all-output-to-a-log-file">Duplicate All Output to a Log File</a></li>
<li><a href="#send-all-output-to-syslog" id="markdown-toc-send-all-output-to-syslog">Send All Output to Syslog</a></li>
<li><a href="#send-all-output-in-an-email" id="markdown-toc-send-all-output-in-an-email">Send All Output in an Email</a></li>
<li><a href="#output-log-file-data-until-a-message-or-timeout-is-reached" id="markdown-toc-output-log-file-data-until-a-message-or-timeout-is-reached">Output Log File Data Until A Message or Timeout is Reached</a></li>
</ol>
<h3 id="redirect-all-output-to-a-log-file">Redirect All Output to a Log File</h3>
<p>Suppose you’re writing a script called <code class="language-plaintext highlighter-rouge">myscript</code> and want to send its output to a log file rather than to the console whenever it’s run.
Running <code class="language-plaintext highlighter-rouge">myscript &>>myscript.log</code> every time would be cumbersome and easy to forget,
so let’s do this instead:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># some setup we'll use in other examples</span>
<span class="nv">prog</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">0</span><span class="p">##*/</span><span class="k">}</span><span class="s2">"</span>
<span class="nv">logfile</span><span class="o">=</span><span class="s2">"</span><span class="nv">$prog</span><span class="s2">.log"</span>
<span class="c"># send all output to log file</span>
<span class="nb">exec</span> &>> <span class="s2">"</span><span class="nv">$logfile</span><span class="s2">"</span>
<span class="c"># alternatively:</span>
<span class="c">#exec &> "$logfile"</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">exec &>> "$logfile"</code> here will redirect your script’s stdout and stderr,
appending all of your script’s output to <code class="language-plaintext highlighter-rouge">$logfile</code> without overwriting its existing contents.
You can use <code class="language-plaintext highlighter-rouge">&></code> instead of <code class="language-plaintext highlighter-rouge">&>></code> to overwrite <code class="language-plaintext highlighter-rouge">$logfile</code>,
effectively replacing its contents with your script’s output.</p>
<!--
Quick aside about the declaration of `$prog` above:
`${0##*/}` is identical to `$(basename "$0")` except that the former doesn't fork a new process.
-->
<h3 id="conditionally-send-output-elsewhere">Conditionally Send Output Elsewhere</h3>
<p>Let’s look at one way we can extend the previous example.
Suppose you want your script to output to the terminal when you call it normally from the command line,
but when it’s run from cron you want it to output to a log file instead.
In that case, you can do the following.</p>
<p>Note that I’m using <code class="language-plaintext highlighter-rouge">$logfile</code> from the previous example without declaring it as I did in the previous example.
For the rest of this article, I’ll assume that <code class="language-plaintext highlighter-rouge">$prog</code> and <code class="language-plaintext highlighter-rouge">$logfile</code> have already been declared.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># check if stdout isn't a terminal</span>
<span class="k">if</span> <span class="o">[[</span> <span class="o">!</span> <span class="nt">-t</span> 1 <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<span class="c"># send all output to log file</span>
<span class="nb">exec</span> &>> <span class="s2">"</span><span class="nv">$logfile</span><span class="s2">"</span>
<span class="k">fi</span>
</code></pre></div></div>
<p>Just like it says in the comments,
this checks if stdout is a terminal and sends all output to <code class="language-plaintext highlighter-rouge">$logfile</code> if it’s not.
So it will always send its output to <code class="language-plaintext highlighter-rouge">$logfile</code> when run from cron or launched from another daemon process.
And it will send its output to the terminal if you call it from the command line in most, but not all cases.
Consider what happens in the following examples:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>myscript <span class="c"># output to terminal</span>
<span class="o">(</span> myscript <span class="o">)</span> <span class="c"># output to terminal -- subshells don't affect stdout</span>
<span class="nb">echo</span> | myscript <span class="c"># output to terminal -- stdin is a pipe, but stdout is still a terminal</span>
myscript 2> foo.txt <span class="c"># output to terminal -- stderr is a file, but stdout is still a terminal</span>
myscript < foo.txt <span class="c"># output to terminal -- stdin is a file</span>
myscript <span class="o">></span> foo.txt <span class="c"># output to $logfile -- stdout is a file</span>
myscript | <span class="nb">grep</span> <span class="nb">.</span> <span class="c"># output to $logfile -- stdout is a pipe</span>
</code></pre></div></div>
<p>As you can see by reading the comments or by testing the commands yourself from a shell,
it’s stdout that matters when it comes to reasoning about how the command will function.
If that behavior isn’t what’s desired,
you can try using <code class="language-plaintext highlighter-rouge">-t 0</code> instead of <code class="language-plaintext highlighter-rouge">-t 1</code> to detect whether stdin is a terminal.
Consider what happens in that case if you were to run the examples above. Try testing it out in a shell if you’re unsure.</p>
<p>Ultimately, using <code class="language-plaintext highlighter-rouge">[[ -t $fd ]]</code> in this way is a heuristic for determining whether a human is likely to see our output or not,
so it’s important to understand its limitations.
While it’s useful in scripts that we expect to call in simple ways,
for a script that’s designed to do one thing and do it well,
we often want to be able to pipe data into and/or out of it in a lot of arbitrary ways.
In that case this pattern becomes an anti-pattern since its behavior can be unintuitive,
and you’re better off implementing an option that changes where the script sends its output.</p>
<h3 id="duplicate-all-output-to-a-log-file">Duplicate All Output to a Log File</h3>
<p>If you want to send all script output to <em>both</em> your terminal and a log file,
you can use a <code class="language-plaintext highlighter-rouge">{ group command; }</code> to pipe all of your script’s output into <code class="language-plaintext highlighter-rouge">tee</code> like so:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># copy all script output to log file</span>
<span class="o">{</span>
<span class="c"># your script here</span>
<span class="o">}</span> 2>&1 | <span class="nb">tee</span> <span class="nt">-a</span> <span class="s2">"</span><span class="nv">$logfile</span><span class="s2">"</span>
</code></pre></div></div>
<!--
If you want to overwrite `$logfile` rather than append to it, omit the `-a` flag in the `tee` command.
-->
<p>Note that, since both stderr and stdout are piped into <code class="language-plaintext highlighter-rouge">tee</code>,
if your script had a command like e.g. <code class="language-plaintext highlighter-rouge">echo foo >&2</code> that sends output to stderr,
that output will show up on your script’s stdout instead.</p>
<h3 id="send-all-output-to-syslog">Send All Output to Syslog</h3>
<p>To redirect all script output to syslog, you can do something like:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># redirect all output to syslog</span>
<span class="nb">exec</span> &> <span class="o">>(</span>logger <span class="nt">-e</span> <span class="nt">-t</span> <span class="s2">"</span><span class="nv">$prog</span><span class="s2">"</span> <span class="nt">--id</span><span class="o">=</span><span class="nv">$$</span> <span class="nt">-p</span> local0.info<span class="o">)</span>
<span class="c"># with older versions of util-linux:</span>
<span class="c">#exec &> >(grep -v '^$' | logger -t "$prog" -p local0.info)</span>
</code></pre></div></div>
<p>This uses process substitution to send stdout and stdin to <code class="language-plaintext highlighter-rouge">logger</code>, which sends our output to syslog.
The <code class="language-plaintext highlighter-rouge">-e</code> option to logger prevents empty lines from being logged –
in most cases we probably don’t care to store empty log messages.
The <code class="language-plaintext highlighter-rouge">--id</code> option logs our script’s process id.
If you’re unfamiliar with the other <code class="language-plaintext highlighter-rouge">logger</code> flags, you can look them up with <code class="language-plaintext highlighter-rouge">man logger</code>.</p>
<p>On systems with older versions of util-linux, <code class="language-plaintext highlighter-rouge">logger</code> might not have the <code class="language-plaintext highlighter-rouge">-e</code> and <code class="language-plaintext highlighter-rouge">--id</code> flags.
In that case we can do that next best thing, using <code class="language-plaintext highlighter-rouge">grep</code> to suppress empty lines,
and just living without the PID in our syslog messages.
You can use <code class="language-plaintext highlighter-rouge">-i</code> to pass the logger process’s PID instead,
but I worry that could be a point of confusion for anyone reading the logs who’s not familiar with the script,
so I don’t do that.</p>
<h3 id="send-all-output-in-an-email">Send All Output in an Email</h3>
<p>For simple scripts,
this can be done with process substitution.
We’ll just redirect our output to <code class="language-plaintext highlighter-rouge">sendmail</code>,
making sure that the first thing we output is the necessary email headers.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># redirect all output to sendmail</span>
<span class="nb">exec</span> &> <span class="o">>(</span>sendmail <span class="nt">-t</span><span class="o">)</span>
<span class="c"># print headers for notification email</span>
<span class="nb">cat</span> <span class="o"><<-</span><span class="no">EOF</span><span class="sh">
Subject: Output of </span><span class="nv">$prog</span><span class="sh">
To: somebody@example.com
From: </span><span class="si">$(</span><span class="nb">whoami</span><span class="si">)</span><span class="sh">@</span><span class="si">$(</span><span class="nb">hostname</span> <span class="nt">-f</span><span class="si">)</span><span class="sh">
</span><span class="no">
EOF
</span></code></pre></div></div>
<p>In many cases you only want to deliver mail when your script fails, though.
In order to do that, I’ve started using a wrapper script on systems that I’m responsible for.
The script is called
<a href="https://raw.githubusercontent.com/natewoodward/code-snippets/master/bin/mailfail"><code class="language-plaintext highlighter-rouge">mailfail</code></a>,
and it’s
<a href="https://github.com/natewoodward/code-snippets/blob/master/bin/mailfail">available on GitHub</a>.</p>
<p>To use the script, you call it with a comma-delimited list of email addresses and the command you want to run, like so:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mailfail someone@example.com,somebody@example.com /some/command <span class="nt">--that</span> <span class="nt">-might</span> fail
</code></pre></div></div>
<p>For more detailed usage info, run <code class="language-plaintext highlighter-rouge">mailfail --help</code>.</p>
<p>You might need to inspect the command’s output on occasion if system mail isn’t configured correctly,
or if you just need to inspect the output of a successful command.
So <code class="language-plaintext highlighter-rouge">mailfail</code> always logs the script’s output to a file in addition to sending an email notification on failure.</p>
<p>For cron jobs, another way you can approach this problem is to configure cron to send out emails if a job has any output,
and design your scripts so that they never output anything during a successful run.
<!--
If you already have a lot of scripts that weren't designed that way,
changing them to accomodate that setup can be a lot of work, though.
--></p>
<h3 id="output-log-file-data-until-a-message-or-timeout-is-reached">Output Log File Data Until A Message or Timeout is Reached</h3>
<p>It’s not unheard of to give a developer access to restart a service through <code class="language-plaintext highlighter-rouge">sudo</code> and a restart script.
On a couple occasions I’ve needed to do this with a service that takes a long time to start up,
so to give the user feedback as to what’s going on while the service starts,
I had the script <code class="language-plaintext highlighter-rouge">tail</code> the service’s log file until some startup message is reached.
In order to prevent the script from hanging indefinitely in case the service never logs the startup message I’m expecting,
I used a combination of <code class="language-plaintext highlighter-rouge">sleep</code>, <code class="language-plaintext highlighter-rouge">wait</code>, and <code class="language-plaintext highlighter-rouge">kill</code>.</p>
<p>Here’s what that looks like:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># output log file until pattern or timeout is reached</span>
<span class="c"># define some variables</span>
<span class="nv">pattern</span><span class="o">=</span><span class="s1">'Server startup'</span> <span class="c"># regex that matches the service's startup message</span>
<span class="nb">timeout</span><span class="o">=</span>300 <span class="c"># number of seconds to wait for service to start</span>
<span class="c"># launch a process in the background that will time out eventually</span>
<span class="nb">sleep</span> <span class="s2">"</span><span class="nv">$timeout</span><span class="s2">"</span> &
<span class="nv">timerpid</span><span class="o">=</span><span class="nv">$!</span>
<span class="c"># kill the timeout process if we exit abnormally</span>
<span class="nb">trap</span> <span class="s2">"kill '</span><span class="nv">$timerpid</span><span class="s2">' 2>/dev/null"</span> EXIT
<span class="c"># output logfile until we see $pattern</span>
<span class="nb">tail</span> <span class="nt">-Fn0</span> <span class="nt">--pid</span> <span class="s2">"</span><span class="nv">$timerpid</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$logfile</span><span class="s2">"</span> 2>/dev/null | <span class="o">{</span>
<span class="nb">sed</span> <span class="nt">-r</span> <span class="s2">"/</span><span class="nv">$pattern</span><span class="s2">/ q"</span>
<span class="nb">kill</span> <span class="s2">"</span><span class="nv">$timerpid</span><span class="s2">"</span> 2>/dev/null
<span class="o">}</span> &
<span class="c"># restart the service</span>
service tomcat restart
<span class="c"># wait until the backgrounded timeout process exits</span>
<span class="o">{</span>
<span class="nb">wait</span> <span class="s2">"</span><span class="nv">$timerpid</span><span class="s2">"</span>
<span class="o">}</span> &>/dev/null
</code></pre></div></div>
<p>Let’s unpack what’s going on here:</p>
<ol>
<li>First we define some variables. No surprises here.</li>
<li>Then we start a <code class="language-plaintext highlighter-rouge">sleep</code> process in the background. Other child processes of our script will end if this process terminates.</li>
<li>Set a trap to <code class="language-plaintext highlighter-rouge">kill</code> the sleep process if our script exits early. This ensures our child processes are cleaned up sooner rather than later.</li>
<li>Then we launch a pipeline in the background.
<ul>
<li>First, we <code class="language-plaintext highlighter-rouge">tail</code> the log file, passing the following options to tail:
<ul>
<li><code class="language-plaintext highlighter-rouge">-F</code>: Follow the log file by name, rather than by file descriptor. That way if the service rotates the logfile when it restarts, we’ll find the new log file and continue printing messages from it.</li>
<li><code class="language-plaintext highlighter-rouge">-n0</code>: Follow the log file starting where it currently ends. We don’t need the context of the previous 10 log messages.</li>
<li><code class="language-plaintext highlighter-rouge">--pid $timerpid</code>: If the sleep process exits, so does the tail process. So we’ll stop outputting the log file after <code class="language-plaintext highlighter-rouge">$timeout</code> seconds at most, or sooner if the sleep process is killed early.</li>
</ul>
</li>
<li>We pipe tail’s output to a <code class="language-plaintext highlighter-rouge">sed</code> command that prints the output until it reaches the service’s startup message. Then it exits, and</li>
<li>We kill the sleep process.</li>
<li>Also, we run this whole pipeline in the background.</li>
</ul>
</li>
<li>In other words, the <code class="language-plaintext highlighter-rouge">tail</code> pipeline and <code class="language-plaintext highlighter-rouge">sleep</code> process interact to print the service’s log file until either the startup message is printed, or a timeout is reached, whichever happens first. And all of this happens in the background, because once we’re done setting this up we need to:</li>
<li>Restart the service.</li>
<li>Then, we wait until the sleep process is killed or exits.</li>
</ol>
<p>As you can see, this is a lot of complexity to add to a script,
so I recommend looking into other options before resorting to it.</p>
<!--
### Footnotes
[^1]: More specifically, `${0##*/}` is a type of parameter expansion,
and parameter expansions typically run faster than process forks like `$(basename "$0")`.
Forking a process over and over in a tight loop can add up pretty quickly and bog a script down,
and you never know when a script might be used in *another* script's tight loop,
so I try to avoid forking when possible.
On the other hand, using `basename` is a lot easier to read,
so overall I think there's a case to be made for using either of them.
On the other *other* hand, you only need `${0##*/}` explained to you once before you grok it,
so my personal preference is that it's objectively better.
[^2]: There are
[ways around this](https://stackoverflow.com/a/11886837)
but
[they're](https://unix.stackexchange.com/questions/524811/not-being-set-to-the-pid-of-a-process-substitution-used-with-an-exte)
all
[terrible](https://unix.stackexchange.com/questions/388519/bash-wait-for-process-in-process-substitution-even-if-command-is-invalid).
If you're not sure what I mean by this,
try making the SO answer in the first link work in a sane way, without any race conditions or relying on undocumented `bash` behavior.
[^3]: When I say that `tail` is "first" here,
I only mean that it's the first command in the pipeline when read top to bottom, left to right.
As many of you are well aware, processes in the same pipeline run concurrently.
-->