EuAndreh's website

Awk snippet: ShellCheck all scripts in a repository

2020-12-15T00:00:00+00:00

Inspired by Fred Herbert’s "Awk in 20 Minutes", here’s a problem I just solved with a line of Awk: run ShellCheck in all scripts of a repository.

In my repositories I usually have Bash and POSIX scripts, which I want to keep tidy with ShellCheck. Here’s the first version of assert-shellcheck.sh:

1
2
3

#!/bin/sh -eux

find . -type f -name '*.sh' -print0 | xargs -0 shellcheck

plaintext

This is the type of script that I copy around to all repositories, and I want it to be capable of working on any repository, without requiring a list of files to run ShellCheck on.

This first version worked fine, as all my scripts had the ‘.sh’ ending. But I recently added some scripts without any extension, so assert-shellcheck.sh called for a second version. The first attempt was to try grepping the shebang line:

1 2	$ grep '^#!/' assert-shellcheck.sh #!/usr/sh

plaintext

Good, we have a grep pattern on the first try. Let’s try to find all the matching files:

$ find . -type f | xargs grep -l '^#!/'
./TODOs.org
./.git/hooks/pre-commit.sample
./.git/hooks/pre-push.sample
./.git/hooks/pre-merge-commit.sample
./.git/hooks/fsmonitor-watchman.sample
./.git/hooks/pre-applypatch.sample
./.git/hooks/pre-push
./.git/hooks/prepare-commit-msg.sample
./.git/hooks/commit-msg.sample
./.git/hooks/post-update.sample
./.git/hooks/pre-receive.sample
./.git/hooks/applypatch-msg.sample
./.git/hooks/pre-rebase.sample
./.git/hooks/update.sample
./build-aux/with-guile-env.in
./build-aux/test-driver
./build-aux/missing
./build-aux/install-sh
./build-aux/install-sh~
./bootstrap
./scripts/assert-todos.sh
./scripts/songbooks
./scripts/compile-readme.sh
./scripts/ci-build.sh
./scripts/generate-tasks-and-bugs.sh
./scripts/songbooks.in
./scripts/with-container.sh
./scripts/assert-shellcheck.sh

plaintext

This approach has a problem, though: it includes files ignored by Git, such as builld-aux/install-sh~, and even goes into the .git/ directory and finds sample hooks in .git/hooks/*.

To list the files that Git is tracking we’ll try git ls-files:

$ git ls-files | xargs grep -l '^#!/'
TODOs.org
bootstrap
build-aux/with-guile-env.in
old/scripts/assert-docs-spelling.sh
old/scripts/build-site.sh
old/scripts/builder.bats.sh
scripts/assert-shellcheck.sh
scripts/assert-todos.sh
scripts/ci-build.sh
scripts/compile-readme.sh
scripts/generate-tasks-and-bugs.sh
scripts/songbooks.in
scripts/with-container.sh

plaintext

It looks to be almost there, but the TODOs.org entry shows a flaw in it: grep is looking for a '^#!/' pattern on any part of the file. In my case, TODOs.org had a snippet in the middle of the file where a line started with #!/bin/sh.

So what we actually want is to match the first line against the pattern. We could loop through each file, get the first line with head -n 1 and grep against that, but this is starting to look messy. I bet there is another way of doing it concisely…

Let’s try Awk. I need a way to select the line numbers to replace head -n 1, and to stop processing the file if the pattern matches. A quick search points me to using FNR for the former, and { nextline } for the latter. Let’s try it:

$ git ls-files | xargs awk 'FNR>1 { nextfile } /^#!\// { print FILENAME; nextfile }'
bootstrap
build-aux/with-guile-env.in
old/scripts/assert-docs-spelling.sh
old/scripts/build-site.sh
old/scripts/builder.bats.sh
scripts/assert-shellcheck.sh
scripts/assert-todos.sh
scripts/ci-build.sh
scripts/compile-readme.sh
scripts/generate-tasks-and-bugs.sh
scripts/songbooks.in
scripts/with-container.sh

plaintext

Great! Only TODOs.org is missing, but the script is much better: instead of matching against any part of the file that may have a shebang-like line, we only look for the first. Let’s put it back into the assert-shellcheck.sh file and use NULL for separators to accommodate files with spaces in the name:

#!/usr/sh -eux

git ls-files -z | \
  xargs -0 awk 'FNR>1 { nextfile } /^#!\// { print FILENAME; nextfile }' | \
  xargs shellcheck

plaintext

This is where I’ve stopped, but I imagine a likely improvement: match against only #!/bin/sh and #!/usr/bin/env bash shebangs (the ones I use most), to avoid running ShellCheck on Perl files, or other shebangs.

Also when reviewing the text of this article, I found that { nextfile } is a GNU Awk extension. It would be an improvement if assert-shellcheck.sh relied on the POSIX subset of Awk for working correctly.

Update

After publishing, I could remove { nextfile } and even make the script simpler:

#!/usr/sh -eux

git ls-files -z | \
  xargs -0 awk 'FNR==1 && /^#!\// { print FILENAME }' | \
  xargs shellcheck

plaintext

Now both the shell and Awk usage are POSIX compatible.

Useful Bash variables

2020-11-12T00:00:00+00:00

GNU Bash has a few two letter variables that may be useful when typing on the terminal.

`!!`: the text of the last command

The !! variable refers to the previous command, and I find useful when following chains for symlinks:

$ which git
/run/current-system/sw/bin/git
$ readlink $(!!)
readlink $(which git)
/nix/store/5bgr1xpm4m0r72h9049jbbhagxdyrnyb-git-2.28.0/bin/git

plaintext

It is also useful when you forget to prefix sudo to a command that requires it:

$ requires-sudo.sh
requires-sudo.sh: Permission denied
$ sudo !!
sudo ./requires-sudo.sh
# all good

plaintext

Bash prints the command expansion before executing it, so it is better for you to follow along what it is doing.

`$_`: most recent parameter

The $_ variable will give you the most recent parameter you provided to a previous argument, which can save you typing sometimes:

# instead of...
$ mkdir -p a/b/c/d/
$ cd a/b/c/d/

# ...you can:
$ mkdir -p a/b/c/d/
$ cd $_

plaintext

Conclusion

I wouldn’t use those in a script, as it would make the script terser to read, I find those useful shortcut that are handy when writing at the interactive terminal.

Find broken symlinks with "find"

2020-11-08T00:00:00+00:00

The find command knows how to show broken symlinks:

1	find . -xtype l

plaintext

This was useful to me when combined with Git Annex. Its wanted option allows you to have a "sparse" checkout of the content, and save space by not having to copy every annexed file locally:

1	git annex wanted . 'exclude=Music/* and exclude=Videos/*'

plaintext

You can find any broken symlinks outside those directories by querying with Git Annex itself, but find . -xtype l works on other places too, where broken symlinks might be a problem.

Simple filename timestamp

2020-08-12T00:00:00+00:00

When writing Jekyll posts or creating log files with dates on them, I usually struggle with finding a direct way of accomplishing that. There’s a simple solution: date -I.

1 2	./my-program.sh > my-program.$(date -I).log cp post-template.md _posts/$(date -I)-post-slug.md

plaintext

Using this built-in GNU/Linux tool allows you to touch $(date -I).md to readily create a 2020-08-12.md file.

I always had to read man date or search the web over and over, and after doing this repeatedly it became clear that both date -I and date -Is (s here stands for seconds) are the thing that I’m looking for 95% of the time:

# inside my-program.sh
echo "Program started at $(date -Is)"
# output is:
# Program started at 2020-08-12T09:04:58-03:00

plaintext

Both date formats are hierarchical, having the bigger time intervals to the left. This means that you can easily sort them (and even tab-complete them) with no extra effort or tool required.

EuAndreh's website

Awk snippet: ShellCheck all scripts in a repository

Update

Useful Bash variables

!!: the text of the last command

$_: most recent parameter

Conclusion

Find broken symlinks with "find"

Simple filename timestamp

`!!`: the text of the last command

`$_`: most recent parameter