This may make some people pull their hair out, but Iād love to hear some arguments. Iāve had the impression that people really donāt like bash, not from here, but just from people Iāve worked with.
There was a task at work where we wanted something thatāll run on a regular basis, and doesnāt do anything complex aside from reading from the database and sending the output to some web API. Pretty common these days.
I canāt think of a simpler scripting language to use than bash. Here are my reasons:
- Reading from the environment is easy, and so is falling back to some value; just do
${VAR:-fallback}; no need to write another if-statement to check for nullity. Wanna check if a variableās set to something expected?if [[ <test goes here> ]]; then <handle>; fi - Reading from arguments is also straightforward; instead of a
import os; os.args[1]in Python, you just do$1. - Sending a file via HTTP as part of an
application/x-www-form-urlencodedrequest is super easy withcurl. In most programming languages, youād have to manually open the file, read them into bytes, before putting it into your request for the http library that you need to import.curlalready does all that. - Need to read from a
curlresponse and itās JSON? Reach forjq. - Instead of having to set up a connection object/instance to your database, give
sqlite,psql,duckdbor whichever cli db client a connection string with your query and be on your way. - Shipping isā¦ fairly easy? Especially if docker is common in your infrastructure. Pull
Ubuntuordebianoralpine, install your dependencies through the package manager, and youāre good to go. If you stay within Linux and donāt have to deal with differences in bash and core utilities between different OSes (looking at you macOS), and assuming you tried to not to do anything too crazy and bring in necessary dependencies in the form of calling them, it should be fairly portable.
Sure, there can be security vulnerability concerns, but youād still have to deal with the same problems with your Pythons your Rubies etc.
For most bash gotchas, shellcheck does a great job at warning you about them, and telling how to address those gotchas.
There are probably a bunch of other considerations but I canāt think of them off the top of my head, but Iāve addressed a bunch before.
So whatās the dealeo? What am I missing that may not actually be addressable?
Iām afraid your colleagues are completely right and you are wrong, but it sounds like you genuinely are curious so Iāll try to answer.
I think the fundamental thing youāre forgetting is robustness. Yes Bash is convenient for making something that works once, in the same way that duct tape is convenient for fixes that work for a bit. But for production use you want something reliable and robust that is going to work all the time.
I suspect you just havenāt used Bash enough to hit some of the many many footguns. Or maybe when you did hit them you thought āoops I made a mistakeā, rather than āthis is dumb; I wouldnāt have had this issue in a proper programming languageā.
The main footguns are:
shellcheck. I have too. Thatās not a criticism. Itās basically impossible to get quoting completely right in any vaguely complex Bash script.set -e, but then that breaks pipelines and conditionals, and you end up with really monstrous pipelines full ofpipefailnoise. Itās also extremely easy to forgetset -e.No. If itās missing
$1will silently become an empty string.os.args[1]will throw an error. Much more robust.Absolutely not. Python is strongly typed, and even statically typed if you want. Light years ahead of Bashās mess. Quoting is pretty easy to get right in Python.
I actually started keeping a list of bugs at work that were caused directly by people using Bash. Iāll dig it out tomorrow and give you some real world examples.
Agreed.
Also gtfobins is a great resource in addition to shellcheck to try to make secure scripts.
For instance I felt upon a script like this recently:
#!/bin/bash # ... some stuff ... tar -caf archive.tar.bz2 "$@"Quotes are OK, shellcheck is happy, but, according to gtfobins, you can abuse tar, so running the script like this:
./test.sh /dev/null --checkpoint=1 --checkpoint-action=exec=/bin/shends up spawning an interactive shellā¦So you can add up binaries insanity on top of bashās mess.
I imagine adding
--so it becomestar -caf archive.tar.bz2 -- "$@"would fix that specific caseBut yeah, putting bash in a position where it has more rights than the user providing the input is a really bad idea
This runs into a part of the unix philosophy about doing one thing and doing it well: Extending programs to have more (absolutely useful) functionality winds up becoming a security risk. The shell is generally geared towards being a collection of shortcuts rather than a normal, predictable but tedious API.
For a script like that youād generally want to validate that the input is actually what you expect if it needs to handle hostile users, though. Itāll likely help the sleepy users too.
Meh, most in that list are just āif it has the SUID bit set, it can be used to break out of your security contextā.
I donāt disagree with your point, but how does
set -ebreak conditionals? I use it all the time without issuesPipefail I donāt use as much so perhaps thatās the issue?
It means that all commands that return a non-zero exit code will fail the script. The problem is that exit codes are a bit overloaded and sometimes non-zero values donāt indicate failure, they indicate some kind of status. For example in
git diff --exit-codeorgrep.I think I was actually thinking of
pipefailthough. If you donāt set it then errors in pipelines are ignored, which is obviously bad. If you do then you canāt usegrepin pipelines.My sweet spot is
set -uebecause I like to be able to use things likeif grep -q ...; thenand I like things to stop if I misspelled a variable.It does hide failures in the middle of a pipeline, but itās a tradeoff. I guess one could turn it on and off when needed
I honestly donāt care about being right or wrong. Our trade focuses on what works and what doesnāt and what can make things work reliably as we maintain them, if we even need to maintain them. Iām not proposing for bash to replace our web servers. And I certainly am not proposing that we can abandon robustness. What I am suggesting that we think about here, is that when you do not really need that robustness, for something that may perhaps live in your production system outside of user paths, perhaps something that you, your team, and the stakeholders of the particular project understand that the solution is temporary in nature, why would Bash not be sufficient?
Wrong assumption. Iāve been writing Bash for 5-6 years now.
Maybe itās the way Iāve been structuring my code, or the problems Iāve been solving with it, in the last few years after using
shellcheckandbash-language-serverthat Iāve not ran into issues where I get fucked over by quotes.But I can assure you that I know when to dip and just use a āproper programming languageā while thinking that Bash wouldnāt cut it. You seem to have an image of me just being a ābash glorifierā, and Iām not sure if itāll convince you (and I would encourage you to read my other replies if you arenāt), but I certainly donāt think bash should be used for everything.
Youāll probably hate this, but you can use
set -uto catch unassigned variables. You should also use fallbacks wherever sensible.Not a good argument imo. It eliminates a good class of problems sure. But you canāt eliminate their dependence on shared libraries that many commands also use, and thatās what my point was about.
And Iām sure you can find a whole dictionaryās worth of cases where people shoot themselves in the foot with bash. I donāt deny thatās the case. Bash is not a good language where the programmer is guarded from shooting themselves in the foot as much as possible. The guardrails are loose, and itās the script writerās job to guard themselves against it. Is that good for an enterprise scenario, where you may either blow something up, drop a database table, lead to the lost of lives or jobs, etc? Absolutely not. Just want to copy some files around and maybe send it to an internal chat for regular reporting? I donāt see why not.
Bash is not your hammer to hit every possible nail out there. Thatās not what Iām proposing at all.
If youāre proposing Bash, then yes you are.
I actually didnāt know that, thanks for the hint! I am forced to use Bash occasionally due to misguided coworkers so this will help at least.
Not sure what you mean here?
Well if itās just for a temporary hack and it doesnāt matter if it breaks then itās probably fine. Not really what is implied by āproductionā though.
Also even in that situation I wouldnāt use it for two reasons:
Iām going to downvote your comment based on that first quote reply, because I think thatās an extreme take thatās unwarranted. Youāve essentially dissed people who use it for CI/CD and suggested that their pipeline is not robust because of their choice of using Bash at all.
And judging by your second comment, I can see that you have very strong opinions against bash for reasons that I donāt find convincing, other than what seems to me like irrational hatred from being rather uninformed. Itās fine being uninformed, but I suggest you tame your opinions and expectations with that.
About shared libraries, many popular languages, Python being a pretty good example, do rely on these to get performance that would be really hard to get from their own interpreters / compilers, or if re-implementing it in the language would be pretty pointless given the existence of a shared library, which would be much better scrutinized, is audited, and is battle-tested. libcrypto is one example. Pandas depends on NumPy, which depends on, I believe, libblas and liblapack, both written in C, and I think one if not both of these offer a cli to get answers as well. libssh is depended upon by many programming languages with an ssh library (though there are also people who choose to implement their own libssh in their language of choice). Any vulnerabilities found in these shared libraries would affect all libraries that depend on them, regardless of the programming language you use.
If production only implies systems in a userās path and not anything else about production data, then sure, my example is not production. That said though, I wouldnāt use bash for anything thatās in a userās path. Those need to stay around, possible change frequently, and not go down. Bash is not your language for that and thatās fine. Youāre attacking a strawman that youāve constructed here though.
If your temporary small script morphs into a monster and youāre still using bash, bash isnāt at fault. You and your team are. Youāve all failed to anticipate that change and misunderstood the ātemporaryā nature of your script, and allowed your ātemporary thingā to become permanent. Thatās a management issue, not a language choice. Youāve moved that goalpost and failed to change your strategy to hit that goal.
You could use Deno, but then my point stands. You have to write a function to handle the case where an env var isnāt provided, thatās boilerplate. You have to get a library for, say, accessing contents in Azure or AWS, set that up, figure out how that api works, etc, while you could already do that with the awscli and probably already did it to check if you could get what you want. Whatās the syntax for
mkdir? Whatās it formkdir -p? What about other options? If you already use the terminal frequently, some of these are your basic bread and butter and you know them probably by heart. Unless you start doing that with Deno, you wonāt reach the level of familiarity you can get with the shell (whichever shell you use ofc).And many argue against bash with regards to error handling. You donāt always need something that proper language has. You donāt always need to handle every possible error state differently, assuming you have multiple. Did it fail? Can you tolerate that failure? Yup? Good. No? Can you do something else to get what you want or make it tolerable? Yes? Good. No? Maybe you donāt want to use bash then.
Yes, because that is precisely the case. Itās not a personal attack, itās just a fact that Bash is not robust.
Youāre trying to argue that your cardboard bridge is perfectly robust and then getting offended that I donāt think you should let people drive over it.
You mean āthird party librariesā not āshared librariesā. But anyway, so what? I donāt see what that has to do with this conversation. Do your Bash scripts not use third party code? You canāt do a lot with pure Bash.
Well thatās why I donāt use Bash. Iām not blaming it for existing, Iām just saying itās shit so I donāt use it.
Handling errors correctly is slightly more code (āboilerplateā) than letting everything break when something unexpected happens. I hope you arenāt trying to use that as a reason not to handle errors properly. In any case the extra boilerplate isā¦
Deno.env.get("FOO"). Wow.await Deno.mkdir("foo"); await Deno.mkdir("foo", { recursive: true });Whatās the syntax for a dictionary in Bash? What about a list of lists of strings?