Pages

Tuesday, 18 July 2023

How Bash processes command args with variables and quotation marks

0. TLDR;

Only literal quotation marks right in the command line are treated  specially by Bash. Quotation marks as part of the value of a variable have no special meanings at all.

Bash array is treated differently than non-array variables.

1. Examples

To illustrate the command line processing in Bash, a simple script is used.

$ cat test.sh

#!/bin/bash
echo "I was given $# args as below:"
while [[ $# -ne 0 ]]; do
        echo "$1"
        shift
done

This script will print out the arguments it received. Let the code say now.

Ex1
$ ./test.sh hello world
I was given 2 args as below:
hello
world

Ex2
$ ./test.sh "hello world"
I was given 1 args as below:
hello world


Ex3
$ args="hello world"
$ ./test.sh $args
I was given 2 args as below:
hello
world

Ex4
$ args="hello world"
$ ./test.sh "$args"
I was given 1 args as below:
hello world


Ex5
$ args="'hello world'"
[zhaojon2@w0 argv]$ ./test.sh $args
I was given 2 args as below:
'hello
world'

2 Exaplanation

Bash metacharacters, characters with special meaning in bash, lose their special meaning if they are surronded by quotation marks.

One the the metacharacters, the "WHITE SPACE", is treated as borders between command line arguments.

Just as other metacharacters, if "WHITE SPACE" is surrounded by a pair of quotations marks, it is restored to the literal meaning, and not args seperator any more.

The quotation mark itself is a metacharacter as well. But it ONLY has this special meaning right in the command line. E.g 

X="--name 'Taylor Swift'"

The pair of double quotes are metacharacters, and treated specially by Bash.
The pair of single quotes sourrounding Taylor Swift are part of the variable X's value, have no special meanings.

$ X="--name 'Taylor Swift'" 
$ ./test.sh $X
I was given 3 args as below:
--name
'Tllaylor
Swift'

Bash treats the value of variable X literally, and does NOT think the single quotes differently. In fact, not only single quotes here, but also many other metacharacters in Bash like $ also lose special meaning when it's part of the value of a variable.

But, the WHITE SPACE in the variable's value is STILL treated as a metachracter.

3 In Practice

3.1 Problems

Below command line is used a lot in real life.

$ curl -H 'Content-Type: applicatoin/json' https://server/api/a
$ curl -H 'Content-Type: applicatoin/json' https://server/api/b
$ curl -H 'Content-Type: applicatoin/json' https://server/api/c

As a programmer, we certainly don't like the duplicated args in each line. So, you probably rewrite it as below.

CURL_OPTS="-H 'Content-Type: application/json'"
curl $CURL_OPTS https://server/api/a
curl $CURL_OPTS https://server/api/b
curl $CURL_OPTS https://server/api/c

Look good, right? Unfortunately, it doesn't work.
The reason is simple, the single quotes are not metacharecters anymore, but the spaces still are.
Let verify it by our script.

$ ./test.sh $CURL_OPTS
I was given 3 args as below:
-H
'Content-Type:
application/json'

3.2 Solutions

One solution is to forcefully let the BASH treat the variable's value as literal command line.

$ eval ./test.sh $CURL_OPTS
I was given 2 args as below:
-H
Content-Type: application/json

The bash works as before, giving 3 args to eval.

eval 
-H 
'Cotent-Type: 
application/json'

But the "eval" command doesn't care about its args much, it just connects all the args and treats the concatenated string as a script to run.

Let's try another test for this.

$ ./test.sh --name 'Taylor         Swift'
I was given 2 args as below:
--name
Taylor         Swift

$ args="--name 'Taylor         Swift'"
$ ./test.sh $args
I was given 3 args as below:
--name
'Taylor
Swift'
$ eval ./test.sh $args
I was given 2 args as below:
--name
Taylor Swift

The white spaces between Taylor and Swift were contracted into one.


A much better solution is bash array.

$ args=(hello world)
$ ./test.sh "${args[@]}"
I was given 2 args as below:
hello
world

Look, this looks unbelivable! Even double quoted array can give you mulitple arguments.
I don't understand the logic under the hood yet.

Update on Nov 2,2024: the quotes in "${args[@]}" are applied into the element of the array, not the array itself. So spaces in each element are protected. This rule is true for "$@" as well.

For our curl example above,

$ CURL_OPTS=('-H' 'Content-Type: application/json')
$ ./test.sh "${CURL_OPTS[@]}"
I was given 2 args as below:
-H
Content-Type: application/json

4 Useful docs


No comments:

Post a Comment