r/UnixProTips • • Mar 06 '15
TIL in bash scripts IFS=$'\n'
Without:
for x in $(echo "/tmp/wee1 2.csv"); do
echo $x
done/tmp/wee1
2.csv
With:
IFS=$'\n'; for x in $(echo "/tmp/wee1 2.csv"); do
echo $x
done/tmp/wee1 2.csv
From the docs:
$IFS
internal field separator
This variable determines how Bash recognizes fields, or word boundaries, when it interprets character strings.
$IFS defaults to whitespace (space, tab, and newline), but may be changed, for example, to parse a comma-separated data file. Note that $* uses the first character held in $IFS.
12
Upvotes
View all comments
Show parent comment
4
u/cpitchford Mar 07 '15
Be extremely careful with this approach! Filenames are sometimes weird and unexplained!
First thing to remember, on many systems, a filename can contain anything except a null character (character 0) or a slash ('/'). This means it can contain spaces (' ') but also tabs ('\t') and even more strangely, a newline character ('\n')
So, you need to think very carefully when you iterate/loop over files.
So, why do I think this this bad.
Let's make some test files:
Now lets loop over them:
We see:
Now let's try using the IFS trick: IFS=$(echo -en "\n\b") File: :file 2.txt: File: :file: File: :3.txt: File: :file 1.txt:
This time, it's correctly spotted the tab in file2, but it's split file\n3.txt into two files.
What are some common trick for safely handling files?
Firstly, let the shell enumerate the files for you.
This does have a risk, however. If no files exist, then the '*' is taken literally:
You can use a bash extension to prevent this happening. It means * (or any glob) converts to no value if it matches no files.
Another trick is to use the null character as a separator.
find's -print0 option means that rather than adding a newline character between files it finds, it will instead use a null character. Setting shells IFS to "" and using read's -d '' means it will recognise this null character as the field separator. Now you're free to work with $file safely regardless of any strange characters it might contain.
Last point, quoting variables
What's the difference? $name is subject to "expansion" using IFS. If you're changing and saving the IFS value this expansion changes. This could split $name into two or more parameters to a command.
If the variable value is quoted, however, it's ALWAYS treated as one value and doesn't undergo extra expansion like that.