[This article no longer available from its original location, www.unixreview.com/articles/2003/0306/. This version is dated Feb 4 2009 and was downloaded from The Internet Archive — Wayback Machine. This article along with others on xargs is reprinted at Unix Xargs Piping Toolkit Utility.]

Using the xargs Command

Ed Schaefer

(July 2003 — see Web-exclusive update at end of article)

Many UNIX professionals think the xargs command, construct and execute argument lists, is only useful for processing long lists of files generated by the find command. While xargs dutifully serves this purpose, xargs has other uses. In this article, I describe xargs and the historical "Too many arguments" problem, and present eight xargs "one-liners":

Find the unique owners of all the files in a directory.
Echo each file to standard output as it deletes.
Duplicate the current directory structure to another directory.
Group the output of multiple UNIX commands on one line.
Display to standard output the contents of a file one word per line.
Prompt the user whether to remove each file individually.
Concatenate the contents of the files whose names are contained in file into another file.
Move all files from one directory to another directory, echoing each move to standard output as it happens.

Examining the "Too Many Arguments" Problem

In the early days of UNIX/xenix, it was easy to overflow the command-line buffer, causing a "Too many arguments" failure. Finding a large number of files and piping them to another command was enough to cause the failure. Executing the following command, from Unix Power Tools, first edition (O'Reilly & Associates):

pr -n 'find . -type f -mtime -1 -print'|lpr

will potentially overflow the command line given enough files. This command provides a list of all the files edited today to pr, and pipes pr's output to the printer. We can solve this problem with xargs:

find . -type f -mtime -1 -print|xargs pr -n |lp

With no options, xargs reads standard input, but only writes enough arguments to standard output as to not overflow the command-line buffer. Thus, if needed, xargs forces multiple executions of pr -n|lp.

While xargs controls overflowing the command-line buffer, the command xargs services may overflow. I've witnessed the following mv command fail -- not the command-line buffer -- with an argument list too long error:

find ./ -type f -print | xargs -i mv -f {} ./newdir

Limit the number of files sent to mv at a time by using the xargs -l option. (The xargs -i () syntax is explained later in the article). The following command sets a limit of 56 files at time, which mv receives:

find ./ -type f -print | xargs -l56 -i mv -f {} ./newdir

The modern UNIX OS seems to have solved the problem of the find command overflowing the command-line buffer. However, using the find -exec command is still troublesome. It's better to do this:

# remove all files with a txt extension
find . -type f -name "*.txt" -print|xargs rm

than this:

find . -type f -name "*.txt" -exec rm {} \; -print

Controlling the call to rm with xargs is more efficient than having the find command execute rm for each object found.

xargs One-Liners

The find-xargs command combination is a powerful tool. The following example finds the unique owners of all the files in the /bin directory:

# all on one line
find /bin -type f -follow | xargs ls -al | awk ' NF==9 { print $3 }
'|sort -u

If /bin is a soft link, as it is with Solaris, the -follow option forces find to follow the link. The xargs command feeds the ls -al command, which pipes to awk. If the output of the ls -al command is 9 fields, print field 3 -- the file owner. Sorting the awk output and piping to the uniq command ensures unique owners.

You can use xargs options to build extremely powerful commands. Expanding the xargs/rm example, let's assume the requirement exists to echo each file to standard output as it deletes:

find . -type f -name "*.txt" | xargs -i ksh -c "echo deleting {}; rm {}"

The xargs -i option replaces instances of {} in a command (i.e., echo and rm are commands).

Conversely, instead of using the -i option with {}, the xargs -I option replaces instances of a string. The above command can be written as:

find . -type f -name "*.txt" | xargs -I {} ksh -c "echo deleting {}; rm {}"

The new, third edition of Unix Power Tools by Powers et al. provides an xargs "one-liner" that duplicates a directory tree. The following command creates in the usr/project directory, a copy of the current working directory structure:

find . -type d -print|sed 's@^@/usr/project/@'|xargs mkdir

The /usr/project directory must exist. When executing, note the error:

mkdir: Failed to make directory "/usr/project/"; File exists

which doesn't prevent the directory structure creation. Ignore it. To learn how the above command works, you can read more in Unix Power Tools, third edition, Chapter 9.17 (O'Reilly & Associates).

In addition to serving the find command, xargs can be a slave to other commands. Suppose the requirement is to group the output of UNIX commands on one line. Executing:

logname; date

displays the logname and date on two separate lines. Placing commands in parentheses and piping to xargs places the output of both commands on one line:

(logname; date)|xargs

Executing the following command places all the file names in the current directory on one line, and redirects to file "file.ls":

ls |xargs echo > file.ls

Use the xargs number of arguments option, -n, to display the contents of "file.ls" to standard output, one name per line:

cat file.ls|xargs -n1  # from Unix in a Nutshell

In the current directory, use the xargs -p option to prompt the user to remove each file individually:

ls|xargs -p -n1 rm

Without the -n option, the user is prompted to delete all the files in the current directory.

Concatenate the contents of all the files whose names are contained in file:

xargs cat < file > file.contents

into file.contents.

Move all files from directory $1 to directory $2, and use the xargs -t option to echo each move as it happens:

ls $1 | xargs -I {} -t mv $1/{} $2/{}

The xargs -I argument replaces each {} in the string with each object piped to xargs.

Conclusion

When should you use xargs? When the output of a command is the command-line options of another command, use xargs in conjunction with pipes. When the output of a command is the input of another command, use pipes.

References

Powers, Shelley, Peek, Jerry, et al. 2003. Unix Power Tools. Sebastopol, CA: O'Reilly & Associates.

Robbins, Arnold. 1999. Unix in a Nutshell. Sebastopol, CA: O'Reilly & Associates.

Ed Schaefer is a frequent contributor to Sys Admin. He is a software developer and DBA for Intel's Factory Integrated Information Systems, FIIS, in Aloha, Oregon. Ed also hosts the monthly Shell Corner column on UnixReview.com. He can be reached at: shellcorner@comcast.net.

July 2003 UPDATE from the author:

I've received very positive feedback on my xargs article. Other readers have shared constructive criticism concerning:

1. When using the duplicate directory tree "one-liner", reader Peter Ludemann suggests using the mkdir -p option:

   find . -type d -print|sed 's@^@/usr/project/@'|xargs mkdir -p

instead of :

   find . -type d -print|sed 's@^@/usr/project/@'|xargs mkdir

mkdir's "-p" option creates parent directories as needed, and doesn't error out if one exists. Additionally, /usr/project does not have to exist.

2. Ludemann, in addition to reader Christer Jansson, commented that spaces in directory names renders the duplicate directory tree completely useless.

Although I'm unable to salvage the duplicate directory command, for those find and xargs versions that support -0 (probably GNU versions only), you might try experimenting with:

find ... -print0 | xargs -0 ...

Using Ludemann's email example, suppose your current directory structure contains:

   foo
   bar
   foo bar

find . -type f -print | xargs -n 1 incorrectly produces:

   foo
   bar
   foo
   bar

while find . -type f -print0 | xargs -0 -n 1 delivers the correct results:

   foo
   bar
   foo bar

According to the 7.1 Red Hat Linux man page for xargs and find, the -0 doesn't use the null terminator for file names disabling the special meaning of white space.

3. Reader Peter Simpkin asks the question, "Does the use of xargs only operate after the find command has completed?

find. -type f -name "*.txt" -print | xargs rm

If not, I was under the impression that the above was a bad idea as it is modifying the current directory that find is working from, or at least this is what people have told me, and, thus the results of find are then undefined."

My response is "no". Any Unix command that supports command-line arguments is an xargs candidate. The results of the find command are as valid as the output of the ls command:

# remove files ending with .txt in current directory 
ls *.txt|xargs rm

If a command such as this is valid:

chmod 444 1.txt 2.txt 3.txt

then:

find . \( -name 1.txt -o -name 2.txt -o -name 3.txt \) -print|xargs chmod 444

is valid.

In closing, If I had the opportunity to rewrite "Using the Xargs Command", it would look somewhat different.