From man bash, for “process substitution” it says

It takes the form of <(list) or >(list)

Note: there is no space between < or > and (.

Also, it says,

The process list is run with its input or output connected to a FIFO or some file in /dev/fd. The name of this file is passed as an argument to the current command as the result of the expansion.

The <(list) Form

$ diff <(echo a) <(echo b)
1c1
< a
---
> b

Usually the diff takes two files and compares them. The process substitution here, <(echo a), creates a file in /dev/fd, for example /dev/fd/63. The stdout of echo a command is connected to /dev/fd/63. Meanwhile, /dev/fd/63 is used as an input file/parameter of diff command. Similar for <(echo b). After Bash does the substitution, the command is like diff /dev/fd/63 /dev/fd/64. In diff’s point of view, it just compares two normal files.

In this example, one advantage of process substitution is eliminating the need of temporary files, like

$ echo a > tmp.a && echo b > tmp.b \ 
    && diff tmp.a tmp.b            \
    && rm tmp.{a,b}

The >(list) Form

$ echo . | tee >(ls)

Similar, Bash creates a file in /dev/fd when it sees >(ls). Again, let’s say the file is /dev/fd/63. Bash connects /dev/fd/63 with stdin of ls command, also the file /dev/fd/63 is used as a parameter of tee command. The tee views /dev/fd/63 as a normal file. tee writes content, here is ., into the file, and the content will “pipe” into the stdin of ls.

Compare with Pipe

Pipe, cmd-a | cmd-b, basically just passes stdout of the command on the left to the stdin of the command on the right. Its data flow is restricted, which is from left to right.

Process substitution has more freedom.

# use process substitution
$ grep -f <(echo hello) file-a
hello
# use pipe
$ echo hello | xargs -I{} grep {} file-a
hello

And for commands like diff <(echo a) <(echo b), it’s not easy to be done by pipe.

More Examples

$ paste <(echo hello) <(echo world)
hello   world

How it works

From man bash,

Process substitution is supported on systems that support named pipes (FIFOs) or the /dev/fd method of naming open files.

Read more about named pipe and /dev/fd.

For /dev/fd,

The main use of the /dev/fd files is from the shell. It allows programs that use pathname arguments to handle standard input and standard output in the same manner as other pathnames.

In my OS (Ubuntu), the Bash uses /dev/fd for process substitution.

$ ls -l <(echo .)
lr-x------ 1 user user 64 12月 19 11:19 /dev/fd/63 -> pipe:[4926830]

Bash replaces <(echo .) with /dev/fd/63. The above command is like ls -l /dev/fd/63.

Or find the backing file via,

$ echo <(true)
/dev/fd/63

(After my Bash does the substitution, the command becomes echo /dev/fd/63, which outputs /dev/fd/63.)

# filepaths.txt is a file with thousands lines
cat filepaths.txt | xargs -n 1 basename

It takes a while (seconds) to finish running the above command. A file with thousands lines usually is not considered as a big volume. Why is xargs slow in the above command?

After read a SO post, it turns out xargs in the above command runs basename thousands times, therefore it has bad performance.

Can it be faster?

According to man xargs,

xargs reads items from the standard input … delimited by blanks … or newlines and executes the command … followed by items read from standard input. The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). … In general, there will be many fewer invocations of command than there were items in the input.
This will normally have significant performance benefits.

It means xargs can pass a batch of “items” to the command. Unfortunately, the -n 1 option in the command forces xargs to just take one “item” a time. To make it fast, use the -a option of basename, which let basename be able to handle multiple arguments at once.

time cat filepaths.txt | xargs -n 1 basename > /dev/null 

real    0m2.409s
user    0m0.044s
sys     0m0.332s

time cat filepaths.txt | xargs basename -a > /dev/null 

real    0m0.004s
user    0m0.000s
sys     0m0.000s

Thousands times faster.

–show-limits

cat /dev/null | xargs --show-limits --no-run-if-empty

Your environment variables take up 2027 bytes
POSIX upper limit on argument length (this system): 2093077
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2091050
Size of command buffer we are actually using: 131072
Maximum parallelism (--max-procs must be no greater): 2147483647

It shows xargs can feed a lot bytes into the command once (2091050 bytes here).

-P

Some commands can usefully be executed in parallel too; see the -P option.

脚本文件（可执行的脚本文件）的第一行会有一个shebang，比如#!/bin/sh，表示用哪个程序来执行该脚本。 每种可执行文件的开头会有一个相应的magic number。 可执行的脚本文件对应的magic number是0x230x21，也就是ASCII码形式的#!。

在shebang里使用env可以增加脚本的可移植性。 比如#!/usr/bin/env ruby相比于#!/usr/bin/ruby有着更好的可移植性。 因为在某个系统上，ruby不一定安装在/usr/bin/下，也有可能安装在/usr/local/bin/等其它目录里。 而应用#!/usr/bin/env ruby这种形式的shebang时，只要在执行脚本的用户的PATH里能搜索到ruby程序，就可以执行脚本。

当用rbenv来管理ruby版本时，ruby程序一般会安装在用户目录下，然后rbenv通过设置用户的PATH变量来找到需要的ruby版本。 这种情况下使用#!/usr/bin/env ruby会保证脚本在执行时可以找到正确的ruby程序。

在绝大部分的系统上，env都安装在/usr/bin/目录下，所以不用担心/usr/bin/env的可移植性。

参考

Shebang (Unix)

man env

env – set environment and execute command, or print environment

The env utility executes another utility after modifying the environment as specified on the command line.

OneFeed的push notification实现借助了Sneakers。 在部署服务时，Ruby环境是通过rbenv管理的，环境变量（比如API Token等）则通过rbenv-vars管理。 为了不“污染”root用户以及系统的稳定，希望新建一个专门的用户来运行服务。

可以通过foreman把Sneakers项目导出成Upstart或者systemd service。 因为Ubuntu 16.04用systemd替换了Upstart，所以尝试用systemd来启动服务。

sudo foreman export systemd /etc/systemd/user -a notification-service -u hong

因为想以非root用户来启动服务，所以导出时加了-u选项指定用户，/etc/systemd/user是存放systemd user unit的目录。 导出的.service文件里会出现User=hong语句。

尝试启动服务，

su - hong
systemctl --user start notification-service.target

却始终报错，Failed to connect to bus: No such file or directory ubuntu。 Google后发现是su的问题。不用su，直接以用户hong登陆系统后再运行systemctl，上面的错误就消失了。 但是通过ps命令没有发现相关的ruby进程。再通过journalctl -r发现了如下错误信息，

Failed at step GROUP spawning /bin/bash: Operation not permitted

这可能是systemctl --user的一个bug。 一个解决方案是删除User=hong语句。但是删除该语句后，systemctl会尝试用root用户的ruby来启动服务。 因为root用户的ruby没有安装服务所依赖的gem，所以会启动失败。 而且以root用户来启动服务也违背了初衷。

到此为止，这次对systemd的尝试以失败告终。而且前前后后花费了一两天的时间。 所以专门的DevOps还是必须的🤔。

The Linux Programming Interface，💯。 亚马逊上的评价98%为五星，2%为四星。即便不做Linux开发，读一遍对于了解Linux还是非常有帮助的。

亚马逊链接。 豆瓣链接。

How is the /tmp directory cleaned up

永远不要把重要文件放到/tmp目录下，即使是临时存放也不要，因为只要系统一重启/tmp目录就会被清空。

曾经在Ubuntu虚拟机上测试OneFeed部署的时候，想记录下部署时涉及到的每个命令方便后来在生产环境部署。 因为涉及到多个Linux user（一个没有sudo权限的Linux user用于跑Rails App，另一个有sudo权限的user用于安装各种软件）， 所以就把文件放在了/tmp目录下。 在差不多部署好的时候，为了能在主机上访问虚拟机上部署的应用，需要配置VirtualBox的Host-only Networks， 配置好后需要重启虚拟机。。。😱

The cleaning of /tmp is done by the upstart script /etc/init/mounted-tmp.conf. The script is run by upstart everytime /tmp is mounted. Practically that means at every boot.

The script does roughly the following: if a file in /tmp is older than $TMPTIME days it will be deleted.

The default value of $TMPTIME is 0, which means every file and directory in /tmp gets deleted. $TMPTIME is an environment variable defined in /etc/default/rcS.

Linux共享库简介

之前写的关于Linux上的共享库（shared library）的一个简单介绍。