Process Substitution

2019 Dec 19

From man bash, for “process substitution” it says

It takes the form of <(list) or >(list)

Note: there is no space between < or > and (.

Also, it says,

The process list is run with its input or output connected to a FIFO or some file in /dev/fd. The name of this file is passed as an argument to the current command as the result of the expansion.

The <(list) Form

$ diff <(echo a) <(echo b)
< a
> b

Usually the diff takes two files and compares them. The process substitution here, <(echo a), creates a file in /dev/fd, for example /dev/fd/63. The stdout of echo a command is connected to /dev/fd/63. Meanwhile, /dev/fd/63 is used as an input file/parameter of diff command. Similar for <(echo b). After Bash does the substitution, the command is like diff /dev/fd/63 /dev/fd/64. In diff’s point of view, it just compares two normal files.

In this example, one advantage of process substitution is eliminating the need of temporary files, like

$ echo a > tmp.a && echo b > tmp.b \ 
    && diff tmp.a tmp.b            \
    && rm tmp.{a,b}

\ The >(list) Form

$ echo . | tee >(ls)

Similar, Bash creates a file in /dev/fd when it sees >(ls). Again, let’s say the file is /dev/fd/63. Bash connects /dev/fd/63 with stdin of ls command, also the file /dev/fd/63 is used as a parameter of tee command. The tee views /dev/fd/63 as a normal file. tee writes content, here is ., into the file, and the content will “pipe” into the stdin of ls.

Compare with Pipe

Pipe, cmd-a | cmd-b, basically just passes stdout of the command on the left to the stdin of the command on the right. Its data flow is restricted, which is from left to right.

Process substitution has more freedom.

# use process substitution
$ grep -f <(echo hello) file-a
# use pipe
$ echo hello | xargs -I{} grep {} file-a

And for commands like diff <(echo a) <(echo b), it’s not easy to be done by pipe.

More Examples

$ paste <(echo hello) <(echo world)
hello   world

How it works

From man bash,

Process substitution is supported on systems that support named pipes (FIFOs) or the /dev/fd method of naming open files.

Read more about named pipe and /dev/fd.

For /dev/fd,

The main use of the /dev/fd files is from the shell. It allows programs that use pathname arguments to handle standard input and standard output in the same manner as other pathnames.

In my OS (Ubuntu), the Bash uses /dev/fd for process substitution.

$ ls -l <(echo .)
lr-x------ 1 user user 64 12月 19 11:19 /dev/fd/63 -> pipe:[4926830]

Bash replaces <(echo .) with /dev/fd/63. The above command is like ls -l /dev/fd/63.

Or find the backing file via,

$ echo <(true)

(After my Bash does the substitution, the command becomes echo /dev/fd/63, which outputs /dev/fd/63.)

Deploy a Rails App with Docker

2019 Aug 14

This site is deployed using Docker containers on a small DigitalOcean Droplet.

Since I never have a private Docker registry, the original deploy flow was,

  • push the source code to the remote Git repository
  • on the Droplet, fetch the source code from the Git remote
  • build an Docker image from the source code
  • start the Docker container

Since some time ago, the docker build has kept failing on the bundle install instruction on the Droplet. During bundle install, the Bundler fails to build the sassc gem. The Bundler complains that,

virtual memory exhausted: Cannot allocate memory

, while building the sassc gem.

A workaround I tried was,

  • build the Docker image on a more powerful machine, like the local dev machine
  • save the image into a tar archive (docker save img-foo | bzip2 >
  • copy the tar to the Droplet
  • load the image from the tar (bunzip2 && docker load -i foo.img)
  • start the Docker container

One problem of the workaround was the volume to transfer is a bit large (the image is around 1G before compression, and 370M after compression). It was time consuming over a slow network.

Current Deploy Workflow

Actually, Bundler supports fetching gems from local cache, not only from Using bundle package --all can,

Copy all of the .gem files needed to run the application into the vendor/cache directory.

In the future, when running bundle install, use the gems in the cache in preference to the ones on

The sassc gem is built on the local machine while bundle package, and put into vendor/cache directory.

Now, the deploy flow is like,

  • run bundle package --all if the application’s dependencies change
  • check in vendor/cache folder and push it to the Git remote
  • on the Droplet, fetch the source code from the Git remote
  • build an Docker image from the source code
  • start the Docker container

The Dockerfile is like,

USER app
RUN mkdir -p /home/app/source-code
WORKDIR /home/app/source-code
COPY vendor/cache ./vendor/cache/
COPY Gemfile* ./
# use RVM, so "bash -lc ..."
RUN bash -lc 'bundle install --deployment --without development test'
# copy the app source code
COPY . ./

Other Notes

If a gem is located at a particular git repository using the :git parameter, like

gem "rails", "2.3.8", :git => ""

Then the --all option should be used in bundle package.

Some gems are built using CMake. When run bundle package --all, CMakeCache.txt files are created and stored in vendor/cache. The problem with CMakeCache.txt files is that they use hardcoded paths of the machine runs bundle package --all. When build the Docker image in the remote server, the hardcoded paths fail the bundle install. Therefore, in the .dockerignore file, add the following line.


Java Generics, Bounded Wildcards and PECS

2019 Aug 8

With a class hierarchy below,

class Fruit {
    public void hello() {}
class Apple extends Fruit {}
class Banana extends Fruit {}

Lower Bounded Wildcards and Upper Bounded Wildcards

? super Fruit is called lower bounded wildcard.

? extends Fruit is called upper bounded wildcard.

Usually a class hierarchy is illustrated like,

   /   \
Apple Banana

The base class is placed upper, subclasses placed lower.

For ? super Fruit, the lower bound of the type argument is determined, which is Fruit, so it’s called lower bounded wildcard. Similar, for type argument ? extends Fruit, its upper bound is fixed, so it’s called upper bounded wildcard.

”? super Fruit”

void func(List<? super Fruit> fruits) {
    fruits.add(new Fruit()); // ok
    fruits.add(new Apple()); // ok
    fruits.add(new Banana()); // ok
    // Error: Cannot resolve method 'hello'

The ? in the declaration List<? super Fruit> means fruits is a List of an unknown type. The ? super Fruit means the unknown type is limited to be Fruit or its super class (Object in this case). When invoke func(List<? super Fruit> ), List<Fruit> or List<Object> can be passed as the argument.

All the fruits.add(...) in the above code is ok, because no matter what type actually the type argument is Fruit, Apple, and Banana are the subtypes of that type. An Apple object is a Fruit (or Object) object, so an Apple object can be added into a List of Fruit (or Object).

Why fruits.get(0).hello() fails?

The actual type of the elements in the fruits List is undetermined when declare the method. When call a method on an element, the compiler needs to make sure the unknown type has the method. ? super Fruit in the type argument declaration sets bounds for the unknown type, one bound (“lower”) is type Fruit, the other bound (“upper”) is type Object To safely invoke methods on the unknown type, only methods of the most upper type can be invoked. Obviously, method hello is not from the most upper type (i.e. Object). If passing a List<Object> as the parameter, then obviously the elements in fruits would not have hello() method. Therefore the compiler raises the error.

”? extends Fruit”

void func(List<? extends Fruit> fruits){
    // Error: add(capture<? extends Fruit>) in List cannot be applied to add(Fruit)
    fruits.add(new Fruit()); 
    fruits.add(new Apple()); // similar error
    fruits.add(new Banana()); // similar error
    fruits.get(0).hello(); // ok

Similar, declaring the parameter as List<? extends Fruit> means fruits is an unknown type, and can be List<Fruit>, List<Apple> or List<Banana> when the method is invoked.

Why fruits.get(0).hello() is ok in this case?

Similar, ? extends Fruit sets bounds for the unknown type, the upper bound is determined as the type Fruit. The hello() method is from the most upper type, so the compiler knows it’s safe to call it within the func body.

Why statements like fruits.add(new Fruit()) fails?

The type of fruits is unknown, it doesn’t mean the type of fruits is “dynamic”. Being “dynamic” means the type of fruits can be List<Fruit> at some point, then be List<Apple> at some other point. It’s not possible in Java, since Java is a static language. The two types, List<Fruit> and List<Apple>, have nothing to do with each other (List<Apple> is not a subtype of List<Fruit>). The type of the fruits parameter is determined on the invocation of func. For a specific invocation, the type of fruits is determined and static. For example, with an invocation like func(new ArrayList<Apple>()), the statement fruits.add(new Fruit()) would raise compiler error (since a Fruit cannot be add()ed into a List<Apple>). To ensure all the possible invocation of func works, the compiler just can’t allow the statements like fruits.add(new Fruit()) appear in the method body.

What is “capture” (capture<? extends Fruit>)?

"ssh" to a Windows Host

2019 Jun 25

Every now and then, we may want to connect to a Windows host via ssh. Maybe it’s due to the frustration on connecting a Windows host via Remote Desktop Connection over a slow network.

Below is a way to ssh (not really, actually) to a Windows host.

Step 0, install a Linux virtual machine in the Windows host

I use Oracle Virtual Box, and install a Ubuntu guest.

Step 1, share folders between the Windows host and the guest

Share the working folders between the Windows and the Ubuntu, so that these folders are accessible within the guest. Just google “virtualbox share folder”.


Step 2, configure the guest’s network settings

Choose NAT as the adapter type.


In the “Advanced” settings, add a port forwarding rule. Choose an available port in the Windows host for “Host Port”. Set “Guest Port” to 22, which is the default port sshd listening on in the Ubuntu guest.

port forwarding

To find an available port in the Windows host, run netsh firewall show state in the cmd.exe. In my corporate network, for example, port 443 is allowed.

Step 3, make a connection

Open a ssh client in another machine, type the IP (or host name) of the Windows host as the hostname, use the “Host Port” as the port, 443 in my case, for the ssh connection.

When connected, change the directory to one of the shared folders set up in step 1, then do the work.

Other ways

One way is to set the adapter as “Bridged Adapter” for the Ubuntu guest, so that the guest get an IP from the corporate network. Configure the port of sshd to a port the corporate network allows to connect from outside. One inconvenience is that sometimes the corporate network may assign a new IP for the guest after reboot.

Another way is to install the OpenSSH Server directly in the Windows host.

Daily Dev Log

2019 Jun 24


“foo –help” vs “man foo”

We just can’t remember options of CLI tools. In most cases, --help, like grep --help, is the go-to way to look for help.

For example, if you forget what -H of grep does.

$ grep --help | grep -- -H
  -H, --with-filename       print the file name for each match

man is too formal, wordy and overwhelming, comparing with --help. Usually, we can find what we want in the output of --help, without turning to man.

Another advantage of --help is that it’s always available, and built in the command itself. If you have ever used Git Bash for Windows, you will find that man is not available there. Relying on man only means you have to google the man(ual) in the browsers.

Or to save some typing, add a bash function into ~/.bashrc.

h() { $1 --help; }

Then type h grep to show the help.

xargs is Slow

2019 May 29
# filepaths.txt is a file with thousands lines
cat filepaths.txt | xargs -n 1 basename

It takes a while (seconds) to finish running the above command. A file with thousands lines usually is not considered as a big volume. Why is xargs slow in the above command?

After read a SO post, it turns out xargs in the above command runs basename thousands times, therefore it has bad performance.

Can it be faster?

According to man xargs,

xargs reads items from the standard input … delimited by blanks … or newlines and executes the command … followed by items read from standard input. The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). … In general, there will be many fewer invocations of command than there were items in the input.
This will normally have significant performance benefits.

It means xargs can pass a batch of “items” to the command. Unfortunately, the -n 1 option in the command forces xargs to just take one “item” a time. To make it fast, use the -a option of basename, which let basename be able to handle multiple arguments at once.

time cat filepaths.txt | xargs -n 1 basename > /dev/null 

real    0m2.409s
user    0m0.044s
sys     0m0.332s
time cat filepaths.txt | xargs basename -a > /dev/null 

real    0m0.004s
user    0m0.000s
sys     0m0.000s

Thousands times faster.

cat /dev/null | xargs --show-limits --no-run-if-empty

Your environment variables take up 2027 bytes
POSIX upper limit on argument length (this system): 2093077
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2091050
Size of command buffer we are actually using: 131072
Maximum parallelism (--max-procs must be no greater): 2147483647

It shows xargs can feed a lot bytes into the command once (2091050 bytes here).


Some commands can usefully be executed in parallel too; see the -P option.

Fix an Escaped Hyperlink Bug

2019 Mar 26

There was a bug in timeline page. Hyperlinks (<a>) were unwanted escaped, showed in the page like below.

if a feedback is submitted in <a href="/about">About</a> page, ...

The fix is rather simple as below.

-    <li><%= p %></li>
+    <li><%= p.html_safe %></li>

Basically, html_safe tells rails not to HTML escape a string by claiming the string is “safe” (doing nothing on the string but setting a flag). html_safe should not be called on any user input strings, otherwise you’ll be under risk of XSS attacks. For this site, the fix is safe since all the content in the timeline page is solely input by myself, not by other malicious users.

More about html_safe

html_safe is a method of SafeBuffer, which is a String wrapper designed to prevent XSS attack. SafeBuffer is almost the same as String, beside having a difference behavior on concatenation. When a “safe” SafeBuffer A is appended by an “unsafe” SafeBuffer or String B, B will be HTML escaped before concatenation. By default a SafeBuffer/String is not marked safe.

Rails uses SafeBuffer to prevent XSS attack. For the below erb,

  <li><%= p %></li>

Rails translates it into something like,

  '<li>'.html_safe + p + '<li>'.html_safe

Therefore, p, which may be from user input, is HTML escaped when concatenated, and rendered safely in the result page.

More than One Way in Ruby

Using raw has the same effect, but show the intention in a much clearer way.

  <li><%= raw p %></li>

And one more, <%== is equivalent to raw, in case you really want to save some keystrokes.

will_paginate with Bootstrap

2019 Jan 29

will_paginate doesn’t come with Bootstrap style pagination by default. However, as its doc says, it does support customization by providing your own LinkRenderer.

to customize HTML output of will_paginate, you’ll need to subclass WillPaginate::ActionView::LinkRenderer

Below is a simple implementation to render pagination links with Bootstrap(4) css components.

# app/helpers/application_helper.rb

module ApplicationHelper
  def will_paginate(collection_or_options = nil, options = {})
    if collection_or_options.is_a? Hash
      options, collection_or_options = collection_or_options, nil
    unless options[:renderer]
      options = options.merge :renderer => BootstrapRenderer
    super *[collection_or_options, options].compact

  class BootstrapRenderer < WillPaginate::ActionView::LinkRenderer
    def html_container(html)
      tag :nav, tag(:ul, html, class: "pagination pagination-sm"), container_attributes

    def page_number(page)
      tag :li, link(page, page, rel: rel_value(page), class: 'page-link'),
        class: (page == current_page ? 'page-item active': 'page-item')

    def previous_or_next_page(page, text, classname)
      tag :li, link(text, page || '#', class: 'page-link'),
        class: ['page-item', classname, ('disabled' unless page)].join(' ')
# app/views/posts/index.html.erb

# use 'text-center' to center inline-blocks (<ul>) within <nav>
<%= will_paginate @posts, params: @will_paginate_params, class: 'text-center' %>

Daily Dev Log

2019 Jan 24


“su - app” vs “su app”

From man su,

   -, -l, --login
      Provide an environment similar to what the user would expect had the user logged in directly.

So with su - app, after switch to the user app, you end up in the user’s HOME directory, and have the user’s ~/.bash_profile (not ~/.bashrc) executed.

Tools like RVM need a “login shell”.

RVM by default adds itself currently to ~/.bash_profile file

So if use su app, RVM will not be ready there for you.

Daily Dev Log

2019 Jan 15


Convert Line Endings from DOS/Windows Style to Unix/Linux Style

tr -d '\015' <DOS-file >UNIX-file

For what character \015 is, see man 7 ascii. Or, there is a command called ascii, if it’s installed, type ascii '\015'.

More ascii Commands

$ ascii '\r'
ASCII 0/13 is decimal 013, hex 0d, octal 015, bits 00001101: called ^M, CR
Official name: Carriage Return
C escape: '\r'
Other names: 

Search Manuals

-k Search the short descriptions and manual page names for the keyword

$ man -k ascii
ascii (1)            - report character aliases
ascii (7)            - ASCII character set encoded in octal, decimal, and hexadecimal


Do Not Use the Same File to Redirect Input and Output

tr -d '\015' <DOS-file >DOS-file

The above command will delete all content in the file!

From man bash,

[n]>word, if it does exist it is truncated to zero size.

(How did I find the file back? Luckily, the working directory is managed by Dropbox, and I found it back in the Dropbox.)

← Previous