Browsers Ignore Change in Hosts File

2018 Aug 2

I modified C:\Windows\System32\drivers\etc\hosts for a local test. However, the browser did not respect to the change in hosts file. At last, I found it’s due to the proxy settings in my machine. Change in hosts took effect once after I unchecked all proxy configuration in Control Panel -> Internet Options -> Connections -> LAN settings.

If unfortunately in your network, connections are proxied by force (for example, in a corporate network), you can try to let the proxy bypass some domains by adding the domains into LAN settings -> Proxy server -> Advanced -> Exceptions.

Bug Tracking and Feature Planning

2018 Jul 27

Now OneFeed is using Trello to track bugs and plan features, though it’s a tiny application in any respect. A nice thing of having a bug tracking system is that history is logged, which can be used for future reference.

Trello is a simple tool to create/manage boards, lists, and cards. And based on these basic elements, one can build his/her own workflows.

trello usecase

Trello has a free plan, which is enough for OneFeed to track its bugs. The free plan allows users to add one Power-Up (add-on to extend Trello’s capability) per board. For OneFeed’s boards, Custom Fields Power-Up is added, so that cards can have custom fields like “Bug ID” and “Feature ID”, which can be used for example in Git commit messages.

bug card

Also Trello has mobile apps for accessing from anywhere :)

OneFeed is Serving HTTPS

2018 Jul 20

Now OneFeed is living on https. Its certificate is signed by Let’s Encrypt, “a free, automated, and open Certificate Authority”. Using Let’s Encrypt’s ACME (Automatic Certificate Management Environment) protocol and a client of the protocol, requesting and renewing site certificate is just done automatically.

ACME protocol defines serveral challenges which a protocol client can use to prove it (the host running the client) owns the domain. Also the protocol defines how to request, renew, and revoke certificates. With clear definition of interaction with ACME server (CA) and client (your site), all process can be automated. Certbot is a recommended ACME client.

To set up https on this site, I use this great post as a reference. Basic steps are:

  • use a certbot docker image to get certificate from Let’s Encrypt for the first time.
  • update configuration of web server using the certificate.
  • set up a cron job to auto-renew the certificate.
Use Docker Images to Get Certificate

Certbot is in active development. Use the certbot docker image (by default latest image), so that we don’t bother ourselves with updating certbot to newest version. And use the nginx docker image to set up a basic web server to fulfill ACME challenges, so that our production web server’s configuration gets untouched when requesting a certificate. (Also plus since OneFeed is already living within docker containers, using docker/docker-compose is an easy decision.) The containers used in this step are discarded/cleaned up as soon as certificate fetched for the first time.

Update Configuration of the Production Web Server

Just google how to set up https on the web server. For OneFeed, https is set up on a nginx server.

Set Up a Cron Job to Renew and Reload/Restart Web Server

The reference post uses docker kill --signal=HUP production-nginx-container to send signal to nginx container’s nginx process for server reloading. However, since OneFeed is not using plain nginx container, but a passenger-docker, therefore using docker-compose restart to reload certificate instead.

0 23 * * * docker run --rm -it --name certbot-renew \
-v /CERTBOT_VOLUME/etc/letsencrypt:/etc/letsencrypt \
-v /CERTBOT_VOLUME/var/lib/letsencrypt:/var/lib/letsencrypt \
-v /CERTBOT_VOLUME/data/letsencrypt:/data/letsencrypt \
-v /CERTBOT_VOLUME/var/log/letsencrypt:/var/log/letsencrypt \
certbot/certbot renew --webroot -w /data/letsencrypt --quiet \
&& cd YOUR_DOCKER_COMPOSE_WORKING_DIR \
&& docker-compose restart

X-Forwarded-For, Forwarded, X-Real-IP and Nginx

2018 May 22
X-Forwarded-For

Http header X-Forwarded-For can be used to get the IP address of the REAL client, especially in a network with proxies and load balancers.

The X-Forwarded-For (XFF) header is a de-facto standard header for identifying the originating IP address of a client connecting to a web server through an HTTP proxy or a load balancer. When traffic is intercepted between clients and servers, server access logs contain the IP address of the proxy or load balancer only. To see the original IP address of the client, the X-Forwarded-For request header is used.

The syntax is,

X-Forwarded-For: <client>, <proxy1>, <proxy2>
X-Forwarded-For: 203.0.113.195, 70.41.3.18, 150.172.238.178

When a Http request flows through a proxy, the proxy appends its IP address to X-Forwarded-For header (if it respects this header).

Forwarded

However, since X- headers are not recommended anymore,

Custom proprietary headers can be added using the ‘X-‘ prefix, but this convention was deprecated in June 2012.

a standardized and enhanced header, Forwarded, is introduced.

# the original request is from 192.0.2.60, and passed through proxy 203.0.113.43
Forwarded: for=192.0.2.60; proto=http; by=203.0.113.43

# client can also append some obfuscated identifier like "secret" here, server can 
# then use it validate the integrity of a client.
Forwarded: for=23.45.67.89;secret=egah2CGj55fSJFs, for=10.1.2.3
X-Real-IP

Another somehow relevant header is X-Real-IP, which contains a single IP. You may find it, for example, somewhere in Nginx docs (ngx_http_proxy_module doc, ngx_http_realip_module doc).

在Windows上安装tmux

2018 Apr 11

Windows上的Git BASH提供了大部分常用的Linux命令行工具,比如grep、sed等,但是并没有提供tmux。 实际上Git for Windows提供了包管理(package management)功能,

Git for Windows is based on MSYS2 which bundles Arch Linux’ Pacman tool for dependency management.

借助pacman,Git for Windows可以安装额外的命令行工具,比如tmux。 但是,在Git BASH里,pacman并没有默认开启

This is intended. We do not ship pacman with Git for Windows. If you are interested in a fully fledged package manager maintained environment you have to give the Git for Windows SDK a try.

需要安装Git for Windows SDK来开启pacman。 安装好之后,打开Git SDK(和Git Bash一样,是一个终端模拟器),

$ pacman -Ss tmux

会找到两个包,

msys/tmux 2.6-1
	A terminal multiplexer
msys/tmux-git 2.5.94.g73b9328c-1 
	A terminal multiplexer
$ pacman -S msys/tmux-git

安装的时候可能会报下面的错误,

$ pacman -S msys/tmux
warning: database file for 'git-for-windows-mingw32' does not exist
error: failed to prepare transaction (could not find database)

打开/etc/pacman.conf文件,注释掉下面的行即可,

#[git-for-windows]
#Server = https://wingit.blob.core.windows.net/x86-64

#[git-for-windows-mingw32]
#Server = https://wingit.blob.core.windows.net/i686

安装好之后,就可以在Windows上(Git SDK)使用tmux了。

tmux on Windows

pacman的用法可参见Git for Windows的Wiki

环境:Windows 10

(如果发现某些程序,比如ssh,报错,可以尝试用pacman -Syu升级所有package。)

删除Arrays.asList返回的列表的元素会发生异常

2018 Feb 5

Arrays.asList(...)返回的List进行remove()/removeAll()操作时,会抛出UnsupportedOperationException异常。 据Arrays.asList(...)javadoc,这个方法返回的List实现类是基于数组的。

Returns a fixed-size list backed by the specified array. (Changes to the returned list “write through” to the array.)

这个List实现类是Arrays类的一个私有静态类,所有方法基本上只是简单地代理到内部的一个数组成员E[]。 数组是不支持删除操作的,所以remove()会抛异常。

实际上对所有基于数组的List实现类最好都不要进行删除操作。 ArrayList虽然支持remove(),但是remove()的实现会导致内部数组的拷贝“平移”,影响效率。

Guava Cache异步刷新的一个实现

2018 Jan 16

Guava的cache提供了refresh功能。 在指定的时间间隔后,guava可以(惰性地/lazily)更新缓存。 默认的refresh实现是同步的(synchronously),一个线程在更新缓存时,其它线程会等待。 具体见LoadingCacheCacheLoader的javadoc。

LoadingCache
void refresh(K key)
… Loading is asynchronous only if CacheLoader.reload(K, V) was overridden with an asynchronous implementation.

CacheLoader
public ListenableFuture<V> reload(K key, V oldValue) throws Exception
… This implementation synchronously delegates to load(K).

下面提供了一个异步的CacheLoader实现,使得一个线程在refresh缓存时,其它线程可以不必等待,继续使用旧的缓存值。

不要在HTTP的GET请求里携带Request Body

2017 Nov 30

在设计RESTful service时,可不可以在一个GET请求中携带body? 答案是:最好不要这些做。

有一些RESTful service允许GET请求携带request body,比如Elasticsearch的search接口。 但这不是一种好的实现。 HTTP的规范在定义GET的语义时,规定server端应该只根据请求的URI来返回response。

[…] if the request method does not include defined semantics for an entity-body, then the message-body SHOULD be ignored when handling the request. […] The GET method means retrieve whatever information ([…]) is identified by the Request-URI. […] A payload within a GET request message has no defined semantics; sending a payload body on a GET request might cause some existing implementations to reject the request.

一个特定的http server,比如Elasticsearch,可以支持带body的GET。 但是也有很多http server会遵守规范,忽略GET的body,比如代理、cache server等。 假如在Elasticsearch前放置一个http cache server,比如Varnish,那么Varnish在丢弃GET的body后会返回一个错误的响应。

另外,一些http的客户端也可能不支持发送带body的GET请求,比如一些Javascprit的http client。 (虽然用X-HTTP-Method-Override header可以让http client以POST的方式发送GET请求来绕过这个限制, 但是如果请求流过某些代理,这些自定义的header可能会被丢弃。)

所以如果要在GET请求里携带数据,最好放在query string里,或者自定义的header(比如X-CSRF-TOKEN)、cookie里。

实际上,Elasticsearch也意识到这个问题,所以它支持通过query string来进行搜索。 (把整个body url encode之后放在source参数里,然后在source_content_type参数里指定source的内容格式是application/json。) Elasticsearch还保留着对带body的GET的支持,其中一个原因是大部分http client对URL的长度有限制。 如果你不像Elasticsearch这样需要在GET请求中发送大量数据,最好还是使用query string来发送数据。

运行时发生IncompatibleClassChangeError异常

2017 Nov 28

执行Java程序时遇到一个IncompatibleClassChangeError异常。 IncompatibleClassChangeError的javadoc是这么说的,

Thrown when an incompatible class change has occurred to some class definition. The definition of some class, on which the currently executing method depends, has since changed.

意思是Java程序在执行时发现类的定义发生了变化。

通常,如果编译时依赖的类定义和运行时依赖的类定义不一致就有可能抛出这种异常。

比如,有如下程序,

 public class ChildType extends BaseType {
 }

在编译时,BaseType是一个类,编译顺利通过。 在运行时,classpath里的BaseType如果是一个接口,就会在运行时抛出IncompatibleClassChangeError异常。

其它会抛出IncompatibleClassChangeError异常的场景有,

  • implements a class
  • 静态成员变成非静态成员
  • 非静态成员变成了静态成员

我的Java程序抛出IncompatibleClassChangeError异常的原因是运行时的classpath里同时存在不同版本Guava jar包。 在一个早期的Guava版本里,Ticker是一个接口(且这个Ticker定义更早地在运行时被加载); 在另一个较新的Guava版本里,Ticker变成了一个类,并有其它代码extends Ticker。 所以在运行到新版本的Guava代码时,就会抛出异常。

另外,升级库版本时,为了防止运行时抛出IncompatibleClassChangeError异常,最好针对新的库重新编译程序 (这样编译时就能发现错误); 而不是简单替换库的jar包。

Java的赋值操作是原子的

2017 Nov 23
  Object object = someObject;

Java的写操作是原子的(atomic),所以上面的引用赋值操作是一个原子操作。 (一个例外,对longdouble类型的写操作是非原子的;会分成两步32位的写操作。)

详见JLS:

Writes to and reads of references are always atomic, regardless of whether they are implemented as 32-bit or 64-bit values.

操作是原子的,不代表操作的结果对于其它线程的可见性(visibility)。 可见性由“happens-before”来保证。