Let’s Encrypt is a free, open, and automated certificate authority. And its Certbot is a fully-featured, extensible client for Let’s Encrypt CA that can automate the tasks of getting, renewing and even installing SSL certificates.

Sounds great! However, not yet to be simple and automated, especially working cloud providers such as Google Cloud Platform and its Google App Engine or GAE.

But it’s free. Yes, it’s free. Free software works better. Free certificate authority works better than others.

GAE is a managed service. The place to stored SSL certificate is in separate machines (load balancers). The current automated domain validation by Certbot mostly work with a single machine. Therefore, when the machine issues certificate request is not the same machine to be validated, we need find another way, hopefully an automated method to perform domain validation across machines.

Before creating an automated method, let’s see if we can do it manually. Certbot supports a number of different plugins that can be used to obtain and/or install certificates. A plugin is like an extension that supports a particular web server. Let’s see if we can find a plugin that supports GAE.

Here are some supported by Certbot:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

$ certbot --help plugins
plugins:
  Certbot client supports an extensible plugins architecture. See 'certbot
  plugins' for a list of all installed plugins and their names. You can
  force a particular plugin by setting options provided below. Running
  --help  will list flags specific to that plugin.
  --apache              Obtain and install certs using Apache (default: False)
  --nginx               Obtain and install certs using Nginx (default: False)
  --standalone          Obtain certs using a "standalone" webserver. (default:
                        False)
  --manual              Provide laborious manual instructions for obtaining a
                        cert (default: False)
  --webroot             Obtain certs by placing files in a webroot directory.
                        (default: False)

And there are also a number of third-party plugins, see the User Guide in Certbot Documentation. But there is none for GAE. It looks like there are only three possible options to try: standalone, webroot and manual.

Let’s start with the standalone method, and issue that from the local machine:

1

$ sudo certbot certonly --standalone -d example.com

If you’re the first time running the command, you will be prompted for email and agreement screens. Both email and agreement can be automated via --email and --agree-tos options. That’s the automated part.

After freeing up the ports 80 and 443, run into some issues:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

Failed authorization procedure. example.com (tls-sni-01): urn:acme:error:connection 
:: The server could not connect to the client to verify the domain :: Failed to 
connect to 0.0.0.0:443 for TLS-SNI-01 challenge, example.com (tls-sni-01): 
urn:acme:error:connection :: The server could not connect to the client to verify the 
domain :: Failed to connect to 0.0.0.0:443 for TLS-SNI-01 challenge
IMPORTANT NOTES:
 - The following errors were reported by the server:
   Domain: example.com
   Type:   connection
   Detail: Failed to connect to 0.0.0.0:443 for TLS-SNI-01
   challenge
   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

The standalone plugin runs its own simple web server to prove that you control the domain. Ownership or domain validation is the key here. It needs the current computer that just issued the certbot command to have a publicly routable IP address. That’s not going to be happening in my local computer behind NAT. And webroot plugin needs a running web server. It can’t be run from the local machine as well. Domain validation are done automatically with both standalone and webroot plugins. Furthermore, domain validation requests are coming from Let’s Encrypt servers, therefore, you can’t have the machine issuing the certificate request behind a NAT or load balancing methods without properly routing the requests.

Since automated methods mostly require the requester and domain owner to be residing on the same machine, we can try to move the request to the Google cloud. Otherwise, there is one more plugin to try, the manual plugin. The manual method (plugin) helps you obtain a cert by giving you instructions to perform domain validation yourself.

After deploying a Node.js app with custom runtime to Google App Engine, when uploading file larger than 1MB, the server returns HTTP 413 status code, meaning the server is refusing to process the request because the uploading entity size is too large.

I had the following setup:

• The App Engine application is Google managed
• The application is Node.js app with custom Dockerfile
• The application Docker container runs node command directly on port 8080

After accessing the server (which changed from Google managed to user managed), and checked the Docker containers:

1
2

$ sudo docker ps -a
gcr.io/google_appengine/nginx-proxy:latest      "/usr/sbin/nginx"

It looks like Nginx is running as the proxy to Node.js application. When uploading file that is larger than 1MB, the response is HTTP 413 error:

1
2
3
4
5
6
7
8
9
10
11
12

<html>
    <head>
        <title>413 Request Entity Too Large</title>
    </head>
    <body bgcolor="white">
        <center>
            <h1>413 Request Entity Too Large</h1>
        </center>
        <hr>
            <center>nginx</center>
        </body>
    </html>

And the Nginx access log from the server:

1
2

$ tail /var/log/nginx/error.log 
2015/06/25 12:00:00 [error] 7#0: *101623 client intended to send too large body: 1696805 bytes

By default, Nginx limits uploading size or client_max_body_size to be 1MB. The default size is just too small. A short video upload could easily exceed the limit. So, how to accept file upload larger than 1MB via Google Managed VMs in App Engine? Is there a way to change the value through app.yamlcustom runtime configuration file? Or a custom Nginx Docker container can be deployed along with the application Docker container?

I was helping to examine a server that was impacted by Heartbleed. According to the developer who was patching the server, he had updated the OpenSSL library to the following:

1
2
3
4
5
6
7
8
9
10
11
12

$ openssl version -a
OpenSSL 1.0.1g 7 Apr 2014
built on: Fri Apr 18 11:04:34 EDT 2014
platform: linux-x86_64
options: bn(64,64) rc4(16x,int) des(idx,cisc,16,int) idea(int) 
blowfish(idx) 
compiler: gcc -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H 
-Wa,--noexecstack -m64 -DL_ENDIAN -DTERMIO -O3 -Wall -DOPENSSL_IA32_SSE2
 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m 
-DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM 
-DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
OPENSSLDIR: "/usr/ssl"

And the developer claimed: “According to http://heartbleed.com/. OpenSSL 1.0.1g is NOT vulnerable. Also I have restarted all services on this server.”

So, OpenSSL has been updated and the all services have been restarted, but why does the problem still persist?

I took a look at the command history he ran:

1
2
3
4
5
6
7
8
9
10

wget http://www.openssl.org/source/openssl-1.0.1g.tar.gz
ls
tar xvzf openssl-1.0.1g.tar.gz
cd openssl-1.0.1g/
sudo ./config --prefix=/usr
sudo make
sudo make install
exit
openssl version -a
sudo reboot

The OpenSSL library has been built from the source, which is fine, but the problem is that the Nginx server was still using the old library distributed by Ubuntu:

1
2
3
4

$ ldd `which nginx` | grep ssl
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fafe82a3000)
$ strings /lib/x86_64-linux-gnu/libssl.so.1.0.0 | grep '^OpenSSL '
OpenSSL 1.0.1c 10 May 2012

In effect, there were two versions of OpenSSL library installed in the system, one was built from the source, and another one was managed by dpkg:

1
2
3
4

$ dpkg -l openssl
||/ Name                          Version             Architecture
+++-=============================-===================-===================
ii  openssl                       1.0.1c-4ubuntu8.2   amd64

However, the bigger problem is the version of the operating system:

1
2
3
4
5
6

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 13.04
Release:        13.04
Codename:       raring

Ubuntu 13.04 is not supported anymore according to https://wiki.ubuntu.com/Releases. The developer probably issued apt-get upgrade, but nothing to be updated, because Ubuntu stopped supporting the release. Therefore, no security update. And Ubuntu 13.04 is not listed in Ubuntu Security Notice USN-2165-1. So, the developer opted for building the library from the source. After installation from the source, the binary openssl was overridden by the source build, and the command openssl version showed the latest and patched version 1.0.1g.

To fix the problem, we need to reinstall the package first:

$ sudo apt-get install --reinstall openssl

Now, this will revert control back to apt-get and overwrite the binary /usr/bin/openssl:

1
2
3
4
5
6
7

$ openssl version -a
OpenSSL 1.0.1c 10 May 2012
built on: Wed Jan  8 20:51:55 UTC 2014
platform: debian-amd64
options:  bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx) 
compiler: cc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -m64 -DL_ENDIAN -DTERMIO -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DOPENSSL_NO_TLS1_2_CLIENT -DOPENSSL_MAX_TLS1_2_CIPHER_LENGTH=50 -DMD32_REG_T=int -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
OPENSSLDIR: "/usr/lib/ssl"

And then we must perform the distribution upgrade to the latest long term support version, in order to continue receiving updates.

The lesson I have learned from this is that if you are going the wrong direction, no matter how hard you work, you are not going to make it. Make sure to take the initial investment, and really understand the true cause of the problem before attempting to resolve the issue. And don’t blindly follow the procedure. Understand it first, and adapt to your specific situation. As Abraham Lincoln once said:

“If I have nine hours to chop down a tree, I’d spend the first six sharpening my ax.”