How to analyze 404 HTTP code from weblogs

The dreaded 404 HTTP code means page not found. However multiple 403 and 404 on weblogs also can also mean there are attempts to crack the website.

The awk script down here can be useful tool to analyze weblogs and identify multiple attempts at cracking the web application.

 

awk '($9 ~ /404/)' access.log | awk '{print $7}' | sort | uniq -c | sort -n

The script can also be tweaked for other HTTP status code too.

Solving “Connection is encrypted using obsolete cipher suite” warning from Chrome

Here is a how to on how to solve the dreaded warning “Your connection is encrypted using obsolete cipher suit” from Google Chrome.

Firstly the warning had nothing to do with using cheap or self-signed TLS/SSL security certificate, but it has to do with cipher suite used on the server part.

obsolete-cipher-suite

So if you are a system administrator, you can edit the site config to include a more modern cipher.

NGINX Server

Using nginx, add the line containing “ssl_cipers” to the site config.

# /etc/nginx/sites-enable/example.conf 
server {
 listen 443 ssl;
 root /var/www/example.com/;
 server_name example.com;
   ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA';

        ssl_protocols TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
...
...
...

}

sudo service nginx restart

Apache HTTP Server

For those who are using Apache HTTP server, you can edit the VirtualHost file from “/etc/apache2/sites-enable/” directory.

<VirtualHost *:443>
    ...
    SSLEngine on
    SSLCertificateFile      /path/to/signed_certificate
    SSLCertificateChainFile /path/to/intermediate_certificate
    SSLCertificateKeyFile   /path/to/private/key
    SSLCACertificateFile    /path/to/all_ca_certs

    # Intermediate configuration, tweak to your needs
    SSLProtocol             all -SSLv2 -SSLv3
    SSLCipherSuite          ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA
    SSLHonorCipherOrder     on
    SSLCompression          off

    # OCSP Stapling, only in httpd 2.3.3 and later
    SSLUseStapling          on
    SSLStaplingResponderTimeout 5
    SSLStaplingReturnResponderErrors off
    # On Apache 2.4+, SSLStaplingCache must be set *outside* of the VirtualHost
    SSLStaplingCache        shmcb:/var/run/ocsp(128000)
 
    # Enable this if your want HSTS (recommended)
    # Header add Strict-Transport-Security "max-age=15768000"
 
    ...
</VirtualHost>

You can restart Apache HTTP server by running

sudo service apache2 restart

How to set Android *.apk mime-type for Nginx web server

Here’s a simple guide on how to add the correct mime-type for Android APK file for Nginx webserver.

sudo nano /etc/nginx/mime.types

In “mime.types” file, add this line within the “types” block


types {
     ...
     ...
     application/vnd.android.package-archive     apk;
     ...
     ...
}
     

Restart nginx server

sudo service nginx restart

Done!

Generating TLS/SSL Self Signed Certificate for Nginx in Ubuntu LTS

This post concerns on generating self-signed TLS/SSL certificate for Nginx in Ubuntu LTS and assumes that you’ve configured nginx server with a default site.

Step 1: Generate OpenSSL certificate

sudo mkdir /etc/nginx/ssl
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/nginx.key -out /etc/nginx/ssl/nginx.crt

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:CA
Locality Name (eg, city) []:Palo Alto
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Mypapit LLC
Organizational Unit Name (eg, section) []:Billing
Common Name (e.g. server FQDN or YOUR name) []:Mypapit
Email Address []:mypapit+cert@gmail.com

Step 2: Edit nginx site config

You can edit nginx site config here, replace ‘default’ with your own server config.


sudo nano -c /etc/nginx/sites-enable/default

You will see this server block.


server {
        listen 80;
        listen [::]:80;
        server_name your_domain.com;
        root /var/www/your_domain.com;
        index index.html index.htm;

...
...
}

Add additional line (in italic)

server {
        listen 80;
        listen [::]:80;

    listen 443 ssl;

       server_name your_domain.com;
        root /var/www/your_domain.com;
      index index.html index.htm;

        ssl_certificate /etc/nginx/ssl/nginx.crt;
        ssl_certificate_key /etc/nginx/ssl/nginx.key;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        add_header Strict-Transport-Security max-age=31536000;



...
...
}

Save file, and restart nginx server

sudo nginx -t
sudo service nginx restart

Test configuration by going to https://your_domain.com.

Done!

Bonus: Add HSTS header and Serve only TLS

HSTS header

Install NGINX with PageSpeed using *.deb for Ubuntu LTS (AMD64)

Hello there, I’ve made an easily installable *.deb NGINX package with PageSpeed. The package is made for Ubuntu LTS on AMD64 machine.

Ubuntu 14.04 LTS – nginx 1.8.0 with PageSpeed

  1. nginx-full_1.8.0-1+trusty1-mypapitubuntu4_amd64.deb Full package
  2. nginx-extras_1.8.0-1+trusty1-mypapitubuntu4_amd64.deb Extra package

Ubuntu 14.04 LTS – nginx 1.8.0 with PageSpeed: Other Package

  1. nginx-common_1.8.0-1+trusty1-mypapitubuntu4_all.deb
  2. nginx_1.8.0-1+trusty1-mypapitubuntu4_all.deb
  3. nginx-doc_1.8.0-1+trusty1-mypapitubuntu4_all.deb

Installing nginx-extras or nginx-full is as easy as running this command

sudo dpkg -i nginx-common_1.8.0-1+trusty1-mypapitubuntu4_all.deb
sudo dpkg -i nginx-full_1.8.0-1+trusty1-mypapitubuntu4_amd64.deb
sudo dpkg -i nginx_1.8.0-1+trusty1-mypapitubuntu4_all.deb

Attention : Once installed, the PageSpeed configuration file can be found in “/etc/nginx/conf.d/pagespeed.conf”

Verify Installation
To verify whether nginx with pagespeed has been installed, type

nginx -V

Verify Installation with a preinstalled nginx
If you’ve another version of nginx installed on your system, take note that the nginx-pagespeed from *.deb is installed in “/usr/local/bin”

/usr/local/bin/nginx -V

It will output something like this:

nginx version: nginx/1.8.0
built with OpenSSL 1.0.1f 6 Jan 2014
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_spdy_module --with-http_sub_module --with-http_xslt_module --with-mail --with-mail_ssl_module --add-module=/home/mypapit/source/nginx-1.8.0-1+trusty1/debian/modules/nginx-auth-pam --add-module=/home/mypapit/source/nginx-1.8.0-1+trusty1/debian/modules/nginx-dav-ext-module --add-module=/home/mypapit/source/nginx-1.8.0-1+trusty1/debian/modules/nginx-echo --add-module=/home/mypapit/source/nginx-1.8.0-1+trusty1/debian/modules/nginx-upstream-fair --add-module=/home/mypapit/source/nginx-1.8.0-1+trusty1/debian/modules/ngx_http_substitutions_filter_module --add-module=/home/mypapit/source/nginx-1.8.0-1+trusty1/debian/modules/ngx_pagespeed-release-1.9.32.4-beta

Take note at the bolded text to verify whether pagespeed module has been installed.

Python code: List most popular URL from Apache/NGINX ‘access.log’ file

Found a great Python code snippet for listing the most popular URL from Apache / NGINX ‘access.log’ file. Very practical!

import collections

logfile = open("access.log", "r")

clean_log=[]

for line in logfile:
    try:
        # copy the URLS to an empty list.
        # We get the part between GET and HTTP
        clean_log.append(line[line.index("GET")+4:line.index("HTTP")])
    except:
        pass

counter = collections.Counter(clean_log)

# get the Top 50 most popular URLs
for count in counter.most_common(50):
    print(str(count[1]) + "\t" + str(count[0]))

logfile.close()

The code is very handy if you want to find out the most popular URL or pages in your website, crucial information for optimization, IMHO.