How to convert character encoding in text files

Here is how to convert text files from one character encoding to another in GNU/Linux:

#eg1
iconv -f ASCII -t UTF-8//IGNORE file.txt -o output.txt

#eg 2
iconv -f ISO-8859-1 -t UTF-8//TRANSLIT file.txt output.txt

The -f parameter denotes “from” and -t parameter denotes “to” character set.
//IGNORE means the “iconv” will ignore any characters that are not available in the target character set.

While “//TRANSLIT” means the converter will attempt to substitute characters that are not available in the target character set to the closest characters available, failing that, “???” will be replaced in its place.

Most GNU/Linux distribution have iconv preinstalled, if not, please consult your distribution documentation.

Check if your web server supports Brotli Compression

Brotli is a new compressed data format developed by Google for compressing web data. It is documented in RFC7932. Currently, almost all modern web browser support Brotli which compressed better and faster than Deflate.

Brotli is can be enabled in most popular web server including:

  • Apache HTTPD – through mod_brotli (for release after 2.4.26)
  • Nginx – ngx_brotli (provided by Google)
  • Node.js (trough shrink-ray module
  • LightSpeed (since version 5.2)
  • Microsoft IIS (through IIS-brotli extension, for IIS 7.5 and above)

Testing for Brotli Support

KeyCDN.com has provided a tool for testing whether your website supports Brotli compression.

You can go over the website and get your server tested. For nginx webserver, ngx_brotli will automatically downgrade to gzip if the browser does not support brotli encoding

 

How to Change nginx server signature from source code

This post is rather a note to myself just in case the next time I want to recompile a new nginx server (I recently compiled a new nginx release in order to enable support for the new brotli encoding and http2 protocol)

The exact file that you need to change is :
/nginx-1.13.7/src/http/ngx_http_header_filter_module.c

Here’s a snippet of what you need to change:

static u_char ngx_http_server_string[] = "Server: yourservername" CRLF;
static u_char ngx_http_server_full_string[] = "Server: yourservername/1.0" CRLF;
static u_char ngx_http_server_build_string[] = "Server: " NGINX_VER_BUILD CRLF;

replace NGINX_VER and NGINX_VER_FULL with your own string.

IMPORTANT: Do not redefine NGINX_VER constant!

Do not redefine NGINX_VER constants as it would be used in various installation scripts after compilation. For example, the Let’s Encrypt “certbot” tool is dependent on the factory setting of NGINX_VER constant.

BONUS: Change default error page

You can further confuse various network scanners by changing the nginx built-in default error page. Although you can change 4xx and 5xx error page easily in server configuration. Nmap is known to detect nginx installation by looking into the nginx built-in error page.

In order to prevent nmap from further detecting your webserver version and configuration, you can change the default built-in error page using through this file:

/nginx-1.13.7/src/ngx_http_special_response.c

Look for these lines:

static u_char ngx_http_error_full_tail[] =
"<p>&nbsp;</p><hr><center>Copyright &copy; 2018 Mohammad Hafiz bin Ismail (mypapit at gmail.com )" CRLF
"<br /><small><a href=\"https://blog.mypapit.net\">Mypapit Personal Blog</a></small></center>" CRLF
"</body>" CRLF
"</html>" CRLF
;

 

And change the HTML tags accordingly to suit your need, note that you can remove NGINX_VER_BUILD entirely to hide your NGINX version.

You can also customized the built in HTTP code special response, from this :

static char ngx_http_error_502_page[] =
"<html>" CRLF
"<head><title>502 Bad Gateway</title></head>" CRLF
"<body bgcolor=\"white\">" CRLF
"<center><h1>502 Bad Gateway</h1></center>" CRLF
;

to include the “Viewport” meta-tag in order to support mobile devices:

static char ngx_http_error_502_page[] =
"<html>" CRLF
"<head>" CRLF
"<link href=\"https://fonts.googleapis.com/css?family=Lato|Slabo+27px\" rel=\"stylesheet\" />" CRLF
"<meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">" CRLF
"<title>502 Bad Gateway</title></head>" CRLF
"<body>" CRLF
"<h1>502 Bad Gateway</h1>" CRLF
"<p>Somebody just fucked up at our end :(</p>" CRLF
;

Just make sure you test the nginx configuration after compiling before deploying it in production environment.

Further Reading: Compiling nginx

A rather complete nginx compiling guide can be found from these websites:

  1. How to Compile Nginx From Source on Ubuntu 16.04
  2. Install Nginx from source code on Ubuntu 14.04 LTS