Improved Compression Ratios Using Zopfli

Compressing files can be a very effective way of reducing their size as they are transferred between the server and a user's browser. Most modern browsers work effectively with compressed files and use lossless compression formats such as Gzip, Deflate or Zlib.

In order to compress a file, an encoder is needed to process the file and compress it into the required format. There are a number of different encoders and many of them are very effective at quickly processing and compressing files on-the-fly.

Most of these encoders achieve good file compression ratios, but each of them has their pros and cons. A new compression algorithm, called Zopfli, was created by researchers at Google and delivers a 3 - 8% improvement compared to other encoders set at maximum compression. Whilst this may seem like a small improvement, remember every byte saved makes download speeds just that little bit faster!

Google Fonts actually uses Zopfli to compress all of the fonts that they serve. Using Zopfli, they managed to shrink them by an average of 6%. At first glance this may seem like a small win, but this is huge if you consider the fact that Google’s Web Fonts serve 50 million daily requests, across roughly 800,000 unique websites!

It’s important to note that Zopfli is not a "replacement" for Gzip, Deflate or Zlib. Rather, Zopfli is an improvement to the encoding algorithm for these formats.

You must be thinking - this is amazing, when can I start using it? You can start using Zopfli immediately, but while it does come with improved compression ratios, it does unfortunately come at a cost. In order to achieve such high compression rates, Zopfli encodes at a significantly slower rate when compared to other encoders.

This doesn't make it ideal for on-the-fly encoding for files that frequently change, but there is no reason why you can’t process static files before you deploy them. This means that static files such as JavaScript, CSS, PNG and HTML, can be easily processed and still gain the benefits of the Zopfli compression algorithm.

In this article, I am going to run through a simple example using Grunt, the JavaScript task runner, to compress a file using the Zopfli encoder in a GZip format. If you aren’t familiar with Grunt - I have written a number of articles using this great tool and there are also very detailed guides on the Grunt website to get you started. We are going to use the Grunt Zopfli plugin; the source code is available on Github.

Before we get started, we need to use the Zopfli library binary to process our files. You can either download this compiled version that I have created on Github, or download a copy of the original source and compile it for yourself. Once you have the binary, add it to your project directory - I have created a subfolder for my project and named it zopfli_lib for this example.

Let’s get started by firing up the Node.js command prompt and navigating to our project folder in the console. Next, install a copy of Grunt in your development directory by typing the following into the console.

npm install grunt --save-dev

Install a copy of the Grunt Zopfli plugin by typing the following into the console.

npm install grunt-zopfli --save-dev

The command above will save a copy of the Grunt Zopfli plugin to your development directory. Before we begin compressing our files, we need to create a Gruntfile in order to reference the files that we want to process. My Gruntfile looks a little something like the following:

In the code above, I am pointing to the original version of my file and creating a new compressed copy of the file called processed.js.gz. As is the convention with Gzipped files, the extension should end in a ".gz."

Next, I am adding a reference to the zopfli.exe binary that I placed in the zopfli_lib_ directory. Instead of referencing the full file path, I am using Node’s _dirname value to get my current directory value and joining the directories together.

All that is left to do is run the task against our file. Simply type "grunt" into the console.

Woohoo! You should notice a file called processed.js.gz in your project directory. This file is processed and ready to roll. GZip is a standard format supported by most modern servers and browsers, and in order to use your newly processed file you may need to configure your server accordingly. Apache, IIS and different CDNs may require some initial configuration.

The great thing about this plugin is that once you have automated your build process, any changes will be picked up and compressed accordingly. For further details, there is a great video interview with the Google Web Fonts team about how they used Zopfli to improve their compression ratios. I also recommend having a look at the Google Developers blog for more information. Finally, if you are having trouble getting up and running, I have created a repo on Github with the full source code for this example.