The World's Simplest Dynamic Image Service

At Mombo Labs, my team has been striving for simplicity. Our mantra is "Do the simplest thing that could possibly work." This approach has us attempting to avoid complexity by refusing to address problems before they arise. This leads to some shockingly simple solutions to very complex problems.

I'll give you an example by building up a dynamic image service addressing one concern at a time. You can see the code up on GitHub. There will also be links to specific tags in the following example, so you can pull the code and try it yourself.

We had a need to serve image files to both iOS and the web. We are using Amazon S3 for file storage, so the simplest thing that could work would be to simply serve up the S3 urls. We could even throw the CloudFront CDN infront of it and call it a day, but that would not be a very dynamic image service.

The images on S3 are full resolution images. We want to get an app in front of the images, so we can have some nice dynamic processing. This will allow us to serve the images more appropriately for the devices and platforms viewing them.

The simplest thing that could fit that bill looks something like this.

var express = require('express'),
    request = require('request'),
    config  = require('./config'),
    app     = express(),
    server;

app.get(/d\/(.+)/, function(req, res) {
    var url = config.get('images') + req.params[0];
    request({ url: url })
    .pipe(res)
});

app.listen(config.get('port'), function(){
    console.log('server started on port ' + config.get('port'));
});

If you take a look at the tag for this code in the repository, you will see a simple express application that handles a single route. The route starts with "/d/" and ends with a relative path to the image we want. The handler adds the relative path to a base url for a public folder on my personal Dropbox account. Using the request module, it makes an http request for the image and pipes the response out to the user.

With a quick "heroku apps:create imageprocess && git push heroku master", I have a publically available app serving up images stored in Dropbox. You can hit it here, give it a try.

Next up, I would like the ability to add query string parameters that change the way the image is served up. For instance, I would like to be able to compress the images at differing levels of compression. To do this I am going to introduce a transform module to my app that will use ImageMagick to do the heavy lifting. Luckily, there is already a great module called gm that wraps ImageMagick in a nice node like interface.

The transform code looks like this.

var gm = require('gm').subClass({ imageMagick: true });

exports = module.exports = function(stream, options) {
    if(options.q) {
        return gm(stream)
                .compress('JPEG')
                .quality(options.q)
                .stream();
    } else {
        return gm(stream).stream();
    }
};

This code takes an image stream and a set of options and passes them into the gm module returning a stream. If the options object has a "q" value we use JPEG compression to compress the file to the value of "q".

The route handler can use this module like so.

var express = require('express'),
    request = require('request'),
    transform = require('./lib/image-transformer'),
    config  = require('./config'),
    app     = express(),
    server;

app.get(/d\/(.+)/, function(req, res) {
    var url = config.get('images') + req.params[0];
    request({ url: url })
        .on('response', function(image_stream) {
            transform(image_stream, req.query)
            .pipe(res);
        });
});

app.listen(config.get('port'), function(){
    console.log('server started on port ' + config.get('port'));
});

This simply takes the response stream of our http request, feeds it to the transform module along with the query string object and pipes the result into the response stream. The complete code for this version is located here. I can now get versions of my images compressed with a quality of "1" to "100".

Next, I would like the ability to fetch the image at different sizes. I can accomplish this by beefing up my transformer like this.

var gm = require('gm').subClass({ imageMagick: true }),
    _ = require('lodash');

var transformMap = {
    // route example: /d/<img id>?w=100&h=100
    w: function(item, options) {
        return item.resize(options.w, options.h);
    },
    // route example: /d/<img id>?c=true&q=50
    q: function(item, options){
        return item
                .compress('JPEG')
                .quality(options.q);

    },
    default: function(item){ return item;}
};

exports = module.exports = function(stream, options) {
    var item = gm(stream);

    _.each(_.keys(options), function(key){
        var transform = transformMap[key] || transformMap['default'];
        item = transform(item, options);
    });

    return item.stream();
};

This refactoring creates a map of query string parameter names to transformations that I can use in the transform function by iterating over the keys of the options object, looking up the transform in the map and finally applying it to the current stream. I also pass along the options object to the call to the transform, so that keys that rely on each other can be handled together. I can now easily get am image that is 100px or 200px wide. I can even combine the trasnforms so I can get a 250px wide image compressed at 50%!

If a parameter doesn't have a transform the default one is used. It simply does nothing to the stream. This way I can add additional transforms easily enough by adding functions to the map. For instance, maybe I don't always want to resize images based on pixel dimensions. Instead I would like to do it by percent.

var gm = require('gm').subClass({ imageMagick: true }),
    _ = require('lodash');

var transformMap = {
    // route example: /d/<img id>?w=100&h=100
    w: function(item, options) {
        return item.resize(options.w, options.h);
    },
    // route example: /d/<img id>?s=50
    s: function(item, options) {
        return item.resize(options.s, null, '%');
    },
    // route example: /d/<img id>?c=true&q=50
    q: function(item, options){
        return item
                .compress('JPEG')
                .quality(options.q);

    },
    default: function(item){ return item;}
};

exports = module.exports = function(stream, options) {
    var item = gm(stream);

    _.each(_.keys(options), function(key){
        var transform = transformMap[key] || transformMap['default'];
        item = transform(item, options);
    });

    return item.stream();
};

No problem, now I can get an image scaled to 10%, 30%, or 75%.

Now that I have my simple dynamic image transform service. I have a problem, the service has no caching at all. Every request, it fetches the image from Dropbox via http request and processes the result. This is not only time consuming but it is also processor intensive. All I need is to be linked by Hacker News and my little service is going to go up in flames.

To solve this issue, I can lean on Amazon CloudFront. I simply created a new distribution using my heroku app as the origin server. The trick here is to make sure you enable "Forwarded Query Strings". This will pass the query string values along to heroku for processing.

What does this do for me? Well now I can hand out urls for the CloudFront CDN versions of my images including the transformation query string. If the CND has the image it serves it up, if not the request is piped through to my heroku image service where the image is generated on the fly and served up.

You can see this behavior in Chrome's Network monitor.

This basically means that the only time my image service has to generate an image is for the first request of that particular transform. In case you haven't noticed it yet, all the transformed images linked to in this post use the CloudFront url. So, the only time you have been actually hitting the service was if you tried playing with the query string values yourself.

The final piece to this puzzle is cache control. I want the images to fall off the CDN after they become stale. I can simply add some header statements to my route handler which CloudFront will respect, like so.

var express = require('express'),
    request = require('request'),
    transform = require('./lib/image-transformer'),
    config  = require('./config'),
    app     = express(),
    server;

app.get(/d\/(.+)/, function(req, res) {
    var url = config.get('images') + req.params[0];
    request({ url: url })
        .on('response', function(image_stream) {
            res.setHeader('Cache-Control', 'public, max-age=3600');
            transform(image_stream, req.query)
            .pipe(res);
        });
});

app.listen(config.get('port'), function(){
    console.log('server started on port ' + config.get('port'));
});