Responsive Images Specification and Real-World Scenarios

Responsive Images Specification and Real-World Scenarios

By Jon Arne Sæterås

The real problem is not whether machines think, but whether men do.

B. F. Skinner

Responsive ImageFinally the long awaited Responsive Images specification (respimg) is making its way into browsers. Briefly summarized, the Responsive Images specification introduces two new attributes to the img element, sizes and srcset, and a brand new <picture> element. The Responsive Images Community Group (RICG) has outlined 9 different use cases that the new method for displaying images on the web should address.

Seen from web developers’ and content editors’ perspectives it makes sense to combine these 9 use cases into 2 scenarios. Jason Grigsby did exactly that in a recent blog post. These are:

  • Resolution switching – When screen size, pixel density or view port require different versions of the image.
  • Art direction – When the contents of the image need to be different to preserve the meaning or readability.

So far so good. This is all great news! However, when reading the specifications and looking at examples of implementation, I wonder if the average web developer, or average web editor using a CMS, will make use of all the flexibility in the respimg specification.  True, if done right, respimg will reduce the bandwidth consumed by images. Also true that you don’t have to implement all the bells and whistles to make a positive impact.

The question is; how many of these bells and whistles will the average web site implement in foreseeable future? Brad Frost has already told us what he thinks.

This issue is critical for how images are handled in the future. Joshua Benton predicts that the size of graphics continue to increase because developers and designers are using and designing for high DPI desktop screens like the recently announced 5K iMacs.

The answer to the question of practical implementation of responsive images lies somewhere in the classical trade-off between cost, time and quality: “How cheap and quick can I make this web page with a reasonable level of responsive qualities?” 

Simple Resolution Switching Example

Below is a basic example of how srcset can be use to address the common and compelling use case for resolution switching. This snippet tells the browser to download and display pic_480.jpg for 1.5 dpr screens, pic_640.jpg for 2 dpr screens and use the pic_320.jpg as a fallback for everyone else. (We made this simple demo if you want to test in your browser)

<img
  src="pic_320.jpg"
  srcset="pic_480.jpg 1.5x, pic_640.jpg 2x,
  width="320" alt=“fixed width”>

Good. You might say this is obvious, but for the above example to work, you need three versions of the image to exist. Someone, or something, must create them. This costs money and takes time.

And who says there won’t be 15 different device pixel ratios to consider? In ScientiaMobile we battle these issues every day. According to our own data, to cover 90% or of the devices you need to provide images in device pixel ratios 1, 1.5, 2. To get to 95%, add 3.

image001

We can of course make it more specific by using the sizes attribute to tell the browser which image it use in certain viewport sizes by applying media queries. This is quite powerful (but verbose) functionality which also handles device pixel ratio.  As you can see, what started out as a simple scenario for applying srcset can quickly balloon your code and require much more work in preparing multiple sizes of images.

Art Direction

Our next scenario is Art Direction: This scenario requires serving images that are not only different pixel sizes, but also images that should be cropped differently or have a, different aspect ratio. Let’s jump to our example to see how an art directed version may look:

<picture>
  <source media="(min-width: 36em)"
    srcset="2art_full_3050.jpg  3050w,
      2art_full_2048.jpg  2048w,
      2art_full_1280.jpg  1280w,
      2art_full_1024.jpg  1024w,
      2art_full_800.jpg  800w,
      2art_full_400.jpg  400w"
    sizes="50vw" />
  <source
    srcset="2art_crop_1860.jpg 1860w,
      2art_crop_1280.jpg 1280w,
      2art_crop_1024.jpg 1024w,
      2art_crop_800.jpg 800w,
      2art_crop_400.jpg 400w"
    sizes="100vw" />
  <img src="2art_crop_320.jpg" alt="art"/>
</picture>

The inner workings of <picture> is well described elsewhere and you can stretch and tune this how far you want. The example above is quite comprehensive and covers high dpi tablets (like Kindel Fire HDX 8.9”) all the way down to a 320px image. It’s up to the browser to pick whatever src it deems appropriate. Of course, the more options we provide, the better the result will be in terms of saved bandwidth and image quality.

In the example above, we now have 12 versions of the same image. Twelve!

This is without offering different image formats – which also is supported! With all this new code and syntax, one might start wondering if the bytes added will zero out the savings from bandwidth by serving optimized sources. With the power of media queries we can make this as complex or easy as we find necessary.

Back to my initial question. Will this powerful functionality be implemented on the average web page? Possibly, but I have doubts. I think front-end developers and web designers are likely to adopt some of the flexibility, but what about those who make content management systems, web developers and, last but not least, the people who actually produce the content? Will they create twelve different versions of an image? My crystal ball says no. Other peoples crystal balls see the same future.

Although I am convinced that some of respimg functionality will be used in the future, I just don’t believe most designers will want to spend the time and money to achieve great quality. Even if WURFL has been roaming the wilderness since the dawn of the “non-desktop web”, I think WURFL and Device Detection will still roam today’s web for the time to come.

WURFL and Server-Side Detection

While recourse to Server-Side detection and optimization is often frowned upon by the current RWD orthodoxy, it still represents an approach that mobile developers have successfully adopted since the first WAP devices launched in 1999. Mobile developers have used WURFL as a component in solutions that resize their pictures server-size. Using WURFL, they achieve great results in three ways that the Responsive Image specification is not delivering:

  • Automatically: no need for manual image editing
  • Dynamically: no need for explicit deployment of new versions of existing pictures
  • Contextually Aware: understanding the capabilities of the device requesting the image and adjusting accordingly

Of course, WURFL provides only one of the components, with the creation of the actual framework left as an exercise for developers, thanks to tools such as GDImageMagickJAI and Image.io.

Recently, ScientiaMobile has decided that server-side image resizing should no longer be left as an exercise for the reader, and we launched a tool in this space: WIT or the WURFL Image Tailor.  WIT Community Edition is offered free of charge to the community of mobile developers, as an alternative to other existing tools (notably http://src.sencha.io) that do not appear to be very reliable anymore.

WIT and Responsive Images

In its simplest form, WIT is straight forward to use. The following code:

<img src="//wit.wurfl.io/http://example.com/image.jpg">

will resize and optimize the image according to a device screen size. The detection of the device and its screen size comes from the latest version of WURFL. This is WIT’s simplest form. There is plenty of extra control that developers have thanks to resizing directives that can piggyback on the query string (WIT documentation has the details).

We can take this a step further and apply WIT to the example we used to illustrate resolution switching above:

<img
 src=“//wit.wurfl.io/w_200/http://example.com/image.jpg"
 srcset="//wit.wurfl.io/w_300/http://example.com/image.jpg 1.5x, //wit.wurfl.io/w_400/http://example.com/image.jpg 2x"
 width=“200" alt=“Me”>

We have already saved ourselves the work of creating (and managing) two image sources. And, no matter how complex we make our srcset, only one single image source is all we need.

Personally, I am very fond of the art-direction scenario. Some claim it is a theoretical case, but I think that is mainly because of the cost of implementing it. This is why we decided to implement a feature for doing “automatic art-direction” in WIT. We call it “smart cropping” since you can’t really automate art-direction.

At the time of writing, this is still an experimental feature and only available internally. Still, the tests we have performed are very promising. We have added some logic that analyses the picture, looking for faces, objects, focal points and so on. Based on this analysis, we apply zooming and cropping to make sure that the important parts for the picture is clearly visible even in lower resolutions. This means that no faces will be cut in half. Also, as much as possible, the meaning of the image will be preserved and even enhanced for smaller screens.

To illustrate, the picture below is resized to be 150px high, which we’ll say is what our design allows for small screens.

image003

However, the man on the image is too small to provide value, and maybe our design would like the image to be 200px wide too. Then lets crop!

image005

Meh…. not quite. Let’s apply smart cropping:

image007

Value added, automatically. (source image)

Smart cropping is enabled by adding the smart crop directive to the request:

<img src="//wit.wurfl.io/m_smartcrop/w_300/http://example.com/image.jpg">

(not enabled on our production servers yet)

You can already now imagine how convenient it is to produce art directed images with WIT smart crop and the <picture> element with little effort but high value.

You may argue that doing automated art direction is dangerous. We can’t trust a machine to fully understand the meaning behind an image. Of course, that is correct. There will always be examples of images being misinterpreted by a machine. However, for most web sites out there, I believe it’s still a feasible option that will, in most cases, give a better result than if no smart cropping was introduced.

Conclusion

The Responsive Images specification has ambitious and worthy goals. HTML authors can provide hints to the client on how to select the most suitable images for their web pages in multiple ways. In fact, resping is great because you no longer need third party JavaScript hacks to determine which version of an image to present by sniffing the viewport size and pixel ratio. The new syntax does this for you.

While the specification is powerful, support for the new features must be built into browsers first, and into CMS platforms, tools and other web-authoring tools after that. Arguably, support in CMS is going to be harder than support in browsers. Managing different image versions and resolutions, supporting art direction and extending the editorial process is the toughest part – one that can hardly be fixed with a software upgrade.

This is where WIT can greatly help web authors already today whether you start implementing respimg or stick to the familiar method. Once again, the server-side and client-side complete each other.

  • Tim Murtaugh

    The false premise of your argument is making my head hurt:

    Will [the people who actually produce the content] create twelve different versions of an image?

    No, they won’t — firstly because 12 is a nutty number you wouldn’t see in the real world.

    Second, and more obviously, content management systems can automatically create those variants. Plenty of systems already do this — users upload one image, and the system generates the required size variants.

    Art direction is absolutely a use case that will require humans to select and upload more than one image; your “smart cropping” will only help in some of those instances. (And is likely to be built directly into content management systems directly soon anyway.)

    • Jon Arne Sæterås

      Tim, thank you for your comment.
      Yes, 12 is a number. Nutty or not. Point is, it is easy to arrive at such numbers when you’re trying to make the most out of respimg. This is also the process the CMS vendors are going through. Likely, they arrive at a different, lower, number. For many reasons. Most importantly CMS vendors have to be very generic in their approach so even if we have all the bits and pieces to implement respimg in CMSs, this does not mean that it is actually used in a optimal way. Art direction or not. This is where the history is repeating it self.

  • Luca Passani

    [I am not the author of this blog post, but I feel that I need to respond to this comment]

    “12 versions of an image” was used in the example to make a point, but even if it was just 3 or 4 versions, that’s still a lot of pictures to edit and manage, particularly because, at the moment, the standard is one.

    About CMS integrating support for the srcset/picture element, I suspect you are showing a little too much faith in those frameworks and their ability to ensure a seamless transition to the new approach. There are questions around: upgrading CMSes, migrating to different systems, managing legacy content, etc. that constiture delicate aspects of managing an IT infrastructure. Whenever large (and not so large) organizations need to address those issues, they are not taken lightly and a lot of planning is required in advance. There are things that a hobbyist can easily do, that become a lot more complex in the enterprise. In short, I would be careful in accusing others of “false premise that make your head hurt” (words are important. Even on the Internet).

    Finally, WIT is being offered to the community of developers at no charge, so I am not sure why all the animosity against an extra option to manage the complexity of multiserving pictures in a complex world of smartphones, tabets and desktop web.