News

Visit ScientiaMobile at the 2016 Mobile World Congress Barcelona Conference!

Come find us! We will be located in Hall 8.1, Stand C13.
We would love to meet with you and discuss your mobile device detection needs.
Learn more about Mobile World Congress, click here

mwc_2016

Please enter your information below, and we will contact you to schedule a meeting.

Map8_1_C13

Using Client Hints and Image Optimization

Responsive images have been around long enough for most of us to have taken them for a spin, or at least to have learned from the experiences of those who have. Beyond doubt, the responsive images specification is a great win for the web. However, quite a few reports from the front lines suggest that responsive images can become pretty ugly. The good news is that there is a fix! No, not throwing JavaScript at the challenge, but by asking the web server for a helping hand. Enter Client Hints, an initiative spearheaded by Google that is already available in browsers (Chrome and Opera) and that is super-simple to use. Let’s see how Client Hints can reduce both image size and verbosity of the responsive images markup.

This article will not demonstrate the challenges of responsive images. Several other sources have already done that. Instead, we’ll focus on how to address the issues, with a little help from the web server and a new way for the browser, or client, to request images with specific properties. Even if this is called “Client Hints”, it is pretty specific. Let’s dive in!

What Are Client Hints?

Client Hints is a new feature already available with Chrome 46 and Opera 33. More browser vendors are following. It is an initiative by Google (spearheaded by Ilya Grigorik), and, as its progress indicates, it is likely to be “a thing”. The initiative was also recently adopted by the HTTP Working Group.

You can think of Client Hints as the missing link between the browser and the server when it comes to layout information. Instead of specifying every possible image breakpoint, pixel density and format in your responsive images markup, Client Hints append the current scenario to the HTTP request, allowing the web server to pick the perfect fit — also known as content negotiation.

The current way of dealing with responsive images is typically to specify different image sources based on the receiving device’s pixel density, preferred image format and viewport size. If you go through the process of picking breakpoints and formats, your markup might end up something like this:

Brad Frost: I never want to make anything that even remotely resembles this.

Even relatively simple use cases can become quite complex using the responsive images syntax. The code shown here is from A List Apart. (Source: Brad Frost)

I think most of us agree with Brad on this.

To be fair, the example above include different crops of the image too. Which is something Client Hints alone can’t help you with. Still, crafting markup like this is not something a developer or designer should be spending their time on. We need an automated process. Even if the server can generate dynamic markup autimatically, client hints decouples the markup from the image which makes it easier to perform operations on the image without having to worry too much about the markup.

Let The Web Server Know

Imagine if the web server knew the pixel density, the viewport size and also the actual size and format of an image. This is what Client Hints does! Browsers that support Client Hints add a few HTTP headers to requests. The latest draft mentions these hints:

  • DPR
    This stands for “device pixel ratio,” the ratio of physical pixels on the screen to CSS pixels.
  • Viewport-Width
    This is the width of the viewport in CSS pixels. (CSS pixels means the units used in CSS to describe a layout. A CSS width of 100 pixels would be 200 device pixels (DP) if the device pixel ratio (DPR) was 2.)
  • Width
    This is the actual width of an image in real physical pixels (similar to the w descriptor made famous by responsive images).
  • Downlink
    This is the client’s maximum download speed.
  • Save-Data
    This boolean indicates whether extra measures should be taken to reduce the payload.

Downlink and Save-Data are not available in Chrome just yet, but you can imagine their purpose. Let’s focus on the currently available hints. First, we have to tell the browser to send the hints.

Enabling Client Hints In Your HTML

Client Hints flow

With a meta tag added, Client Hints are enabled and additional HTTP headers are sent with the request. (View large version)

You’ll have to opt in to enable Client Hints. The reason for this is that additional data shouldn’t be added to a request unless it is used for something. If it’s not used, it would work against the very purpose by adding data to the payload. Servers can also advertise their support, but for now you’ll have to explicitly opt in by adding the following meta tag to the <head> element in your markup:

<meta http-equiv="Accept-CH" content="DPR,Width,Viewport-Width"> 

That’s all we need. The browser will now append the DPR, Width and Viewport-Width headers to all subsequent requests generated from the HTML, including CSS, JavaScript and so on (with exception of Width, which, in practice, applies only to images). Aside from images, Client Hints might be useful if the CSS file contains breakpoints based on the viewport size or device pixel ratio. Knowing the viewport size before the CSS is returned to the browser, the server can strip unqualified blocks from the CSS file before sending the response. That is another story. For now, let’s look at an example with images.

Our Old Friend, <img>

Imagine we have a page with the image tag below. Let’s display the flower.jpg image at a width of 200 CSS pixels.

<img src="flower.jpg" width="200"> 

With Client Hints enabled, the browser will send the following request to the server:

http headers

With just the meta tag added, the browser appends DPR and Viewport-Width to the request. (View large version)

The browser “hints” to the server that the requesting device has a pixel ratio of 2. As a bonus, we get the size of the viewport, too, since this is known by the browser at the time when the request is being made. Because the DPR is 2 and our page’s design need this image to be 200 pixels wide, we would like to serve an image that is 400 actual pixels wide (200 pixels × 2). But we’re missing information about the intended display size, even if the img tag says width="200". The specification explains that the sizes attribute has to be set in order for the Width header to be sent. In the near future, the width attribute on the image element is likely to be included in the algorithm too, but for now, we’ll have to stick with sizes. The sizes attribute describes the layout and display size of the image. In the beginning, it might be less intimidating to think of it like the good old width attribute or like a CSS property:

<img src="flower.jpg” sizes=“200px"> 

Using pixels might make it easier to “retrofit” existing markup, but using relative units such as vw is recommended to make the page more “responsive”:

<img src="flower.jpg” sizes=“50vw"> 

Now, the request for flowers.jpg will look like this:

HTTP headers

With sizes attribute added in addition to the meta tag, Viewport-Width is added to the request. (View large version)

The browser calculates the intended display size of the image based on the current size of the viewport and the device pixel ratio of the device in time for the preloader to fetch it. You might recognize this from how the selection process work with “regular” markup for responsive images with multiple image sources listed. The difference here is that the information is appended to the HTTP request, rather than an image source being picked from the srcset attribute.

Mind. Blown.

What just happened? Basically, we boiled our verbose responsive images tag down to something that looks very familiar, but with the same responsive functionality. With the pixel ratio and width information, the server can now pick, or generate, the appropriate size of the requested image.

  • The DPR header takes care of resolution switching. We don’t need srcset with x descriptors in our markup.
  • The Width header tells the server the exact width of the image (relative to the viewport) that the browser need to fit in the layout.

Actually, we don’t need srcset at all. The sizes attribute is our new hero! Based on the relative value of sizes, the browser translates this into the actual size in physical pixels. Also, remember that media conditions can be applied if you want the image to have different sizes as the width of the viewport changes:

<img src="flower.jpg” sizes="(min-width: 30em) 100vw, 50vw"> 

The Width header will, of course, reflect this, too. For a deeper dive, Jason Grigsby has written a great introduction to sizes over on Cloud Four

But what about type? Responsive images allows you to define different formats, or mime types, of an image by using the type attribute: type="image/webp"

Client Hints does not cover this, but its older brother, the Accept header, may contain valuable information.

Accept: image/webp,image/*,*/*;q=0.8
 

This example is from Chrome, which is nice enough to tell us that WebP is preferred. Other browsers might just say */* which basically means, “Send me anything.” In those cases, you could apply your own rules, or, even better, the server could implement more advanced device intelligence solutions to decide the best image format to return to the client.

The Server Side of Client Hints

We could say that, with Client Hints, we’re moving the responsibility of selecting an image source from the browser to the server. This means, of course, that we need to implement some logic to act on the Client Hints server-side.

The bonus of involving the server is that, rather than just selecting the best match from a list of pre-generated image files like the browser does, the server can generate the perfect fit on the fly! On a small scale, this is quite achievable because we have all the information we need in the HTTP header.

However, if this task is a bit daunting and if performance is a priority, a few web services — image proxies, if you will — already support Client Hints. One of them is the free ImageEngine. Using ImageEngine, for example, we first have to “prefix” our images with the URL of the service.

If your image src’s URL is http://example.com/image.jpg, then we’d have to change the src to http://[key].lite.imgeng.in/http://example.com/image.jpg. [key] is the personal token you get when you sign up. As long as the meta tag is present and that sizes attribute is present in the image tags, we’re good to go. Looking at the response with cURL, we can see how the server responds:

$ curl -I http://try.imgeng.in/http://web.wurfl.io/assets/sunsetbeach.jpg -H "DPR: 2" -H "Width: 150" -H "Viewport-Width: 800"

HTTP/1.1 200 OK Content-Type: image/jpeg Vary: Width Content-DPR: 2 … 

The request has a DPR of 2, a Width of 150 pixels and a Viewport-Width of 800. The server then responds with the Content-DPR header, whose purpose is to confirm to the browser what the pixel ratio is of the returned image, so that the browser can fit it on the page correctly. In the example above, the Content-DPR will always be the same as the DPR request header, because ImageEngine is scaling the inputted image to the exact value of Width. As a bonus, even if Width is not set, ImageEngine will fall back to Viewport-Width and then to screen size data from WURFL, a database of devices. If you implement the server yourself, and you chose to mimic browser behavior by selecting the closest match from a set of pre-generated image sources, then the Content-DPR header might be a different number than the DPR hint from the client. The browser will make use of Content-DPR to scale the image to its display dimensions.

Also worth noting is the Vary header. The purpose of this header is to tell the client (the browser or proxy) that the response from this URI will vary according to the value of the Width header. This enables web proxies and content delivery networks to cache the image better — at least compared to caching based on the User-Agent.

Non-Supporting Browsers

Before you run off and start implementing support for Client Hints, you should know that not all browsers support them. The last image tag above will likely mess up a page viewed in a browser that does not support Client Hints. So, what are our options?

Maybe a JavaScript polyfill? In this case, we’d have to rely on cookies, not separate HTTP headers. Cookies would create problems for content delivery networks and cache proxies because the cookie value must be included in the cache key, which would likely lead to cache pollution. Further, and more importantly, the browser’s preloader would not have any knowledge of the cookie’s value.

Until support for Client Hints has reached critical mass, the safest approach is to combine hints with explicit – but relative – widths and heights to make sure the layout doesn’t break. If the browser doesn’t send Client Hints, but the image server expects it, you need the layout to handle an oversized image if that is the default behaviour of the image server. Moreover, to reduce the risk of serving too big an image to a non-supporting browser, an image optimization service is recommended, as described above. ImageEngine is particularly useful because it handles mobile devices very well. If a mobile device does not support Client Hints, then ImageEngine will never serve an image wider than that particular device’s screen.

Performance

Aside from automation, the motivation for implementing Client Hints is, of course, image performance. It’s difficult to compose a fair test case, but to give an idea of how a Client Hints-based image request performs differently than a “regular” responsive images request, I’ve put together a little demo with the two scenarios. Below is a table showing the bytes transferred and the actual size of the image put on the wire at different viewports. The selected image breakpoints and viewport sizes in the demo are arbitrary.

Responsive images w/srcset Client Hints with server scaling
Viewport width Bytes Actual width (pixels) Bytes Actual width (pixels)
320 39.6 480 16.1 288
480 39.6 480 28.6 432
800 81.7 768 63 720
1100 138 1024 113 990
1400 138 1024 186 1260

While the preselected image breakpoints span portions of the viewport’s size, Client Hints, along with an image server capable of resizing images, addresses the continuum of viewport sizes. With Client Hints, we have surgical precision. On average, the Client Hints approach serves 19% less data. If we exclude the 1400-pixel viewport, to which the responsive images approach serves too small an image (1024 pixels), then the data served is 32% less with Client Hints, which is substantial.

waterfall

Full test result can be viewed on WebPagetest (View large version)

The data sample in the chart above is too small to draw any conclusions, but it illustrates the purpose of Client Hints well. No surprises there. Worth noting, though, is the use of DNS prefetching for the external host name try.imgeng.in. Referring to the table above, the time it takes to download the bytes is as expected. Client Hints works as advertised.

(Almost) Ready For Prime Time

While responsive images (image element with srcset and sizes) implemented in markup only allows the browser to pick the closest match from a list of image resources, Client Hints enable the web server to serve an image tailored to the browser’s needs, which in most cases mean less data and better image quality. It does require some programming if you’re implementing server-side support by yourself; luckily, some content delivery networks and web services already support it.

The future of Client Hints looks promising. Adding hints about the user’s connection and instructions to reduce data traffic are on the way. With this information, the server can increase image compression, avoid serving high-density images or reduce the byte size of images in other ways.

As developers, we have no reason not to start exploring Client Hints right now. The key benefits are less verbose and more maintainable responsive image tags, fewer image bytes transferred and, ultimately, happier end users. Only a few steps are needed from you:

  1. add the meta tag to the head element,
  2. put the sizes attribute in your image tags.

Then make, or pick, your image server. I’ve mentioned the free ImageEngine optimization service above, which supports Client Hints. The best way to find other services that support Client Hints is to search Google, because this is a new thing and more providers are announcing support as we speak.

If you want to implement support for Client Hints yourself, the article “Efficient Image Resizing With ImageMagick” is a great start. Also, the open-source imaging service Thumbor is also considering Client Hints.

(al)

ImageEngine WordPress Plugin

ImageEngine wordpress plugin

Our recent release of the image optimization service ImageEngine has been well received by the web community. The results are amazing. It is not uncommon to see image byte size reduced by 50% and more. Even if ImageEngine already is quite easy to implement, we’ve just made it even easier. For WordPress users, that is. We just launched a WordPress plugin which will automatically optimize your images on your WordPress site. The WP ImageEngine plugin can be downloaded from the WordPress plugin repository, or simply search for “ImageEngine” from the plugin page in your WordPress dashboard.

Features of the ImageEngine WordPress Plugin

ImageEngine alone, has some great features. Including automatic conversion of images to the best image format (WebP in most cases), HTTP/2 support and support for Client Hints and more. These features are of course also available through the WordPress Plugin. The plugin will also automatically enable Client Hints for your images, so no editing of themes or templates is necessary. The only think you have to do is to register for an ImageEngine account. You can choose the full version or the Lite version which comes with limited, but spacious, monthly use.

Supports WordPress 4.4 and Responsive Images

Introduced in WordPress 4.4, the way images are handled has changed. Now, a subset of the Responsive Images is implemented. I say “a subset” because is not a part of the implementation. Nor is type specification. Also, the different sizes provided by WordPress, can be considered sort of “random” at its defaults, and not a perfect fit for your theme probably. Although you can customize this, the ImageEngine plugin help you serve the optimal image according to the device and browsers capabilities.

ImageEngine in WordPress 4.4 looks like this:

    <img class="size-full wp-image-6" 
    src="//try.lite.imgeng.in/w_4000/h_3000/http://example.com/wp-content/uploads/2015/12/GOPR3747.jpg" alt="DCIM103GOPRO" width="4000" height="3000" 
    srcset="//try.lite.imgeng.in/w_300/http://example.com/wp-content/uploads/2015/12/GOPR3747.jpg 300w, 
            //try.lite.imgeng.in/w_768/http://example.com/wp-content/uploads/2015/12/GOPR3747.jpg 768w, 
            //try.lite.imgeng.in/w_1024/http://example.com/wp-content/uploads/2015/12/GOPR3747.jpg 1024w, 
            //try.lite.imgeng.in/w_1200/http://example.com/wp-content/uploads/2015/12/GOPR3747.jpg 1200w" 
    sizes="(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px">

Of course the ImageEngine plugin will work (…maybe even better) with the good old image tag too :

<img class="alignright wp-image-435 size-full" 
src="//try.imgeng.in/w_200/h_200/http://example.com/wp-content/uploads/2015/11/img.jpg" 
alt="small" width="200" height="200">

See the Savings

When you have the account, you can log on your account and actually see how much data you save your end users by using ImageEngine. Remember that every byte counts. Globally the average speed you get on mobile devices is 2Mbps. Saving 1 MB image data, will then save your users of 4-5 seconds of download time. Moreover, it is money saved (depending on the plan) for your user, and the page will render faster and use less battery! If you haven’t tried out ImageEngine yet, enter the url of your site in the test tool below, and see the potential optimization:

PHP API

When you install the plugin, it will automatically optimize all images in posts or pages created in WordPress. If you have images in custom templates or in your theme, there is also a PHP API you can use to display images:
This <?php wpie_get_converted_url( $url_to_convert, $options );?> will output //try.lite.imgeng.in/http://example.com/wordpress44/wp-content/uploads/2015/12/GOPR3747.jpg where the parameters are:

  • $url_to_convert is the full url, including http(s)://, to the image.
  • $options is an array of configuration options. If not set, values from plugin admin page will be used. The array looks like this array("key"=>"xyz","key_type"=>"normal","w"=>0,"h"=>0,"pc"=>0,"fit"=>"")

Configuration Options

ImageEngine_Options_‹_44_—_WordPress

Like most other plugins, WP ImageEngine also has a configuration page. You find it under the settings menu on the WordPress dashboard. The most important thing to do there, is to register for an ImageEngine account to get your unique token. This token is the first part in your hostname to the ImageEngine service. All your images will be prefixed with this.

There is also a switch you can use when you’re developing on a local machine, like localhost, so that images are actually displayed. This is because the local image sources WordPress generate is not accessible from the internet.

The advanced options are in most cases better off left at their defaults. But you can tweak the default width, height (fixed and relative) and the fit method.

So, nothing to wait for if you’re a WordPress user. Install the plugin.

 

Mobile Device Usage Info Available at MOVR Visualization Now!

On November 6, we released our MOVR report for 2015 Q3, providing timely information on mobile device usage around the world. At the same time, we introduced our MOVR Visualization tool.

MOVR-Vizualization Mobile Device Usage

For users of MOVR data, you will find this interactive tool very helpful and informative. You can:

  • Select a timeframe: We provide weekly updates of MOVR mobile device usage data
  • Select a region: You can lookat the global trends of mobile device usage or drill down on a continent
  • Select a manufacturer: Select from the world’s top 15 device manufacturers
  • Select a form factor: Narrow down to Smartphones, Tablets, or Feature phones
  • Select a chart type: Trend, Starburst, TreeMap, WorldMap
  • Save to image: Download a high-quality PNG of the chart you have generated
  • Share via URL: Copy the URFL of the chart. That way you can share it easily or dynamically insert it into your our report.
  • Download information: Combine the raw information with other data sets for even more insights into mobile device usage.

To get full access to MOVR Visualization, you need to register for a free ScientiaMobile account. Current WURFL or ImageEngine customers are already registered.

Optimize Images with ImageEngine

A while back we launched a free image optimizing service over at wurfl.io. This turned out to be a huge success, and we’re now serving billions of images every month, saving the end user kilobytes of data per image!

We’re not only happy because of the great numbers, but also because this proves that the web industry is increasingly concerned of performance and optimization. Another reason may be that the Responsive Images specification didn’t turn out to be the silver bullet after all.

We got some feedback on our blog post where we wrote about Responsive Images and the need for server side logic to make it work properly saying the examples we used were “artificial” and that the scenarios were just not realistic. As the time has passed however, more blog posts out there has concluded similarly. Jason Grigsby of Cloud4, stating it very clearly that the image logistics is work is a task for the server.

Further, Ilya Grigorik of Google takes it a step further with spear-heading Chromes implementation of Client Hints. Google’s work with Client Hints is indeed confirming that services like ImageEngine has a very important role to play in the web performance space.

Client Hints support

Speaking of Client Hints. ImageEngine has full support for Client Hints. This means that we’re  now able to offer lazier precision even on sites using responsive images on desktop browsers! For everything else, we have WURFL which does the same job, but for mobile devices.

HTTP/2 support

Another great thing that we have launched is full HTTP/2 support! Which is surprisingly well supported in both mobile and desktop browsers! The end user will notice this by faster image download because the browser does not need to connect to the server for every single download. Potentially saving hundreds of milliseconds.

webp Support

Yet another feature which we implemented because the browser support is broad, is support for webp, Googles own light weight image format. We only serve webp if the browser supports it, of course. This will also save another 25-34% of image data compared to jpeg.

Launching ImageEngine

So, to summarize, the popularity of the free “WURFL Image Tailor” (WIT) service has been overwhelming and timely. That’s the reason why we have spent a lot of time lately to make it even better with support for the most important new features that will greatly impact performance on your web site. In the process we also renamed the service. Now, it is called ImageEngine. We still offer the free edition, now called ImageEngine Lite. From a free user’s perspective the biggest difference is that you now have to sign up for an account. This is because we need to identify who is responsible for the traffic in case something happens.

We’ve also done a few tests of course. The below image shows the difference between three scenarios:

  1. A site with basic, old fashioned <img> tags
  2. Responsive images implementation, including picturefill.js
  3. <img> tags with ImageEngine.

 

The diagram speaks for itself. Now, you can try out ImageEngine for free!


Growing Device Intelligence From The Cloud

Since our inception back in 2011, we had a vision for device intelligence from the Cloud. Early on, we launched our WURFL Cloud product. It offers a client (which we have recently open-sourced – see here) that provides easy access to an always-updated WURFL DDR (see video). Part of having a reliable and fast Cloud service meant having hosted operations around the Globe. We currently operate in several POPs. And the service has grown very steadily.

In 2014, we launched our WURFL.js service. Using one snippet of JavaScript, web developers dip into our Cloud-based DDR and get back basic device intelligence information. To say that this service has been a success is a big-understatement. We now receive over 1 billion requests per month!

And there is more to come!!

We are constantly looking at ways to improve reliability and performance for our commercial customers. We have our own special requirements and have grown to a scale where it makes sense to invest in our own infrastructure at data-centers. Luckily, with our headquarters in Reston, VA, we have access to a premiere supply of world-class data centers to choose from. In fact, Ashburn, VA is known as “Data Center Alley“. We now leverage hosting infrastructure at a specialized facility known for one of the world’s most redundant, distributed, and scalable power delivery architectures (they call it “2N+2” with a 100% Availability SLA), at RagingWire Data Centers. Here are some cool pictures of our racks.

How to Test Offline With WURFL.js

By Dino Esposito
Dino-Saviour-Quotev2-copy

WURFL is a framework that provides device detection capabilities to web sites written in a variety of languages and platforms. WURFL is based on a server-side engine that is available both through a cloud service and as an on-premise configuration. WURFL.js is simply the JavaScript frontend that makes the backend engine available in HTML browser pages. We will take a closer look at WURFL.js and show you some techniques that allow you to test if you are offline and unable to access the WURFL cloud service.

WURFL.js is the name of the HTTP endpoint you invoke from your client pages to have the server-side WURFL engine take a look at the HTTP user agent of the requesting browser. At the end of the analysis, the HTTP endpoint serializes a nice JavaScript object to the output stream. In the context of the browser, this serialized JSON becomes a JavaScript object that carries valuable information about the current device and browser.

WURFL.js in Action

To use WURFL.js you need only one single line of JavaScript that references the HTTP endpoint of the WURFL server. Here’s an example:

<script type='text/javascript' src="//wurfl.io/wurfl.js"></script>

This code injects a JavaScript object into the current DOM that has three properties: is_mobile, complete_device_name and form_factor. Full details about the values and roles of the properties can be found here.

WURFL.js comes in two flavors. There is a free community edition with the behavior described in the link. In addition, there is WURFL.js Business Edition that offers a JSON object with over 20 properties as well as SLA, SSL support and a bit of extra caching for performance reasons. For more information have a look at the product page.

WURFL.js can be used in many different scenarios. The WURFL.io home page outlines a few examples such as personalization, enhancing analytics and optimizing advertising. Further, if you are a front-end developer and implementing device detection on the server side is not an option for you, WURFL.js is your savior.

For example, determining the URL to some images on the fly to ensure that smaller, or just reworked, images are downloaded on some devices. In this case, you can go with code like this:

<script>
   if (WURFL.form_factor == "smartphone") {
       $("#myImage").attr("src", "http://wurfl.io/assets/small_smartphone_pic.jpg");
   }
</script>

Similarly, you can use the WURFL object to redirect to a specific mobile site if the requesting device looks like a smartphone:

 

<script>
   if (WURFL.form_factor == "smartphone") {
       window.location.href = "http://touch.example.com/";
   }
</script>

So far so good.

Where’s the JavaScript Actually?

In spite of the name, and the fact that WURFL.js actually injects a JavaScript object in the client page, there’s no JavaScript file you can install on your host environment. By the time the effects of WURFL.js are actually visible, the magic has already been performed. The magic happens on the WURFL Cloud and uses the browser’s HTTP header as input to drive the device analysis.

Hey, wait a moment! WURFL.js is an online service? And how can I keep on developing my web site when I’m offline? Not an issue at all, just read on.

Test Using a Local and Fallback WURFL.js File

The key point to consider is how to be notified of when the internet connection is unavailable. In HTML, you can use the error event for this. In general, the error event is triggered whenever something goes wrong during the download of an external file such as a script file or an image. The error event is widely supported across browsers, but its use seems to be a bit controversial and potentially problematic in some cases. But let me illustrate the solution first and leave issues for later. The code below contains the highlights of the solution.

<script type='text/javascript' src="/Content/jquery-1.9.1.min.js"></script>

<script type='text/javascript' src="/Content/WurflJs.Wrapper.js"></script>
<script type='text/javascript' 
        src="//wurfl.io/wurfl.js" 
        onerror="WurflWrapper.setOffline()"></script>

<script type='text/javascript' >
    $(document).ready(function() {
        $("#title").html(WURFL.form_factor);
    });
</script>
<h2 id="title"></h2>

The really relevant parts are those highlighted on a black background. The jQuery link is only functional to the code that displays the form factor. The script block that displays the form factor, and the H2 element, is only the core of such a simple demo. The interesting part is the link to the WURFL.js endpoint and the new script file: wurfl.wrapper.js.

The onerror attribute on the script element that links to WURFL.js points to code that handles any problem during the download. If there’s no connectivity the script bound to the onerror attribute kicks in and takes control of the code. The Wurfl.Wrapper.js file is a local file you keep on your local machine during development. Here’s its content:

// Full source code of Wurfl.Wrapper.js
var WurflWrapper = WurflWrapper || {};
WurflWrapper._isOnline = true;
WurflWrapper.isOnline = function() {
    return WurflWrapper._isOnline;
}
WurflWrapper.setOffline = function () {
    WurflWrapper._isOnline = false;
    WURFL = { is_mobile: 'false', form_factor: 'offline', complete_device_name: 'N/A' };
}

The WurflWrapper object has two functions named isOnline and setOffline. Quite simply, the former function tells whether the WURFL object has been downloaded from the WURFL cloud. The setOffline method sets the content of the WURFL object to any content hard-coded in the wrapper file. The combined effect of the above script and WURFL.js is that you get the real user agent information if connectivity works and some hard-coded information if connectivity doesn’t work. Figure 1 shows the effect of the script.

WURFL_Test_Offline

Figure 1. Two screenshots of the same page with and without connectivity

 

The page remains the same and behaves as usual except that if the WURFL.js endpoint doesn’t respond you get a debug version of the WURFL object instead of a null exception.

Possible Issues with the Error Event

The onerror attribute is a bit controversial and sometimes using it may be problematic. There are two main reasons for it to be controversial.

The first is that it is not validated as XHTML by http://validator.w3.org. If you validate it against HTML5, though, it passes. If W3C markup validation is important for your web site then consider that validation doesn’t pass if you use the onerror attribute, but it does if you bind to the error event instead. You can use addEventListener or jQuery for that.

The second aspect about onerror is that—generally speaking—it can become the carrier of some hijacking attacks. If some hacker manages to inject code as simple as a IMG element that points to a missing image, the presence of onerror can run some external script within the host DOM.

 

<img src="missing-file.jpg" onerror="some-script()" />

Both issues are worth calling out, but frankly, they are quite unlikely to impact you up to  the point of preventing the use of onerror on a web site. The XHTML aspect can be fixed by simply using script code to register an error event handler. As for the hijacking point, it is a real issue if there’s a way for external viewers to type HTML code into the DOM, as it may happen with comments on a blog post or an article. Furthermore, in this particular case the use of the onerror attribute (or the error DOM event) is even more largely acceptable as it can be easily turned off before moving to production.

Further Improvements

Using the WURFL wrapper is an easy two-step procedure. First, you add the wurfl-wrapper.js or write an ad hoc version of it. Second, you add the onerror attribute to the SCRIPT element that links WURFL.js. Looking for improvements, a nice feature to have could be making the offline WURFL object return valid information such as a default user agent configuration (i.e., generic web browser, smartphone). Yet another aspect to improve could be implementing some basic device detection on client—basically just looking for some mobile or tablet keywords with a regex—to help making the default offline configuration to be as close to the real as possible.

There’s really no reason for not trying out WURFL.js online or offline today. Go to http://wurfl.io to read about the documentation and get started.

Device Detection with WURFL.js and jQuery

There is no need to explain how awesome jQuery is. It is the Go-To place for developers looking for tools. However, when it comes to browser and device detection, jQuery comes up short. jQuery.browser was deprecated, and finally removed in version 1.9. So, jQuery doesn’t provide functionality to detect browser or device.

Instead, the documentation recommends to try feature detection. Fair enough. However, we often get questions about whether WURFL and jQuery can play together. A quick Google search confirms that this is a relevant topic. Mostly, the suggested solutions are based on hard-to-maintain patterns, such as performing regular expressions on the User-Agent and looking for an ever increasing number of substrings in the User-Agent. We know from experience that these types of hacks fail over time, and is one of the key reasons people turn to WURFL. Wouldn’t it be wonderful if a free tool like WURFL.js could work with jQuery?

If we set up WURFL.js and jQuery side-by-side, without any integration, then it may be handy to have the device information in the jQuery object and let jQuery be in charge.

Load jQuery

Step one is to load jQuery form your favorite source:

<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js"></script>

Load WURFL.js

Then, we need to load WURFL.js. There are many ways to do this. For now, let’s stick with the getScript() function built into jQuery. This function will download the WURFL.js script asynchronously and execute it. You can also use $.ajax() if you want more control or to tweak some properties of the request. The actual WURFL.js file has to be downloaded. You can’t copy the contents of the file and inline the code. This comes with the cost of one additional HTTP request. However, this is necessary because most of the device detection happens on the WURFL.io server. This is the reason why a WURFL.js-based solution is much easier to maintain: WURFL.io handles the user-agent mess, and lets you concentrate on making a great web experience.

Put WURFL in jQuery

The code would look something like this:

  $.getScript("//wurfl.io/wurfl.js", function() {
    $.wurfl = WURFL;
  });

Now, you’re able to access WURFL capabilities like this:

console.log($.wurfl.is_mobile);
console.log($.wurfl.complete_device_name);
console.log($.wurfl.form_factor);

These are the capabilities that are available for free from WURFL.js Community Edition. If you need more capabilities, then WURFL.js Business Edition can help you with that:

console.log($.wurfl.physical_screen_width);
console.log($.wurfl.resolution_width);
console.log($.wurfl.pointing_method);

Finalize

The next step would be to add some simple error handling.

$.getScript("//wurfl.io/wurfl.js")
  .done(function() {
    $.wurfl = WURFL;
  })
  .fail(function() {
    $.wurfl = {};
  });

Voila! That’s really all it is to it. For your convenience, we’ve made a simple CodePen demo.

Are You Ready For Google’s Mobile-Friendliness Rankings? Lessons From The Top 10,000 Sites

  “Mobilize or die”

– Ian Carrington (Google)

April 21, 2015 is the date when Google starts updating their search algorithms to consider “mobile friendliness” in the ranking of sites. This means that site owners who have not implemented a strategy for adapting their site and content for mobile devices will be ranked lower than sites that are optimized. This change is radical and will have impact on all mobile searches, in all languages, globally.

Google goes as far as recommending site owners to use Responsive Web Design (RWD). In addition to RWD, Google itself implements both Adaptive Delivery and dedicated URLs for mobile (mDot) on their most important web properties. (See our Legend below for further definitions of these terms)

For sites developers, the paths to “mobile-friendly” are many and not straight forward. This, along with Guy Podjarnys previous work, triggered us to analyze what the most popular web sites globally are doing to cater to the diversity of devices, browsers and their capabilities. Is Responsive Web Design, mDot site or Adaptive Delivery the right way to go? (Or as Google would name them; Responsive Web Design, Dynamic Serving and Separate URLs)

Based on the top 10.000 URLs, we set up an automated test facility to analyze what happens when a mobile device hit each or these URLs. If you compare our results with Guy Podjarnys experiment, please note the previous sentence, as one of the differences. Our tests see the web from the mobile user’s perspective. For example, if a mobile user is redirected to an mDot site, then the desktop site may still be RWD. This RWD site is not included in our numbers.

Are the Big Boys Ready?

Based on our research, among the top 100 sites, 86% are mobile friendly. But looking at the longer tail, many of the top 10,000 web sites are still not ready. Only 72% of the top 10,000 sites have implemented some sort of “mobile-friendly” approach. This means an alarming 28% of sites appear to do nothing special! These sites would appear like a “zoomed out web page”, typically with a 980px wide viewport (or whatever is default on the specific mobile browser) and with text and links too tiny to make sense.

mftopsites

Server Side Approaches to Mobile-Friendly Are Popular

We used a methodology (see below) to categorize sites by their primary “mobile friendly” techniques. We categorized mobile-friendly sites into one of three buckets: RWD, mDot, and Adaptive Delivery. We recognize that it is not fair to put a site into one category alone, so we also tracked the elements of responsiveness we found in the mDot and Adaptive Delivery sites, so we will drill into responsiveness too. For the purpose if our mobile friendly approaches chart, the techniques are mutually exclusive and we try to identify the “main” technique. For example, if a site is under an mDot domain, but is also using RWD, it is put in the mDot bucket. Further, an mDot site is not considered Adaptive even if the server response is adapted to the specific device.

A surprising finding is that despite all the fanfare around how client-side alone (RWD) is the best approach to solving mobile issues, most of the top sites opt for server-side approaches too. Among the top 10,000 sites, 63% used a server-side approach that includes device detection – either mDot or Adaptive Delivery. As we move to this top 100, this grows to 88%.

ddrtopsites

Clearly, the top sites are driven by practical, real-world solutions which combine approaches. This combination of server-side and client-side approaches is usually referred to as RESS: “Responsive Design with Server Side Components”. We dig into this more later.

Breaking down the top 10,000’s mobile friendly sites, we find that a separate mDot site is the most popular approach at 40%, just ahead of RWD at 38%, and Adaptive delivery at 22%.

In contrast, Adaptive Delivery is by far the preferred approach (52%) among the top 100 players such as Google, Yahoo, Baidu, Bing or Instagram. Conversely, only 12% of the top 100 rely solely on RWD. A separate mDot hostname for mobile devices stays consistent around upper 30 percent across both the top 100 and the long tail.

74% of Top 10,000 Sites Use Responsive Web Design

While our research shows that leading sites use server-side device detection for its power and effectiveness, let there be no confusion: Responsive Web Design is the leading approach for mobile friendly sites. In fact, 74% of the top 10,000 sites use some form of RWD.

However, the choice to use RWD is not a binary or mutually exclusive choice. Many leading sites are combining RWD with server-side detection, an approach called RESS. We found that 49% of responsive sites combine some kind of server side adaptiveness. For example, among mDot sites, 57% carry traces of RWD. Similarly, around 61% of the Adaptive sites carry traces of RWD.

Responsive Approach Among Mobile-Friendly Top 10,000 Sites
Approach RWD Adaptive mDot Total
Responsive 37.5% 13.6% 23.0% 74.0%
Not Responsive 8.7% 17.3% 26.0%
Total 37.5% 22.2% 40.3% 100.0%

Client-side Adaptiveness

Google calls this approach “JavaScript-adaptive”. From an SEO perspective, this approach can be a bit tricky since Google bots now also require access to JavaScript resources to index the site properly. If Google bots can’t access these assets, the site may be deemed non-mobile friendly and receive lower ranking. Of the total pool of 10,000 sites a bit more than 50% of the sites are using JavaScript to modify the DOM for mobile devices. This is about the same for both tiers. It is quite common to think of this approach as a natural part of RWD, or even Progressive Enhancement (lines are blurred). 62% of the RWD sites in our test also use JavaScript to modify the DOM for mobile devices.

Conclusion

With the April 21st deadline approaching, the top 10,000 sites are moving toward mobile friendliness. For 72% of popular sites, mobile browsing is nothing new to them. These mobile-friendly sites spend enormous resources in this area, and there is a lot to learn from their techniques. The most interesting lesson from our research is that a site owner does not have to choose between RWD and other approaches in order to optimize for mobile. In fact, RWD should be the basis of all web development. Adapting and optimizing content and/or presentation on the server in combination with RWD is actually the preferred approach in our sample.

Test Setup

For the test system we used Selenium to automate our tests. We analyzed each URL both with a desktop (Chrome) browser and a mobile browser (emulating iPhone 5S using Chrome developer tools).

The list of URLs we downloaded from HTTP Archive.

Test Definitions

The goal of this research was to find out what happens when a mobile device hits the URL and which technique is used to handle mobile. This is how we define each category

Responsive Web Design

For a site to qualify as a responsively designed site, these criteria must be fulfilled:

  • Use of Media Queries.
  • There must be no horizontal scrollbar (we give the site 15px slack)
  • The viewport is defined in a meta tag.

Adaptive Delivery

For a site to qualify as server side adaptive the site must

  • Serve different markup to mobile devices compared to desktop device from a single URL. This we defined as more than 10% difference in bytes and different count of structural elements, such as divs. (We exclude mDot sites as those are different by definition)

mDot sites

An mDot site is

  • A site with a subdomain or path stating m, mob*, iphone ,ipad or touch
  • The site may utilize Responsive Web Design or not.

Client Side Adaptive

A site qualifies as client side adaptive when

  • The markup as sent from the server is the same for mobile and desktop devices
  • and the final markup (DOM) rendered is different on mobile and desktop. This we defined as more than 10% difference in bytes and different count of structural elements, such as divs. The site may utilize Responsive Web Design or not.

Disclaimer

As mentioned above, it is difficult and unfair to put a site into one category. It is equally difficult to define these categories so that they are testable programmatically. In our data set there are probably sites put into the wrong bucket. We’ve done some probing and fixed a few issues, but still no guarantee. Further, the sites are not tested with a real mobile device, but by using the emulation facilities in Google Chrome developer tools. Still, we believe that the error margin is within the limits. Also worth noticing is that some of the URLs in the list are invalid. Such as http://www.googleadservices.com/. We’ve done our best to cater for this. And of course, during the test, sites may fail to load or only partly complete the tests. These are left out of our statistics.

WURFL.js Business Edition: At the Intersection of JavaScript and Enterprise

By Luca Passani, CTO @ScientiaMobile

“The Constitution only guarantees the American people the right to pursue happiness. You have to catch it yourself.” ― Benjamin Franklin

 

wbe_logo It was over one year ago that ScientiaMobile announced the WURFL.js Community Edition under the umbrella of the WURFL.io initiative. Believe me. WURFL.js was (and still is) a huge success, just as its siblings,the MOVR report and WIT, the WURFL Image Tailor, are.

For those who are just tuning in now, WURFL.js Community Edition is a free developer tool that enables front-end web programmers to leverage the power of server-side device detection through “JavaScript as they know it”.  Simply embed the following tag into your HTML:

<script type='text/javascript' src="http://wurfljs.com/wurfl.js"></script>

and ScientiaMobile’s Cloud will return an object that looks something like this for all practical purposes:

var WURFL = {
    "complete_device_name":"Nokia Lumia 920",
    "is_mobile":true,
    "form_factor":"Smartphone"
};

It goes without saying that the JSON data reports WURFL data, which would be impossible to derive with a JavaScript only implementation (sorry, front-enders!). The tool has been adopted by thousands of websites across the globe. This is not surprising given the new possibilities and the ease of adoption of the new tool. Front-end optimization is one obvious use of WURFL.js, but not the only one. Many use client-side WURFL to improve their analytics and understand, for example, if their responsive website is performing appropriately on mobile devices.

It did not take long before some of the companies out there asked for a commercial version of the WURFL.js, i.e. a version that offers extra features and fewer of the legal restrictions that came with the free service. This is what convinced us to fly Jon Arne, our VP of Innovation, to Washington DC in the middle of a Virginia winter storm for more brainstorming with Steve and myself.  The question in front of us was “What premium features do businesses need that we can deliver via the simplicity of the WURFL.js concept?” The answer to that question was WURFL.js Business Edition, and specifically all of the new features which I’m about to illustrate here.

Commercial Usage

One of the points that users raised most frequently about WURFL.js Community Edition was the fact that the tool could not be used for commercial initiatives, such as services that require fee-based subscriptions. We never thought of this requirement as particularly troublesome. Just to provide an example, Google offers access to Google Maps with the same constraint, yet people are still using those maps pretty much everywhere. Also, it would be risky to offer a free service to companies that may decide to rely on it for their business model.s

Since there seem to be companies that would be perfectly happy to pay a fee to overcome the issue, WURFL.js Business Edition effectively removes that constraint for paying customers. This comes on top of another key advantage over WCE, a dedicated infrastructure.

Separate Infrastructure

While similar from a software perspective, WBE runs on a separate infrastructure. This means that commercial users will not be sharing the same servers with anyone who may decide to adopt the Community Edition from one minute to the next. They will only be sharing the infrastructure with other commercial customers. This is very concretely reflected in the fact that WURFL.js Business Edition is invoked through a different URL to the JavaScript goodies (the actual URL is customizable for enterprises. More about this later).

<script type="text/javascript" src="//wjs.wurflcloud.com/wurfl.js"> </script>

The invocation will only work when embedded in a page that the customer has enabled through their WBE Control Panel (see picture).

Manage hostnames in WBE

Extra Capabilities

Of course, giving commercial users access to more WURFL capabilities was the obvious thing to do. The new capabilities and integration with analytics platforms, such as Google Analytics, will enable analysis of web site performance along more dimensions than previously possible. For example,the capabilities that describe a device’s physical dimension will provide an understanding of how form factors affect user behavior (e.g. by resulting in lower/higher conversion rate).

 

Custom SSL and CNAME Support

Community Edition already supports SSL through a shared certificate. WURFL.js Business Edition brings this further thanks to support for CNAMING and custom SSL certificates, i.e. the ability to point a customer’s DNS to wjs.wurflcloud.com and refer to the service as if it were run in-house by the customer themselves. Along with the ability to upload one’s own SSL certificate for that server, nesting a reference to WURFL.js Business Edition in the following form becomes possible:

<script type="text/javascript" src="https://wjs.mycompany.com/wurfl.js"> </script>

or simply:

<script type="text/javascript" src="//wjs.mycompany.com/wurfl.js"> </script>

to support SSL and regular HTTP requests irrespective of the protocol of the hosting page. CNAMEs have to be declared in one’s own DNS primarily. In case SSL support is also required, adding the CNAME to the WURFL.js Business Edition Control Panel gives the ability to set up SSL Certificates and Private Keys (see screenshots below):

CNAME

Define SSL Certificate in the WBE Control Panel

Improved Performance

WURFL.js Business Edition users are served with a version of the WURFL.js JSON object that can be cached more aggressively by browsers. This will result in fewer HTTP requests for those who visit the sites of enterprise customers. The caching feature makes WURFL.js Business Edition perfectly equivalent to other externally hosted JavaScript libraries such as jQuery or Google Analytics when it comes to availability and performance, with the added value that the JSON object is a lot smaller than those libraries.

 

Service Level Agreement

ScientiaMobile is not committing to any particular SLA with the users of the free Community Edition. WURFL.js Business Edition is different. We commit to 99.99% uptime and offer a SLA that is in line with what the industry offers for similar Cloud services.

Enterprise-Grade Support

WURFL.js Business Edition is a commercial product, and it comes with the support that ScientiaMobile provides to all of its products through its Forum and through direct ticketed support. This is another key element for enterprises.

Conclusions

The market has spoken. Adoption of WURFL.js Community Edition by thousands of websites around the planet shows that WURFL.js fulfills an important need. The launch of the Business Edition brings the tool to the enterprise level. Those WURFL.js Community Edition adopters who replaced their ‘hacky’ JavaScript device detection with WURFL.js, have gained access to better analytics and deeper insight into how users use their services. This great support in the web design process has conquered the trust of front-end developers and paved the way for the introduction of WURFL.js Business Edition.

  • Categories

  • Recent Posts

  • Tags

  • Monthly Archives