WURFL Microservice for Azure: a Tutorial

You either have heard about WURFL and Device Detection before OR you have not. In either case, read on.

Device Detection is the art of squeezing the most information possible out of HTTP requests generated by browsers on tablets, smartphones, SmartTVs, and the watches of people that surf the web from their wrist. 

Born in 2001, WURFL is the oldest and arguably the most well known Device Detection framework. While it started off as an open source-project, WURFL went commercial in 2011 as ScientiaMobile’s flagship product.

In essence, WURFL allows programs to map HTTP requests to the properties of the browser and devices that generated them. In the beginning of time (which for mobile developers is somewhere around 1998 and 2003) the primary use case for Device Detection was UX optimization. There was no way an HTML website could be scaled down to work on mobile devices available in those years. The arrival of Responsive Web Design and smartphones has made that use case less relevant in the last 10 years or so, but other use cases have become more relevant. Among these are Analytics and Ad Tech, i.e. determining which ads are most suited for a user’s device.

Another use case has come to my attention a couple of years ago as an offshoot of the wider Unified Logging (Kafka, Kinesis, Azure Stream Analytics, …). I am talking about Event Stream analysis, i.e. the ability to extract useful information out of a stream of data that’s too much to store in one place and analyze off-line. This use case has interesting applications in different industries, for example for the purpose of detecting failures and defects in products and services (a good book that explains Event Streams is Event Streams in Action, published by Manning).

Enter WURFL Microservice

WURFL Microservice is the latest incarnation of the WURFL API. From the perspective of the final result, WURFL Microservice is an API that one can deploy locally just like other traditional WURFL APIs.

{
  "Accept-Language": "hu-HU,hu;q=0.8,en-US;q=0.6,en;q=0.4", 
  "Accept-Encoding": "gzip, deflate, lzma, sdch", 
  "Accept": "*/*", 
  "User-Agent": "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko)
                 Chrome/49.0.2623.112 Safari/537.36 OPR/36.0.2130.80",
  "Video-id": "TPhZnruRPsM", 
  "Event": "VIDEO_OK"
}

{
  "Accept-Language": "zh-TW,zh;q=0.9,en-US;q=0.8,en;q=0.7", 
  "Accept-Encoding": "gzip, deflate", 
  "User-Agent": "Mozilla/5.0 (Linux; Android 8.0.0; HTC_U-1u Build/OPR1.170623.032; wv)
                 AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/78.0.3904.108
                 Mobile Safari/537.36 JsSdk/2 TopBuzz/12.5.6 NetType/4G", 
  "Accept": "*/*", 
  "Event": "VIDEO_FAILURE",
  "Video-id": "TPhZnruRPsM"
}

Event-processor would read those events from the standard input and augment them with the following data:

{
  "Accept-Language": "hu-HU,hu;q=0.8,en-US;q=0.6,en;q=0.4", 
  "Accept-Encoding": "gzip, deflate, lzma, sdch", 
  "Wurfl-Form-Factor": "Desktop", 
  "Accept": "*/*", 
  "User-Agent": "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 \
          (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 \
           OPR/36.0.2130.80", 
  "Video-id": "TPhZnruRPsM", 
  "Wurfl-Device-Model": "Opera", 
  "Wurfl-Device-Make": "Opera Software", 
  "Wurfl-Device-OS": "Windows XP", 
  "Event": "VIDEO_OK"
}
{
  "Wurfl-Device-OS": "Android 8.0.0", 
  "Accept-Language": "zh-TW,zh;q=0.9,en-US;q=0.8,en;q=0.7", 
  "Accept-Encoding": "gzip, deflate", 
  "Wurfl-Complete-Name": "HTC U Ultra", 
  "Wurfl-Form-Factor": "Smartphone", 
  "User-Agent": "Mozilla/5.0 (Linux; Android 8.0.0; HTC_U-1u \
                 Build/OPR1.170623.032; wv) AppleWebKit/537.36 \
                 (KHTML, like Gecko) Version/4.0 Chrome/78.0.3904.108 \
                 Mobile Safari/537.36 JsSdk/2 TopBuzz/12.5.6 NetType/4G", 
  "Accept": "*/*", 
  "Event": "VIDEO_FAILURE", 
  "Wurfl-Device-Model": "U Ultra", 
  "Wurfl-Device-Make": "HTC", 
  "Video-id": "TPhZnruRPsM"
}

In this tutorial, we are going to illustrate the following steps:

• Obtain and deploy a WURFL Microservice VM from the Azure Marketplace
• Launch the WURFL Microservice VM instance
• Build and launch the event-sender and event-processor applications
• Illustrate use of WURFL Microservice client API in event-processor

Prerequisites

• .NET Core framework 2.2
• Microsoft Azure account

Obtain the WURFL Microservice Virtual Machine from Azure

Visit the Azure Marketplace and search for WURFL in the search text box. 

Choose the WURFL Microservice version that fits your needs. You can pick Basic, Standard or Professional. For the purposes of this tutorial, you will need either the Standard or the Professional edition.

On the creation page click on the blue “Create” button.

We are now ready to create our first Azure VM instance. To do that you have to fill in some forms. on the “Change size” link. To launch the WURFL Microservice VM, you first need to pick an instance type.

For the purpose of this tutorial, you can go for the B1ms type (more information on Azure instance type sizes can be found here).

Choose your preferred disk type and size.

Security group configuration opens ports 80 and 22 by default. Port 80 is used to talk HTTP to the WURFL Microservice HTTP server, while port 22 allows SSH access to the VM instance.

If for some reason, port 80 is not open by default, you will need to add it by clicking on the “Create new” link of “Configure security group” and “Inbound security rules” of Networking menu item.

It’s now time to review our instance by clicking on the “Review + Create” tab.

Review your configuration and, if everything looks correct, click on “Create”. The deployment of the new virtual machine starts.

This is the screen that you’ll see while the deploy goes on.

When the deploy completes a blue button “Go to resource” will appear. Click on it to access the instance details.

Copy the IP address that you see in the “Public IP address” text field, open a browser and visit the http://<value_of_public_ip_field>/v2/status/json URL address. You should see a small JSON object that vouches for the health of your installation.

Alternatively, you can use curl from the command line: assuming your IP is 40.71.197.253, if your instance responds with something like this, you are golden. Your WURFL Microservice server is up and running!

$ curl  http://40.71.197.253/v2/status/json
{
    "lookup_request": 0,
    "lookup_useragent": 0,
    "lookup_device_id": 0,
    "make_model_requests": 0,
    "server_info_requests": 8,
    "v1_capabilities_requests": 0,
    "not_found_404": 0,
    "server_uptime": 1172
}

Build and launch of the event-sender application and event-processor application

Obtain the demo application from this GitHub Repo. It’s a .NET Core console application that you can build and launch with dotnet command: 

$ cd event-processor
$ dotnet build
$ cd ../event-sender
$ dotnet build

The .Net applications can also be built in Visual Studio. Once the applications are built, you’ll want to open a terminal window and launch both applications, piping the sender output to the processor input and setting the --host flag to the IP of your running WURFL Microservice server instance. 

Assuming the IP is, for example, 40.71.197.253, the command will be:

$ cd bin/netcoreapp2.2/
$ dotnet event-sender.dll | dotnet ../../../event-processor/bin/netcoreapp2.2/event-processor.dll \
    --host 40.71.197.253

If the command is launched from a different directory, the dotnet event-sender.dll command will need the --path <path to event_stream.json> flag i.e. the absolute path to the event_stream.json file in your local repo )

Note: the event-sender application will simply read a local file of HTTP requests in JSON format and “replay” them on the standard output to simulate a stream of events. Not a lot of magic there. 

The event-processor will process the event stream and release data to the standard output. The data is the same JSON data it received in input augmented with WURFL device information:

{
  "Accept-Language": "zh-TW,zh;q\u003d0.9,en-US;q\u003d0.8,en;q\u003d0.7",
  "Accept-Encoding": "gzip, deflate",
  "Accept": "*/*",
  "User-Agent": "Mozilla/5.0 (Linux; Android 9; ASUS_X01AD Build/WW_Phone-201911231308; wv)
                 AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/79.0.3945.93
                 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/251.0.0.31.111;]",
  "Event": "VIDEO_OK",
  "Wurfl-Complete-Name": "Asus X01AD (Zenfone Max M2)",
  "Wurfl-Device-OS": "Android 9.0",
  "Wurfl-Form-Factor": "Smartphone",
  "Wurfl-Device-Make": "Asus",
  "Wurfl-Device-Model": "X01AD",
  "timestamp": 6922396444999
}

WURFL Microservice client API usage in event-processor

As mentioned, the code in event-sender application does not need much explanation. The event-processor code is more interesting to us: it uses what is, effectively, a fully-fledged WURFL API by communicating with the WURFL Microservice server through HTTP (such communication is virtually transparent to the API user, but knowing what is going on under the hood cannot hurt).

Note: ScientiaMobile doesn’t usually refer to cloud-based products as a WURFL API to avoid creating confusion with its “classic” local API products. We prefer to call those Client APIs (which implies that there’s a server somewhere, which is in fact the case). Having said this, from a developer perspective, a Client API and a WURFL API are very similar, almost indistinguishable products: both products will feel as a local API and fulfill the basic function of mapping HTTP requests to a list of device properties (AKA WURFL capabilities).

The .NET Visual Studio project references the WURFL_Microservice_Client.Core Nuget package, which is downloaded with the other dependencies during the dotnet build command execution.

In Processor.cs, we import the client API classes we need:

using Wmclient;

The WURFL Microservice client class is created as soon as the event processor starts:

WmClient wmClient = null;
Try
{
     wmClient = WmClient.Create("http", host, "8080", "");
     wmClient.SetCacheSize(20000);
}
catch (WmException e)
{
// Exception message and exit from app...   
}

The host variable value is the one set by the command line switch in the launch command. 

In our code, we set the cache size to 20,000 elements, but you may want to go for a larger number (say 100,000 or more) in production systems. Keep in mind that, once device information is expunged from cache, retrieving it again will cause an HTTP round-trip to our WURFL Microservice Server, and this operation is several magnitudes slower than in-memory operations.

Once we have our WURFL data, enriching the input JSON data with WURFL device data is easy peasy.

JSONDeviceData device = wmClient.LookupRequest(CreateRequest(eventData));
eventData.WurflCompleteName = device.Capabilities["complete_device_name"];
eventData.WurflDeviceMake = device.Capabilities["brand_name"];
eventData.WurflDeviceModel = device.Capabilities["model_name"];
eventData.WurflFormFactor = device.Capabilities["form_factor"];
eventData.WurflDeviceOS = device.Capabilities["device_os"] + " " \
         + device.Capabilities["device_os_version"];
Console.WriteLine(JsonConvert.SerializeObject(eventData, Formatting.Indented, settings));

Time for some DataViz

Now that we have our enriched JSON data, we might just as well use it for some data visualization. For example, we might want to look deeper into those failed (VIDEO_FAILURE) events.

Let’s capture the streamed data into enriched_event_stream.json and use some basic data visualization tool to make the data speak and tell us what is making those video events fail.

First, let’s prepare the headers of a TSV file that will contain three columns: 

1. OS and Version, 
2. Event Status and
3. Complete name of the client.
   

$ echo -e "OS and Version\tStatus\tComplete Name" > os_breakdown.tsv

If you are reading this tutorial, you may have heard of a tool called jq. If you have not, check it out. It’s pretty cool. It’s a bit like sed and awk together (for those who are familiar with UNIX/Linux command-line tools), but specifically for JSON files. 

The following command will read our JSON, pick the fields we are interested in out of each enriched JSON object and print it in TSV format:

$ jq -r '[."Wurfl-Device-OS", .Event, ."Wurfl-Complete-Name"] | \
          @tsv' enriched_event_stream.json >> os_breakdown.tsv

$ head os_breakdown.tsv
OS and Version  Status Complete Name
Android         VIDEO_OK Opera Mini 5
Android 4.2.2   VIDEO_OK Generic Android 4.0
Android 9       VIDEO_OK Samsung SM-N9600 (Galaxy Note9)
iOS 13.3        VIDEO_OK Apple iPhone
Windows 10      VIDEO_OK Opera Software Opera
Android 9       VIDEO_OK Huawei COL-L29 (Honor 10)
iOS 13.3        VIDEO_OK Apple iPhone 7 Plus
Android 6.0.1   VIDEO_OK Xiaomi Redmi Note 3
Android 5.0.2   VIDEO_OK Generic Android 5.0

$ wc -l os_breakdown.tsv
3001 os_breakdown.tsv

Loading the TSV in a spreadsheet allows a fair amount of analysis and visualization. Let’s first import our data into Google Sheet:

Let’s group all “OS and Version” fields with a pivot table in search of correlations. 

Data > Pivot table and the selection of the data we care about will bring us to this:

This is already sort of revealing. Android 8.0.0 sticks out as a sore thumb. Let’s visualize with one of the many charts offered by Google Sheet (Insert > Chart).

Yes, that’s exactly what we suspected. Devices running Android 8.0.0 don’t seem to stomach this particular bit of video content much. The Petflix engineering team can be informed of where they can start looking to address the issue:

Of course, while we have used a spreadsheet to explore our data manually, all analytics wizards out there will have already recognized the possibilities provided by the several Snowplow, Elastic, Splunk, Tableau, Talend, Snowflake, Zoomdata (Logi Analytics),Databricks, Azure Data Explorer, Amazon Redshift, Yellowbrick and the plethora of  business intelligence tools, ETL tools, data marts, log analytics tools out there. Covering those would fall outside of the scope of this document completely.

Shutdown of the VM instance

There is nothing more to shutting down the WURFL Microservice instance than shutting down any deployed Azure VM.

Simply open the VM instance dashboard, select your WURFL Microservice instance, right-click on it and click on the Stop button.

Note: Be aware that stopping/restarting the WURFL Microservice instance will clear the Client API cache. Depending on the use you are making of the API  this may result in diminished performance as the API “warms up” and the cache is recreated at every VM launch.

Conclusions

This tutorial on WURFL Microservice for Azure has hinged on a demo inspired by the Event Streams use case. We have seen how the WURFL VMs available on the Azure Marketplace can be used to deploy a WURFL API in minutes as part of an existing relationship with Microsoft Azure (i.e. without licensing one of the classical on-prem WURFL APIs from ScientiaMobile).

This use case can easily be extended to other programming languages, other cloud vendors and other data analysis platforms (the AWS and Java versuion of this tutorial is available).

We may amend this tutorial from time to time to point to those resources.

Further Resources

• Demo source code on GitHub.
• This tutorial available for AWS and Java.
• For those who do not wish to rely on the AWS Marketplace, licensing WURFL Microservice directly from ScientiaMobile is an option. The WM Server is deployed through Docker in that case (private ScientiaMobile repo).