Automatically updating commercial WURFL file

General question about two step analysis, matchers and other general API questions

Automatically updating commercial WURFL file

Postby kamermans » Mon Feb 13, 2012 11:21 am

Edit: For the most up-to-date method of automatically updating the wurfl.xml, please read the support documentation here. The information below may be outdated and is left intact to serve as an archive.

All ScientiaMobile API customers are given access to a direct download URL for obtaining the latest WURFL snapshot. This is either the weekly snapshot (released on Sunday night), or an Out-Of-Band snapshot that we release to our customers between weekly snapshots in case of a significant improvement to the data (for example, if a high-profile device is released mid-week and it is not yet in WURFL). The snapshots are licensed to your organization with your commercial license included.

To create a direct download URL, login to http://www.scientiamobile.com/ and go to My Account in the top-right of the screen. Click on "direct download link" to create a URL for your access. This URL is unique to you and must be kept private.

Checking for WURFL updates automatically
The direct download URL supports the following methods to check for updates:
    HTTP HEAD Request: You can send an HTTP HEAD request to your direct download URL and it will return the size of the current snapshot and the date last modified. Note that the snapshot is not created until the first time you visit this link after the snapshot is released, for example, we release a snapshot on Sunday night and you check it on Tuesday, the last modified date will be Tuesday, not Sunday.
    HTTP If-Modified-Since: If you include the If-Modified-Since HTTP header, the server will only return the snapshot if it is newer than the specified date.
    HTTP If-None-Match / ETag: Our download system sends an HTTP ETag header in along with the snapshot. If this ETag is recorded, you can send it back on subsequent requests in the HTTP If-None-Match header, and the snapshot will only be returned if it is different than yours.

The recommended method is If-Modified-Since. It is well supported among download clients and you don't need to record anything on your side to be able to use it. Note that dates in HTTP headers are in UTC, not your local time!

Wget
Warning: Wget does not properly implement any of the above methods, and only simulates them by requesting the entire file and breaking the connection after the HTTP response headers arrive!
To get the latest WURFL Snapshot with Wget in a UNIX-like environment, use this command, replacing /path/for/wurfl and the download URL with your information. Note that /path/for/wurfl is a directory name without the trailing slash; the WURFL file (wurfl.zip) will be located inside that directory.
Code: Select all
WURFL_DIR=/path/for/wurfl; wget -N -P "$WURFL_DIR" --header="If-Modified-Since: $(date -r $WURFL_DIR/wurfl.zip --utc --rfc-2822 2>/dev/null || date --utc --rfc-2822 --date='1 week ago')" http://www.scientiamobile.com/wurfl/xxxxx/wurfl.zip


curl
The curl program supports most methods above, but there seems to be a bug with the way it generates the If-Modified-Since timestamp, causing it to download the file even though it has not changed.
This is the preferred method to use curl in a UNIX-like environment for downloading the WURFL Snapshot:
Code: Select all
curl -Rz "$(date -r /path/to/wurfl.zip --utc --rfc-2822 2>/dev/null || date --utc --rfc-2822 --date='1 week ago')" -o /path/to/wurfl.zip http://www.scientiamobile.com/wurfl/xxxxx/wurfl.zip


Windows
I'm trying to find a good method for retrieving a file only if it has been updated for Windows, but there's not a simple one-liner that I can find. The closest built-in partial solutions that I can find are by using Bitsadmin with /setcustomheaders or by writing a short PowerShell script. Feel free to post a better solution.

aria2
The aria2 project is a very robust file downloader that can be used to update the WURFL file:
Code: Select all
aria2c.exe --remote-time=true --allow-overwrite=true --conditional-get=true --dir=C:\path\to\wurfl https://data.scientiamobile.com/xxxxx/wurfl.zip

You can download aria2 from http://aria2.sourceforge.net/ .

.NET Framework
Here's a fully-function C# Console Application for downloading the latest WURFL.
Code: Select all
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Net;

namespace WURFLDownloader
{
    class Program
    {
        static void Main(string[] args)
        {
            string remoteUrl = "http://www.scientiamobile.com/wurfl/xxxxxx/wurfl.zip";
            string localFileName = @"C:\temp\wurfl.zip";
            DateTime lastModified = File.Exists(localFileName) ? File.GetLastWriteTime(localFileName) : DateTime.Now.AddDays(-7.0);
            HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(remoteUrl);
            request.Timeout = 10000;
            request.AllowWriteStreamBuffering = false;
            request.UserAgent = "WURFL Downloader/dotNet";
            request.IfModifiedSince = lastModified;
            Console.WriteLine("Downloading WURFL file...");

            try
            {
                HttpWebResponse response = (HttpWebResponse)request.GetResponse();
                Stream streamResponse = response.GetResponseStream();
                FileStream fileStream = new FileStream(localFileName, FileMode.Create);
                byte[] read = new byte[4096];
                int count = streamResponse.Read(read, 0, read.Length);
                while (count > 0)
                {
                    fileStream.Write(read, 0, count);
                    count = streamResponse.Read(read, 0, read.Length);
                }
                fileStream.Close();
                File.SetLastWriteTime(localFileName, response.LastModified);
                streamResponse.Close();
                response.Close();
                Console.WriteLine("Successfully downloaded WURFL to: " + localFileName);
            }
            catch (WebException e)
            {
                if (e.Response != null)
                {
                    if (((HttpWebResponse)e.Response).StatusCode == HttpStatusCode.NotModified)
                        Console.WriteLine("The WURFL File is up to date.");
                    else
                        Console.WriteLine("Unexpected status code = " + ((HttpWebResponse)e.Response).StatusCode);
                }
                else
                    Console.WriteLine("Unexpected Web Exception " + e.Message);
            }
        }
    }
}
Thanks,

Steve Kamerman
ScientiaMobile

Make sure you check out our WURFL Cloud, WURFL InSight and WURFL InFuze products!
kamermans
 
Posts: 392
Joined: Mon Jun 06, 2011 9:50 am

Re: Automatically updating commercial WURFL file

Postby sboiling » Thu Mar 28, 2013 1:22 am

Hi Steve,

Quick question for you please:

How do users that do not hold a commercial licence get access to an updated WURFL file?

Many thanks,
Sean
sboiling
 
Posts: 1
Joined: Thu Mar 28, 2013 1:20 am

Re: Automatically updating commercial WURFL file

Postby sm_support2 » Thu Mar 28, 2013 8:51 am

sboiling wrote:How do users that do not hold a commercial licence get access to an updated WURFL file?


They can use our Cloud solution

http://www.scientiamobile.com/cloud

If you are talking about the wurfl.xml for OnSIte/Standalone API installation, the answer is they are not allowed to. This is a feature we reserve to commercial licensees, because this is ScientiaMobile's business model.

Thanks
sm_support2
 
Posts: 294
Joined: Mon Jun 20, 2011 5:04 pm

Re: Automatically updating commercial WURFL file

Postby alex-white » Mon Jul 22, 2013 7:29 am

kamermans wrote:To create a direct download URL, login to http://www.scientiamobile.com/ and go to My Account in the top-right of the screen. Click on "direct download link" to create a URL for your access. This URL is unique to you and must be kept private.


My company can't find that "direct download link" anywhere on the My Account page. Where should it be?

Thanks
alex-white
 
Posts: 1
Joined: Mon Jul 22, 2013 7:19 am

Re: Automatically updating commercial WURFL file

Postby kamermans » Mon Jul 22, 2013 9:08 am

Hi Alex,

Your sales rep will need to create a license for you in our system first. I will ping them to see what is causing the delay.
Thanks,

Steve Kamerman
ScientiaMobile

Make sure you check out our WURFL Cloud, WURFL InSight and WURFL InFuze products!
kamermans
 
Posts: 392
Joined: Mon Jun 06, 2011 9:50 am

Re: Automatically updating commercial WURFL file

Postby sm_support2 » Mon Jul 22, 2013 10:03 am

Alex, we have uploaded the license to the account you indicated. We are happy to hear that the problem is solved and you can now access your customer vault successfully.

Thank you.
sm_support2
 
Posts: 294
Joined: Mon Jun 20, 2011 5:04 pm

Re: Automatically updating commercial WURFL file

Postby trickydee » Mon Aug 10, 2015 2:22 pm

Hi,
Can you please specify also what are the possible error scenarios (ex. account blocked, password expired or any other like is it possible that the download link gets revoked or something similar?)

Thanks
trickydee
 
Posts: 32
Joined: Fri Jul 17, 2015 8:58 am

Re: Automatically updating commercial WURFL file

Postby scientian » Tue Aug 11, 2015 10:03 am

If there is an issue with your account, you will receive a `402 Payment Required` error. Otherwise, the only other error that you should receive is a `400 Invalid Request` if the file is not supported.

Thank you,

Ian
scientian
 
Posts: 1
Joined: Tue Jan 21, 2014 9:15 pm

Re: Automatically updating commercial WURFL file

Postby sjulianprabu@virtusa.com » Thu Dec 17, 2015 6:26 am

Hi Steve,
We are trying to set-up an automatic update of WURFL DDR using the instructions given above for Onsite Implementation.

There are couple of questions around this as listed below. Could you please clarify them.

1. As I understand, there can be changes for API as well. Will the direct download link (for commercial users) provide API as well. Please provide more details on this.

2. What is your recommendation on "detecting" availability of API updates automatically.



Thanks,
Julian
sjulianprabu@virtusa.com
 
Posts: 2
Joined: Mon Jan 12, 2015 6:50 am

Re: Automatically updating commercial WURFL file

Postby aaronp » Thu Dec 17, 2015 11:22 am

Julian,

To answer your questions,

1) Commercial users will be able to download the API through their 'FileX - File Manager' page (located in the customer vault).

2) A notification will be sent through email for any updates to the API.

If you have any further questions of concerns, please let me know.

Aaron
aaronp
 
Posts: 188
Joined: Wed Dec 09, 2015 12:39 pm


Return to General API Questions

Who is online

Users browsing this forum: No registered users and 1 guest