Device Detection is technology that identifies what type of mobile devices are accessing an organization's website. That organization can run device detection in real time, or run it as post-processing data analysis. By using device detection, these companies can deliver improved mobile web user experiences to end users, target advertising, improve analytics of web access data, and accelerate images load times.
In this website, we illustrate the details of Device Detection, the problems it solves and how companies can move from theory to practice with WURFL, the leading Device Detection solution employed by major Internet Companies such as Facebook, Google, Amazon, Akamai and many others.
The Problems Solved by Device Detection
Online users access content with a variety of browsers and devices. When the web was born almost 30 years ago, the idea was that any user could access any web page with any browser.
Yet nobody in the 90's, including those in the academic and research world, could foresee that people would browse the web on tablet, smartphones, smart TVs or even their wrist watches. Nor could they see a world of apps coming.
When you visit a website with your phone, you expect to find the same information that you would find while browsing from your desktop computer. Today's online content generally meets that expectation, but this does not come for free. Behind content that works on all web clients, there is the effort of engineers who strive to make that content adapt to a great variation of devices and browsers. This is the area where Device Detection comes into play. Knowing which devices your users have in their hands is key to tailoring the content to the capabilities of their devices (web optimization), serving targeted ads in the right formats (Ad Tech use-cases) and deciding which services to launch in the future based on "device demographics".
Basics of Device Detection
Web browsers request online content with a mechanism called HTTP (HyperText Transfer Protocol).
Note: An HTTP request is essentially a computer-friendly way for a computer to request a page (be it a static document or a dynamically generated bit of content) available on a different computer, i.e. the web server.
HTTP requests almost invariably come with a field (a 'header' in HTTP speech) called user-agent. The user-agent field is supposed to communicate which web browser is requesting content. But this is where things got a bit messy already in the 1990s. While the user-agent header effectively allocated a slot for this key bit of information, what to write in that slot was left as a free-for-all exercise. The result was confusion right from the start (i.e. from when just a handful of desktop web browsers existed). Such confusion only grew worse with the advent of mobile devices, as the need for recognizing the features of browsers and devices became the most pressing for content providers across the globe.
This is the context in which an engineer named Luca Passani started WURFL, the Wireless Universal Resource File, in the early 2000s. WURFL is what many commonly call a DDR, or Device Description Repository. Luca identified the need for mobile developers to correctly detect device properties in order to serve the right content, analyze device capabilities, and support the right User Experience (UX) for all end users.
Note: While Luca was not the only one to see the problem, the industry believed that the problem was soon to be fixed by the inevitable standardization in the field. Luca perceived that no standard solution was in sight. Standard solutions are easy to implement when there is agreement on what to do among stakeholders. This was never the case with mobile devices and mobile browsers.The requirements of users, device manufacturers and content providers were often clashing. For example, content providers would offer scaled down versions of their services to mobile users, while certain users and browser manufacturers would be willing to go out of their way to escape the "mobile ghetto" and access the fully-fledged version of a website. More about this later.
Luca realized that devices needed to be categorized and recognized in real time by services and designed the framework that would allow precise mapping between a user-agent string and the complete profile that describes a device's capabilities. WURFL, the Wireless Universal Resource File, was born, originally as an Open-Source product in the early 2000s. In 2011 WURFL became a commercial product thanks to ScientiaMobile Inc, the company co-founded by Luca. Identifying mobile devices properly had become a big enough problem for multiple industries. That was the time when the WURFL DDR turned into a commercial-grade offering, with engineering, analysts and support teams behind it.
The Devil is in the Details
The problem of user-agent ambiguity is a real one. We will illustrate it with a few examples. The article we mentioned above already explained how all web browsers (including the ones from Microsoft Internet Explorer, Microsoft Edge, Google Chrome, Firefox and Opera, in addition to Netscape) ended up adopting a user-agent string that starts with "Mozilla", short for "Mosaic Killer". Mosaic was the name of the very first major web browser. When Netscape introduced a better browser, they named it "Mosaic Killer".
From that moment, all following browsers adopted the "Mozilla" string to prevent "segregation", i.e. receiving lesser versions of the web content from sites that were reserving a special treatment to the Netscape browser.
Since those days in the 90's, desktop web browsers have had user-agent (UA) strings that looked something like this:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
When the first internet-enabled devices came to market, they used to announce themselves with user-agent strings that were unambiguous and easy to make sense of. For example, content providers were able to easily recognize the popular (at the time) Nokia N80 by its UA string:
NokiaN80-1/1.0552.0.7 Series60/3.0 Profile/MIDP-2.0 Configuration/CLDC-1.1
That was also the time when certain mobile browser manufacturers figured that this user-agent also allowed some content providers to send mobile-specific content to the device. This was not necessarily what they wanted. Opera, for example, preferred fully-fledged web pages that they could transform into long columns for users to scroll through (and if the user-experience was poor, so be it). In order to avoid "mobile segregation", UA strings started to look something like this (Opera running on a Nokia N80):
Mozilla/4.0 (compatible; MSIE 6.0; Symbian OS; Nokia N80/; 9712) Opera 8.65 [ru]
While it was technically still possible to recognize a Nokia device for whoever tried hard enough with specific logic, this approach would fool the overwhelming majority of websites into believing that they were dealing with a desktop web browser.
It did not take long before Nokia and others adopted the same approach in order to escape mobile segregation. A few months later, also the Nokia stock browser would blithely lie about its identity (the Nokia N95 was the successor of the Nokia 80):
Mozilla/5.0 (Symbianos/9.2; U; Series60/3.1 NokiaN95/10.0.014; Profile/MIDP-2.0 Configuration/CLDC-1.1) AppleWebKit/413 (KHTML, like Gecko) Safari/413
Of course, this was not the end of it. Many content providers started to proactively chase the true identity of devices hiding behind obscure user-agent strings. After all they wanted to be in control of what their content looked like on different browsers and devices. The continuous battle between browser manufacturers and content providers to outsmart one another (along with the constant evolution of the device market) has led to the introduction of Device Detection.
WURFL was the first Device Detection framework and, still today, it remains the most popular by far, being used by all major Internet companies. The WURFL framework is a one-stop solution to maintain device detection in one place and make sure that it always stays updated with the latest and greatest knowledge of mobile devices released around the planet.
Use Cases for Device Detection
We mentioned that Internet content providers are the main users of Device Detection. Let's look at its main use cases.
Web Optimization and Content Negotiation
Web Optimization is about recognizing mobile devices and their properties in order to serve different User Experiences (UX) to users. In some cases, a service may identify mobile devices and offer different versions of the content to them. This is called "content negotiation". After all, mobile users tend to be in a different context than people who browse the web on their laptops. It only makes sense that large companies want to deliver a great User Experience (UX) to their mobile users.
Note: in the past, some have objected to the idea that different content is served to mobile devices on the basis of a presumed "web unity". Experience, though, has demonstrated that the need to serve a different UX depending on the mobile context and the feature of the device is real.
There was a time when this was handled through an m-Dot strategy, i.e. making sure that an alternate website
(typically available at a http://m.company.com URL)
would intercept and serve content to mobile users. Recognizing and redirecting those users to the mobile site was a
job for Device Detection.
Today the situation has somewhat changed. The arrival of Responsive Web Design (RWD) in 2010, along with the spread of smartphones has enabled "one-size-fits-all" websites, i.e. websites that serve the same HTML to all browsers, with "conditional CSS" in charge of performing the magic that "columnifies" web content on small screens (these conditional CSS are called CSS3 Media Queries).
Note: RWD has been somewhat simplified in the description above, but those are the main points.
Today, RWD has effectively replaced m-Dot websites for most major organizations. From an economic viewpoint, having a single HTML template outweighs the cost of maintaining a separate website and the infrastructure required for its support.
Of course, certain mobile-specific optimizations can only happen server-side. This has opened the way to RESS (2011), which stands for REsponsive web design and Server-Side components.
In short, RESS adopts all the advantages brought about by RWD, while avoiding a certain dogmatic approach by early RWD developer communities that called for avoidance of any kind of User-Agent detection in websites. The ability to contain all the front-end technology in HTML, CSS and JavaScript sounds great, but the reality is that Server-Side Rendering (SSR) can deliver huge gains in terms of download speed, page rendering speed and, ultimately, enhanced UX as compared to Client-Side Rendering (CSR).
The most impressive example of this is image optimization. Purely to convey the idea, while 300Kb images are commonly found on websites, mobile devices are better served with 30Kb versions of the same image. This has to do with download speed, rendering speed, device battery-life and general "responsiveness" to user events (clicking, scrolling, etc.).
ScientiaMobile offers solutions for this use case. ImageEngine will retrieve images from the customer's origin website, use WURFL Device Detection and other techniques to determine the optimal image size, format, and compression ratio, then resample the image for each specific device (or category of devices) and make those images available through a Content Delivery Network (CDN), greatly improving the UX for all users while reducing bandwidth.
Note: if you are not familiar with the concept of Content Delivery Network, this is infrastructure comprised of several servers at multiple Points of Presence (POPs) around the globe. These servers are usually referred to as Edge Servers. Large organizations will rely on CDN providers to cache their images and other content on their edge servers. Visitors to the organization's website will enjoy snappy downloads thanks to images being delivered from the closest edge server (as opposed to being retrieved from a server physically located across the globe).
While all companies generally want to improve their UX, this is an absolute must for eCommerce websites. Unnecessary page-load time has demonstrably negative impact on sales. Merchants want to speed up their load times. ImageEngine integrates seamlessly with your image management platform, and popular plugins such as Magento, WordPress, and virtually every other eCommerce platform that supports external CDNs for images.
Ad Tech: Device Detection for the Online Advertising Industry
Device Detection, and specifically WURFL, has been a key component in the advertising industry since AdMob (arguably the company that first created the infrastructure for mobile advertising) opened for business in 2006.
Initially, the main use of WURFL in Ad Tech was to determine (on the server and in real time) what the right size of a banner-ad should be based on the screen size and resolution of the requesting device. Soon after, Device Detection was responsible for one of the dimensions that Ad Networks were exposing to advertisers in its dashboards. Advertisers could now request that their media spend would be limited to users who own, for example, an Android 6.0 device or greater with NFC support.
Ad targeting happens along a multitude of dimensions, such as a user's location, gender and age. When it comes to device properties, most Ad Networks select a Device Detection solution as an alternative to reinventing the wheel. Being the Device Detection of choice for Facebook, Google, AWS, Akamai and many others, it goes without saying that WURFL is the most common selection in the Ad Tech industry. Not only is WURFL fast and scalable, but it is also tailored to the needs of an industry that relies heavily on avoiding false positives.
But there is more. Over the last few years, Ad Tech has grown to become a complex and sophisticated industry. Where we used to see a handful of Ad Networks, we now see DSPs, SSPs and Ad Exchanges.
Note: To provide some background on a pretty complex landscape, DSP stands for Demand Side Platform, i.e. companies that advertisers turn to in order to place their media spend on multiple publisher sites. Similarly, SSPs (Sell-Side/Supply-Side Platforms) aggregate publishers, i.e. those with popular website that have significant advertising space (called "inventory", in Ad Tech speech) to sell. Ad Exchanges may stay in the middle between SSPs and DSPs to serve the needs of all parties.
In the beginning, advertisers would need to go to Ad Networks to place their marketing campaigns on publishers' websites. This was a complex and time consuming process. Plus, publishers would often be left with significant unsold "inventory" (i.e. unclaimed advertising space) that they would prefer to sell cheaply instead of seeing it go to waste.
The solution came from the ability to sell unused inventory "programmatically", based on the same model used for real-time (RT) auctions in the financial world. Essentially, a user accessing a web page triggers a series of RT auctions for the advertising space on the page.
For example, DSPs would be seeing a programmatic object telling them that someone from a given IP is requesting a certain page with a device with a given user-agent string. Based on that information, DSPs would decide how much to offer in order to win that auction. Winners get to show their banners (Image, Adobe Flash program, video, etc.). This is called RTB or Real-Time Bidding. An organization called IAB is responsible for the standardization of the bidding format across the industry (OpenRTB specification).
While originally created to address the issue of unsold inventory, "Programmatic advertising" is now the standard way ads are sold on the Internet. And while not all advertising is RTB, a lot of it is.
This long, and hopefully informative, explanation was needed to explain the role of device detection in programmatic advertising. When a bid is received in the form of a JSON object, advertisers and DSPs will need to make the best use of the information they receive. Location and user profiling are important aspects to assess the true value of a bid, but so is the user-agent string, assuming one knows how to correlate it to a user's propensity to click on the add or perform an action that signals success.
Determining that a user-agent is actually a smartphone is one such piece of data. But DSPs can target even more effectively with device intelligence. A powerful example is the combination of a device's market price and its release date. Someone with a $900 device released in the last 4 months is more likely to pay for an App than someone with a 2 years old $600 device, and definitely more likely than someone with a 3 years old $200 device.
Device detection provides information that Ad Tech companies can use to build models and algorithms that better predict user behavior. In addition, Artificial Intelligence (AI) is just beginning to yield insights based on combining device intelligence with other data sources. WURFL continues to drive improved advertising across both the supply side and demand side of the industry.
Analytics: Making Sense of Your Logs
Everyone's business increasingly depends on data and data analysis. Organizations need to slice and dice their data looking for figures, patterns and correlations that guide their decisions. Device Detection is one of the dimensions along which data analysis happen. Companies will want to know how many of their users have tablets and smartphones, the breakdown of the different OSes and OS versions. Accurate Device Detection also enables a company to analyze their under-served devices (visitors with certain devices that are served a poor experience and thus don't tend to interact with content/services) and then tailor an experience to provider a better experience for those devices.
As an example of adoption of Device Detection for Analytics, the MOVR report (short for Mobile OverView Report) by ScientiaMobile is an example of how WURFL can be harnessed to gain great insight in one's web logs and device data.
Device Detection Solutions
Now that the main Device Detection use cases have been explained, it is time to look at the different technological solutions and how they relate to the use cases we have illustrated.
Different organizations have different needs. The programming language is often the key element that determines which DDR solution to adopt, but there may be others, such as Operating System, software architecture, reliance on a specific technology vendor and so on. Let's take a look at the different dimensions along which organizations may decide to acquire a Device Detection solution.
As far as programming languages go, certain enterprises have adopted the Microsoft technology stack. For those companies .NET is the framework and C# is almost invariably the language of choice. Many other enterprises have historically elected Java as their language of choice to avoid being tied to a single vendor. Other organizations still, particularly start-ups and cost-aware entities, go for PHP and the plethora of Open Source components that the LAMP stack brings with itself. More recently, languages such as Scala, Golang and Node.js have become the favorite of companies that look at languages with built-in multithreading support for scalability and ease of deployment according to recent Microservice-based paradigms.
Device Detection solutions are available for all the major languages we just mentioned.
While programming languages have been the key element for a couple of decades, this is not necessarily the case today. An interesting development over the past few years is that modern applications are now built as components that interact with one another. Actual programming is confined more and more to the pure business logic for each specific organization. "Reinventing the wheel" is the deadly sin nowadays. Historically, the database and the web server (or its glorified brother, the Application Server) were the two main components in most applications. That architecture is now called the monolith. More characters have entered the scene. Proxies, load balancers, log managers, caching proxies and CDNs, just to name a few, are the components that allow modern applications to scale all the way from an engineer's laptop to the virtualized infrastructure in the cloud that serves millions of users. In this context, it only makes sense that organizations are looking at ways to integrate Device Detection into their server instances of NGINX, Varnish Cache, HAProxy or Apache (again, just to name a few).
The software components we have just mentioned have been around for quite a few years. What is relatively new is the push for companies to abandon their own data centers in favor of virtualization (AWS, Azure and Google Cloud, just to name a few) and to adopt mechanisms to deploy their infrastructure with the power of Docker or similar "containerization" frameworks.
Companies that embrace Docker and Microservices are best served with a Device Detection solution that fits into those paradigms.
With these maps in mind, here is how different Device Detection frameworks address different needs.
On Premise Device Detection APIs
The first Device Detection use-cases appeared in the context of web programming and support for mobile devices. Companies needed a way to map HTTP requests to the features of the device and browser accessing their services. The obvious tool for this was an API (Application Programming Interface), namely a set of programming instructions that, given a HTTP request, could return the value of certain properties.
To illustrate this, we use pseudo-code:
import DeviceDetectionFramework;
:
device = DeviceDetectionFramework.getDevice(httpRequest);
property1 = device.getProperty("property_one");
property2 = device.getProperty("property_two")
if (property1 == "smarttv" && int(property2) >= 640) {
redirect("/largeSmartTv")
}
The spirit of this example is to show how certain HTTP clients can be directed to a specific UX tailored for them. It goes without saying that the number of possible examples is endless.
WURFL was the first Device Detection API. It comprises two components:
- A data file containing device data (the
wurfl.xmlfile).- An API, available in all major programming languages, to associate an HTTP request to device values just like in the example above.
The idea of isolating data into its own file was brilliant. As new devices hit the market at a constantly faster pace, updating applications to recognize new devices required a simple configuration update (a resource file) as opposed to a code update.
WURFL OnSite (On Premise APIs)
Java:
www.scientiamobile.com/secondary-products/wurfl-onsite-api-for-java/
(Documentation)
.NET:
www.scientiamobile.com/secondary-products/wurfl-onsite-api-for-net/
(Documentation)
PHP:
www.scientiamobile.com/secondary-products/wurfl-onsite-api-for-php/
(Documentation)
Scala:
www.scientiamobile.com/secondary-products/wurfl-onsite-api-for-scala/
(Documentation)
Note: the initial versions of the WURFL APIs
required manual updates of the Device Description file (wurfl.xml). Since 2015 this is no longer a
requirement. Commercial users of the WURFL OnSite "on premise" API can have an automatic WURFL Updater pull newer
device definitions automatically from their customer vault. ScientiaMobile releases updates on a weekly basis.
In addition to the module mentioned above, more languages are supported. While WURFL InFuze's core C API was originally developed to support integration of Device Detection with the infrastructure (see "device detection"-augmented Infrastructure below), it has also been adopted to support "on premise" APIs for more specialized languages:
Node.js:
www.scientiamobile.com/secondary-products/wurfl-infuze-module-for-node-js/
(Documentation)
Golang:
www.scientiamobile.com/secondary-products/wurfl-infuze-module-for-golang/
(Documentation)
Ruby:
www.scientiamobile.com/secondary-products/wurfl-infuze-module-for-ruby/
(Documentation)
Python:
www.scientiamobile.com/secondary-products/pywurfl-wurfl-infuze-module-for-python/
(Documentation)
C:
www.scientiamobile.com/secondary-products/wurfl-infuze-for-c/
(Documentation)
"Device Detection"-Augmented Infrastructure
While an "on-premise" API is a valuable tool in many situations, large enterprises prefer to frame Device Detection as a System Administration module integrated with existing components such as the web server or the load balancer. In those cases, the component in question is "augmented" with Device Detection knowledge. HTTP requests travelling through the component are enriched with HTTP custom headers containing the value of the properties that are relevant for the business logic of the organization. Examples of this are provided here.
ScientiaMobile supports this usage through its WURFL InFuze product family.
WURFL InFuze products:
NGINX:
www.scientiamobile.com/secondary-products/wurfl-infuze-module-for-nginx/
(Documentation)
NGINX Plus:
www.scientiamobile.com/secondary-products/wurfl-infuze-module-for-nginx-plus/
(Documentation)
Varnish Cache:
www.scientiamobile.com/secondary-products/wurfl-infuze-module-for-varnish/
(Documentation)
HAProxy:
www.scientiamobile.com/secondary-products/wurfl-infuze-module-for-haproxy/
(Documentation)
Apache:
www.scientiamobile.com/secondary-products/wurfl-infuze-module-for-apache/
(Documentation)
IIS:
www.scientiamobile.com/secondary-products/wurfl-infuze-module-for-iis/
(Documentation)
This video from the 2017 NGINX conference in Portland, OR, shows Luca Passani illustrating how the NGINX module works and the general point of augmenting HTTP requests
Device Detection in the Cloud
In spite of the ease of use of "on premise" APIs, some organizations favor the Cloud model, i.e. using a simple local API that reaches out to a service in the Cloud to trigger the service logic. This way, organizations are no longer required to update the device definition file (albeit this process has been greatly simplified and automated by the availability of repository updaters).
While ScientiaMobile does not normally refer to the WURFL Cloud service as 'an API' (to avoid confusion with its OnSite API), the service does rely on users to install a local client API. These local APIs are referred to as "Cloud clients". Cloud clients, available for several programming languages, will speak to the service "under the hood" totally transparently to the API user.
WURFL Cloud:
Getting Started:
www.scientiamobile.com/page/getting-started-with-wurfl-cloud
WURFL Cloud Clients:
www.scientiamobile.com/ wurflCloud/downloads
Device Detection in Microservice Architectures
As we mentioned above, a typical software architecture would place a database (with all the business information) in the middle, and have web applications, desktop applications and mobile apps dance around it. Today some refer to that architecture as "a monolith". While monolithic architectures have been common in the past, they tend to become a bottleneck in terms of maintenance, service availability and development (particularly in the case of geographically distributed engineering teams). The modern way to do things is to have components that work independently and just know the minimum that there is to know when it comes to interacting with other components in the system. This allows complex products to evolve with the support of different teams without the limitations posed by, for example, a database that only a few are allowed to touch. Once subsystems know how to cooperate with one another through an agreed protocol, different teams may pick different technologies for the job.
Adopting microservices comes with extra cost. In the case of large and complex applications that need to scale, this aggravation may well be worth it. Thanks to microservices, DevOps can deploy multiple instances of each subsystem, assign traffic to them, and support high levels of concurrent user traffic. In this context, some organizations may wish to have Device Detection be one such subsystem whose purpose is to attend the device detection needs of the entire system. While several different formats and protocols can fit the definition of Microservices, a very popular approach is to have subsystems speak to one another through an HTTP-based REST API and the exchange of JSON objects.
This is exactly the approach that ScientiaMobile has taken with its WURFL Microservice product:
In essence, a web server will be listening to other components that may need to know what the properties of a given device are. While those other components can speak HTTP directly to the REST server, local APIs (conceptually identical to the Cloud Clients mentioned above) will make interaction as simple as using a traditional on-premise API, including the added advantage of a local caching layer. This allows performances of tens of thousands of device detections per second or more in each of the subsystems accessing the component.
WURFL Microservice
Docker:
www.scientiamobile.com/page/wurfl-microservice-docker-device-detection
AWS Marketplace:
www.scientiamobile.com/products/wurfl-device-detection-microservice-aws/
Azure Marketplace:
www.scientiamobile.com/products/wurfl-microservice-for-azure
Google Cloud Platform Marketplace:
www.scientiamobile.com/products/wurfl-microservice-for-gcp
Device Detection and JavaScript
Web developers familiar with JavaScript tend to prefer JavaScript-only solutions to the issue of identifying browser feature. Over the years, the industry has graciously converged to the term Feature Detection to identify the JavaScript ability to determine quite a few browser properties from within the browser itself.
Without getting too technical, web developers can run JS tests like the following and determine that certain UX optimizations are supported by the current browser or device:
if ( awesomeNewFeature ) {
useAwesomeNewFeature();
} else {
fallBackToOldUserExperience();
}
Note: Modernizr is the "mother" of many Feature Detection frameworks that have appeared over time.
While Device Detection was the way to go for UX optimization back in a time when devices were not so powerful, smartphones and tablets have virtually equaled desktop browsers in terms of capabilities. Feature detection tends to go hand in hand with Responsive Web Design (mentioned above) toward the common goal of optimizing UX.
But not exactly everything can be achieved through JavaScript. In addition to so called undetectables (features whose support cannot be positively determined through JavaScript), JavaScript will have a hard time determining important aspects of the browser or device which may be relevant for UX and for analytics purposes. For example, a device make and model may not always be clear by looking at JavaScript. The actual form factor of the device in the hands of the user may also be equally hard to determine with JavaScript alone.
These are issues that Device Detection will have a better shot at handling, thanks to the considerable effort that goes into maintaining device profiles for each specific device in a DDR.
An interesting product in this regard is WURFL.js (pron: WURFL dot JS) by ScientiaMobile. WURFL.js unites the ease of importing a JavaScript library with the power of server-side Device Detection.
While ScientiaMobile warmly recommends signing up to use the free service (in order to be able to offer support to all users), using the tool is as simple as including this HTML line in your pages:
<script type="text/javascript" src="//wurfl.io/wurfl.js"></script>
From that moment, the JavaScript environment will be able to access JSON objects such as this one:
{
"is_mobile": true,
"complete_device_name": "Google Nexus 5x",
"form_factor": "Smartphone"
}
This means that programmers can exploit Device Detection from within JavaScript with constructs like:
if (WURFL.form_factor === "Smartphone") {
// targetSmartPhoneDevices();
}
Note: there is actually a bit more magic going on for certain devices. For example WURFL.js can also detect the actual hardware, e.g. iPhone 8 Plus, of iOS devices (a piece of information that iPhone user-agents carefully hide from developers) in most circumstances.
UX optimisation is not the only use-case the WURFL.js supports. The tool can also be easily integrated with Google Analytics (GA) to obtain detailed analysis of the traffic to one's website along device dimensions. For example, this tutorial gives detailed instruction on how detection of actual iPhone hardware can be added to Google Analytics. This will allow users to obtain nice GA reports such as this one:
WURFL.js is also available commercially with its Business Edition. The main differences with the free version are the number of properties offered and certain performance optimizations.
WURFL.js
Free Edition:
https://web.wurfl.io/#wurfl-js
WURFL.js Business Edition:
https://www.scientiamobile.com/page/wbe
Device Detection and Image Optimization
While discussing JavaScript above, we mentioned how certain problems are not solvable with a JavaScript only approach (or, at least, not satisfactorily) for everyone. The biggest issue in this sense is Image Optimization.
In spite of the improvements of devices and bandwidth, companies have figured out that, for example, a 300kb images, while suitable for the desktop web, is ill-suited for a mobile device. Always for example, a 30kb version would be better for that device. The actual time and bandwidth needed to download the image are only part of the problem. The other part is actual rendering time and how reactive the browser stays to user events (scrolling, zooming) when large images trigger a lot of calculations on those devices. Studies have revealed direct correlation between loading time and user propensity to stay on the site and complete tasks
To mitigate this huge issue, the industry has converged on a standard that augments HTML with the
ability to specify multiple versions of an image, so that browsers can pick the most suitable one for
the specific browsing context. This standard is the
"picture element".
Web developers can now replace the tradition "img" tag with an object that looks like this:
<picture>
<source media="(min-width: 36em)"
srcset="2art_full_3050.jpg 3050w,
2art_full_2048.jpg 2048w,
2art_full_1280.jpg 1280w,
2art_full_1024.jpg 1024w,
2art_full_800.jpg 800w,
2art_full_400.jpg 400w"
sizes="50vw" />
<source
srcset="2art_crop_1860.jpg 1860w,
2art_crop_1280.jpg 1280w,
2art_crop_1024.jpg 1024w,
2art_crop_800.jpg 800w,
2art_crop_400.jpg 400w"
sizes="100vw" />
<img src="2art_crop_320.jpg" alt="art"/>
</picture>
This code is instructing the browser to look at itself, understand which category of screen width it is falling into, and pick the image that the developer has intended for that category. As an added bonus, old devices will still be able to decipher the old img tag and still show a picture.
The picture element is certainly an important advancement in HTML. For practical purposes, though, it has only transformed a technology problem (i.e. how to serve multiple versions of the same picture) into a problem of Art Direction. Simply put, organizations are left wondering how they can afford an art direction team that can produce and manage all of those different versions of images.
This is a case where Device Detection can help significantly. Image optimization tools typically deploy complex machinery to optimize images. Device Detection is one of the parts that makes real-time on-the-fly optimization possible
One such tool is called ImageEngine and it is extremely simple to deploy and use. In essence, web developers just need to rewrite their image URLs in a way that ImageEngine gets a chance to retrieve and repurpose them for each device or category of devices:
<img src="https://try.imgeng.in/http://example.com/image.jpg">
This code will resize and optimize the image according to a device screen size. The detection of the device and its screen size comes from the latest version of WURFL. This is the simplest way of putting ImageEngine in action. Web developers just need to apply 'URL rewriting'.
In addition to the simple form, developers can also tap additional powerful ImageEngine features through 'resizing directives', i.e. actual commands that can piggyback on the query string (ImageEngine documentation has the details). ImageEngine directives can be used along with the picture element to serve multiple instances of an image, regardless of whether the picture has been available for some time or it has just been published on the website:
<img
src="https://try.imgeng.in/w_200/http://example.com/image.jpg"
srcset="https://try.imgeng.in/w_300/http://example.com/image.jpg 1.5x,
https://try.imgeng.in/w_400/http://example.com/image.jpg 2x"
width="200" alt="Me">
Everything we wrote about ImageEngine so far matters to web developers the most. System engineers tend to have other concerns and worry about the performance and solidity of a third-party system in the cloud for such a key activity as serving images. This is particularly important for e-commerce sites.
ImageEngine is coming out strong also in regard to performance and reliability. While the image optimizing part is doing its job in the background, ImageEngine also includes a Content Delivery Network (CDN) function. In fact, ImageEngine is a CDN. Resampled pictures are stored in one of many Edge Servers and served to users that are close, in the network sense of the term. This contributes to greatly minimizing latency. The main difference with other CDNs is that ImageEngine's Edge Servers are really smart and run fully-fledged WURFL instances to determine which image is right for each user. Among other things, this also translates in significant CDN bandwidth savings, a concept that ScientiaMobile calls Smart Byte.
Device Detection and Data Analysis Tools
A Device Detection solution for Analytics will analyse HTTP logs and find a way to reduce hundreds of thousand of different user-agent strings to only a few thousands entities. Such Extract, Transform, and Load (ETL) tasks are a traditional step in the Business Intelligence process of organizations. Large companies invest significant resources in Business Intelligence tools to:
- Diagnose potential issues with their mobile experience, i.e. problems that may be impacting their mobile users.
- Measure conversions of the mobile channel, possibly through observation of specific mobile device features.
- Plan future developments of the mobile channel based on analysis of current and projected customer device data.
In the field of Data Analysis, there are four major categories of tools that benefit from Device Detection:
- ETL (Extract, Transform and Load) tools are typically used to load data into a storage system. Tools in this category include Talend, Informatica PowerCenter and IBM DataStage.
- Desktop Data Analysis tools are used by data analysts to dig deeply into smaller sets of data in a self-sufficient way. This category includes Tableau, Microsoft Power BI, Microsoft Excel and other Spreadsheets.
- Big Data Analysis tools are used to store and analyze very large sets of data and include Apache Hadoop, Apache Spark, Amazon RedShift and Teradata.
- Real-Time Analytics tools are specifically designed to analyze real-time or near-real-time data that is constantly streaming through a data pipeline and include tools like Splunk, Striim, Zoomdata, Apache Storm and the ELK stack.
When applying Data Analysis to mobile device data, raw user-agents are suboptimal when it comes to joining them with other proprietary data within an organization. This is a big issue. What is missing is a more generic Mobile Device ID that can group virtual identical devices under the same make and model name, possibly with other characteristics such as OS Version or form factor added for more in-depth analysis.
More often than not, Device Detection takes the form of an API also in the Analytics world. While most people in the Analytics and Business Intelligence (BI) spaces know what an API is, not everyone has the knowledge, the time or a cooperative IT department to integrate an API with their BI tools of choice.
Most tools in this Analytics/BI space are Java-based, so the WURFL OnSite for Java API is usually a perfect fit; However, ScientiaMobile also offer plugins for Splunk and Apache Spark, as well as a standalone tool called WURFL InSight for Device Analytics, which can be used to enrich web logs with device intelligence.
For those who have the ability to integrate API without particular effort, discussion about Device Detection and Analytics falls back into the illustration of a Device Detection API. For the others, there is a different approach to the problem.
WURFL InSight supports all the aspects that a Device Detection solution for Analytics demands.
The Five Dimensions of a Device Detection Solution
There are several aspects to consider when evaluating a Device Detection Solution. ScientiaMobile identifies five:
- Accuracy: Can the framework identify the browser and the device to the maxmium extent that the information in the HTTP request allows? This includes avoiding "false positives".
- Speed: Are average and maximum device detection times low enough (order of microseconds per look-up)?
- Memory Usage: Is usage of RAM kept under control?
- Extensibility: Can users augment/tweak the framework independently to address specific detection needs without requiring vendor intervention?
- Support: Does solution vendor has a dedicated support team that can analyse and address the needs of users within a guaranteed timeframe?
Figure: the five dimensions that define a Device Detection solution
We will discuss these five dimensions in more detail.
Accuracy
This is, arguably, the most important aspect of Device Detection. After all, anyone can look for the presence of the "iOS" or "Android" substring in an User Agent string and approximately determine that a User-Agent qualifies as a smartphone. A way to detect iPhones that works in most cases could look something like:
var isiPhone = /iPhone/i.test(navigator.userAgent);
Note: This code snippet is JavaScript for 'consider every user-agent containing iphone an iPhone regardless of capitalization'.
This approach is OK in many situations, but it will hardly cut it in those cases where precise detection is involved. The Ad Tech industry, for example, prefers not to detect the following as an iPhone or any other iOS device.
Tiphone T67/1.0 Browser/wap2.0 Sync/SyncClient1.1 Profile/MIDP-2.0 Configuration/CLDC-1.1
Of course the situation gets worse when one sees things like the following in their logs (Lumia 930):
Mozilla/5.0 (Mobile; Windows Phone 8.1; Android 4.0; ARM; Trident/7.0; Touch; WebView/2.0; rv:11.0; IEMobile/11.0; NOKIA; Lumia 930) like iPhone OS 7_0_3 Mac OS X AppleWebKit/537 (KHTML, like Gecko) Mobile Safari/537
Note: there is space to argue that the JavaScript one-liner above is not an example of Device Detection, but rather an example of what many call User-Agent sniffing. This is a good objection, since UA sniffing usually takes the form of a simple a script, while Device Detection is about employing man-years to create a Device Description Repository and keeping it updated, which is entirely different
When it comes to accuracy, WURFL is the Gold Standard. The ScientiaMobile team constantly monitors the "user-agent" space (as observable thanks to billions of requests logged by its Cloud systems) looking for unrecognized or wrongly recognized user-agent. Upon discovering on unrecognized user-agent, the team will either add new profiles to the repository or, in some cases, design improvements to the matching logic to ensure correct detection (and avoid false positives).
Speed
Paraphrasing a famous Albert Einstein's quote: A Device Detection API should be as fast as possible, but not faster!
This claim may seem self-evident, but we must stress that a fast Device Detection solution makes sense only as long as the results are sufficiently accurate for an organization's goals. While quick and dirty detection may be enough in some cases, other situations call for precise detection that has a cost in terms of speed.
As a rule of thumb, a modern Device Detection API should be able to perform a device detection in a matter of microseconds, i.e. achieve tens of thousands of device detections per second. On top of that, the API should support multithreading, allowing linear scaling to achieve higher throughput.
In the case of WURFL, actual speed depends on the platform of choice. Java, .NET, PHP and C are supported. Speed varies depending on programming language and hardware. Generally speaking, though, all ScientiaMobile APIs perform tens (or even hundreds) of thousands of device detections per second and support multithreading. This makes WURFL suitable for virtually all real use cases. Just as an example, Facebook uses the WURFL PHP API for certain optimizations in the mobile UX.
While higher speeds are possible, typically this only happens at the cost of sacrificing precision to some level, which is something the standard WURFL API will not do as a philosophical choice of prioritizing accuracy. As previous examples above show, certain device and browser manufacturers will go out of their way to disguise the true nature of the user-agent. WURFL is powerful enough to adapt and leverage any single character in the user-agent to differentiate between virtually identical strings, while, at the same time, reconciling different user-agents that refer to the same WURFL profile. Internally, this is achieved with custom logic and a controlled use of Regular Expressions (RegExes).
Note: some have argued that regular expressions are slow. This is not correct. Regular expressions are a powerful way to analyse text. That power may lead to regular expressions that 'backtrack' several times and result in slow processing time. But this does not need be the case. ScientiaMobile regularly performs regression tests on new API versions and ensures that the regular expressions in them always perform in fast and predictable ways.
In addition to its traditional APIs, ScientiaMobile offers Device Detection solutions specialized tailored to the needs of CDN vendors and Hardware Manufacturers with stringent speed usage requirements.
Memory Usage
Similarly to speed, using less memory is generally a good thing, but this requirement should be evaluated in the context of speed and detection accuracy.
WURFL APIs are highly configurable. Organizations that adopt WURFL can trade speed for memory (or the other way around) through configuration. Generally speaking, though, companies are ok with the default setup that comes with the API. Tens or hundreds of megabytes of memory is a small expense when compared to the performance requirements and business impact of an application.
In addition to its traditional APIs, ScientiaMobile offers Device Detection solutions tailored to the needs of CDN vendors and Hardware Manufacturers with stringent memory usage requirements.
Extensibility
There are scenarios where users may want to augment the detection properties of the framework along different dimensions. For example, a mobile carrier (also called MNO, for Mobile Network Operator) may want to prearrange their portal and the services provided by its partners for the launch of a new device, without making the specification of that new device known to third parties.
A different scenario is a company that needs to track its own custom device properties.
For example, they may have a different definition of what constitutes a smartphone or they may want
to hand-pick devices that they consider powerful enough to run a certain app. Other scenarios still,
call for a separate UX for Smart TVs and specific gaming consoles. Custom user-defined properties like, for example,
leanback_ux (true or false) would serve that purpose.
A Device Detection framework should allow its users to expand the detection function along two dimensions:
- New devices: create new device profiles.
- New properties: create new properties and set their values at different levels of granularity, such as a class of devices or even a specific device make and model.
The WURFL API offers full framework extensibility. Through a mechanism called WURFL Patch File, users can define, augment and override devices and device properties to tailor the framework to their specific needs in those scenarios where this makes sense.
In addition to this, ScientiaMobile will license the source code of the WURFL APIs to companies with demonstrable specific needs to override or augment the logic of the API.
Support
As we said, approximate solutions may be sufficient for some, while others need more accurate and reliable device detection as part of their business model. These are the organizations that want to have access to a solid support team for their device detection needs. Along the way, they may find that a not so popular device is actual popular in a certain country, or that a specific App needs to be tracked through device detection. Those are cases where the support team from a solid Device Detection vendor really makes a difference.
A support team will help in a lot of different scenarios. Just to mention a few:
- Verify that the solution is being used correctly.
- Point to updates to the data or the API that will addressed a problem.
- Put specific data updates on the fast track for a specific customer. This may take the form of making sure that the data is available in the first planned update or even emailing a user with a specific out-of-band update that addresses a specific issue.
- Plan updates to the API logic to recognize new categories of user-agents, such as new apps, specific ad-network SDKs, new Smart TV families and so on.
The ScientiaMobile support team responds to customer support requests in a matter of hours, sometimes minutes. In addition to addressing customer needs that require immediate attention, the support team will happily listen to customers whose issues and needs may very well shape the requirements for the evolution of the product. This is simply the continuation of the original open source spirit that lies at the foundation of WURFL and that so many organizations have come to love over the years.
Conclusions
In this document, we have illustrated state-of-the-art Device Detection. As the leading provider in this space, ScientiaMobie's views are certainly biased, but we did our best to provide a fair and balanced overview of the all the different aspects of Device Detection and the actual tools to implement it in practice.
We have elected not to enable comments on this document, but everyone with comments or questions is welcome to email them toFeedBack[XX]@devicedetection.com(please replace[XX] with the sum of number 17 + one).