Are transcoding proxies needed on the mobile?

Most users have never used the mobile web, in some cases they have downloaded a few ringtones, a wallpaper or a java game. The operator portals are the still the first and probably only site that the average user has ever visited. Sometimes some brave individual looks for adventures and surpasses the borders of his operator portal. These users either know exactly what they want and know the URL or don’t have any idea of what they can find. Users who don’t know what they can find will go to some search engine like Google or Yahoo! (hardly the mobile versions) and type a keyword such as “tennis”. They will get a list of links and follow them. There are very good chances that they will reach a web site that doesn’t even take into consideration the possibility that a user with a mobile phone will visit and is not doing any device detection or content adaptation.
If this is case, and unfortunately still is for most sites, there are two chances, either the user has a really smart browser like Opera Mini or Nokia’s webKit-based browser, or he will download a huge page that he will not be able to read. After 2 or 3 different sites have been tested, users will be discouraged and will not take the risk of going out of the operator’s portal ever again.

Operators should be happy about this and try to disincentivate browsing outside of the portal. On the other hand, browsing means traffic, traffic means money, so accessing the “full web” can be a positive thing for mobile operators too. If browsing can be a revenue stream you don’t want your users to be discouraged, you want them to have a taste of what they can get on their mobiles.
Opera Mini is one possible solution, transcoding is the other. Google offers transcoding automatically, operators might choose to get their own transcoding software. The result is that users get what they want (the tennis site), operators get what they want, money from data traffic. Looks like a win-win solution.

Within some limits, this is effectively a win-win solution. The limits are defined by the user experience that the transcoding engine can provide. It is proven that transcoding a web page and fitting it into the small screen of a mobile device will not provide the best user experience, nevertheless it is still better than a kick in the teeth (like the average rendering of a full web site on a Motorola V3).
Transcoding is like a shortcut to get more users on the mobile web. Major web sites should consider developing a mobile version quickly simply because mobile is the future. At the same time there is a number of sites that will never be converted (think of all the geocities pages!) and there are many minor sites that will consider mobile only in 3 or 5 years from now. We do want users to be able to find information on the mobile today, don’t we?
Transcoding engines must be smart enough to recognize sites that are made for the mobiles and leave the content as is and convert the content only when this will not fit the features of the mobile device (or set-top-box). Transcoding engines should be smart and act when needed and never get in the way, they must be an extra tool, not a mousetrap that gets in the way of the developers and designers that are sweating to produce content that is optimized for mobiles. Transcoding the content of a site that was made for mobiles will most likely break the usability and optimizations that the designer put in place.

At the same time, users should be educated that there is a lot of information for them on the internet and that they should be looking and asking for mobile versions. We should not expect the average user to understand mark-ups, XHTML-MP [PDF], transcoding, image rescaling, but they do understand when a page is not usable on their mobile and if they can’t find anything interesting, they won’t come back.

Is it good to hide the user-agent in mobile?

Recently Vodafone plugged a new infrastructure in their mobile network. In two words, it is a proxy that is also capable of transcoding web page contents to be suitable for mobile devices.
While I don’t know the marketing plan behind this, it is obvious that the main driver is to make the entire web available on mobile devices, even if the source content was not designed for use on mobile devices. This is good, of course, because it allows all Vodafone (UK-only at this time) customers to access all the valuable information that you can find on the web normally, all that information you’re used to (and I’m addicted to, I’d say). But how is this software working? It is a proxy sitting sitting in the middle between the mobile device and the open internet, all contents requested by Vodafone-UK users are received by the proxy that performs the requests, gets the page, elaborates it (or maybe simplifies it) and delivers a new page to the mobile browser. While processing the page, the proxy takes out all contents that might create a problem to the mobile browser, tries to clean up the markup if there’s any tagsoup and sends something that is optimized for the mobile.

All this sounds perfect, doesn’t it? Well, not to me.
There are a few glitches here and there that I think are breaking the original plan of improving the user experience. First of all the proxy sends a unique user-agent HTTP header string that hides the original device behind a generic Mozilla-compatible browser. Remote servers will think that the requesting browser is a desktop PC and send the “appropriate” content, unfortunately sometimes even the remote applications have an adaptation engine that will provide content for desktop PC that is not meant for the mobile. Let’s think of a company selling ringtones and contents for mobile phones, when visited by a mobile device will let the user pick a content and download it directly, but if a desktop PC is recognized, it will ask for the mobile device model, show the available contents, ask for the phone number and then send a wappush to let the user download the content. If the proxy is hiding the mobile device and pretending it’s a desktop PC the original plan is failing.
The answer to this would be to have a protocol that describes how the remote web application could tell the Vodafone proxy that it IS capable of providing appropriate content for mobile and stop the adaptation, unfortunately, such a protocol does not exist, as of today.
Is the user-agent SO important? At this time IT IS so important, there isn’t in fact, a technology that lets a web server know the full capabilities of a mobile device. The OMA and the W3C are both working on this topic, but the technology will not be ready before 1 year or so and then we will have to wait for it to be implemented in mobile devices and later to become a common feature: you can easily imagine the timeframe. The solution proposed by Vodafone is that sites that are able to provide mobile content should contact them and provide all the URL’s. Sorry, but this doesn’t seem very scalable.

Another tiny issue is that the above plan makes most sense if you’re thinking of old WML browser that would not be able to parse an HTML or XHTML page, but what about the recent WebKit implementation that Nokia is delivering pre-installed in their Symbian phones? What about Safari on the iPhone? What about Opera Mini (that I have to admit is a quantum leap in browsing as it provides very smart browsing features to all J2ME devices)? All these browsers provide TODAY (and even 1 year ago if you think of Opera and the first devices running the WebKit-based browser) a technology to adapt content that was meant for desktop PC’s with big screens to be viewed and used decently on a mobile device. Unfortunately, again, there isn’t a technology that will tell the Vodafone proxy if the device is running a smart browser…. Oh, wait… The user-agent string DOES tell it. So let me recap: Vodafone reads your browser’s user-agent string, tries to understand how smart your mobile browser is and then hides the very same information to the remote server! Is this a dog biting its own tail?
But, isn’t Opera Mini doing the same, using a unique user-agent for all mobile devices? Yes of course, but the difference is that Opera mini is installed by the user, the user knows about it and also has a chance to set a few options to make the software behave as she better likes it and, by the way, if she doesn’t like it, she can use the native browser or install another one, if available. Vodafone’s proxy is doing this without asking anything to the user and without allowing any personalization and configuration. It looks like Vodafone tries to be smarter than the web designers, developers and users…. All at once. I think that a lot of smart people work for Vodafone, but I suspect they can hardly address ALL needs or ALL users. Offering a choice is always the smartest solution. Vodafone should make a choice, let the user know and let her pick the best option for him.

I think that what Vodafone is missing is the context. From a user perspective Vodafone will never know the context in which the user is and from a technical point of view it is not allowing remote web applications to know the little that the user-agents and normal HTTP headers provide.
I think that this is a very good plan in theory, that unfortunately is not matching the reality. Transcoding without letting the user be in control is so much 1999 and WAP 1. Reminds me of WAP Gateways, proxies that were created to optimize network usage (and did it well), but were removed to open to gates to HTML and open browsing. Vodafone is implementing a new WAP Gateway that seems to be doing the wrong thing at the wrong time.
If and when there will be a technology that lets the user be in control and user a proxy to optimize network usage and resources, then I’ll be in line to use it. For the time being I’m happy to use Opera Mini for some sites and the native browser for other sites.

Some links worth to mention on the topic:
Novarra (the company that developed the transcoding proxy
Vodafone Mobile Internet and Content Services
Opera Mini
Barbara Ballard’s more than just a pretty face
Mobile Web Best Practice Working Group – Content Transformation Task Force
Client-Specific Web Services by Using User Agent Attributes