Measuring the speed of resource loading with JavaScript and HTML5

This is a follow up article to Measuring site performance with JavaScript on mobile, I suggest you read it before you continue. In the previous article I talked about the Navigation Timing spec, here I will talk about the Performance Timeline and Resource Timing specs and how they work in IE10, the first browser to implement them. I created a page that shows some of the data available and a library that generates a HAR that you can later analyse.

Showing basic performance details of page resources

The Performance Timeline is a spec that extends the Performance interface defined in the Navigation Timing spec and adds a new interface called PerformanceEntry. The new additions give to developers a quick overview of the performance of each resource that composes the page and the order in which each resource was loaded. For anyone that wants to quickly see the timing information of the page this is more than enough and very easy to use. Below you can see the most basic use of the API:

function simplePerf() {
 var pe = performance.getEntries();
 for (var i = 0; i < pe.length; i++) {
  if (window.console) console.log("Name: " + pe[i].name + 
   " Start Time: " + pe[i].startTime + 
   " Duration: " + pe[i].duration + "n");
 }
}

Once you have a list of the resources that the browser has loaded you can further explore the details of each HTTP request. Similarly to the Navigation Timing API developers have access to the timings of any redirects, DNS lookups, network connections and of course the time to make the actual request and receive the response. For each resource is also available the initiatorType, i.e. how the resource is referenced in the HTML page; the types available are “css”, “embed”, “img”, “link”, “object”, “script”, “subdocument”, “svg”, “xmlhttprequest”, “other”. If you have IE10 on your Windows PC or a Windows Phone 8 device you can visit this page and try yourself.

How does it work?

The example is made of 4 parts, the HTML itself, a tiny CSS file to give a minimal styling to the tables, a JavaScript that does most of the work and a couple of images. The code is available on Projects.

onload, featureDetect() is fired. featureDetect() checks the browser support for the Performance Timeline and Resource Timings specs and shows a message in the page. If supported, it also enables the two buttons. This is the code:

function featureDetect() {
 var rt = "Oh rats! Your browser does not support the Resource Timing spec :(";
 var pt = "Oh rats! Your browser does not support the Performance Timeline spec :(";

 if (!!window.performance.getEntries) {
  pt = "Excellent! Your browser supports the Performance Timing spec";
  var e = window.performance.getEntries();
  for (var i in e) {
   rt = "Excellent! Your browser supports the Resource Timing spec";
   document.getElementById('myButton2').disabled=false;
   document.getElementById('myButton3').disabled=false;
  }
 }
 document.getElementById('perf_support').innerHTML = 
  document.getElementById('perf_support').innerHTML +
  rt + "<br>" + pt + "<br>" + "<br>";
}

If the user clicks on “Load another image” the function loadResources() is fired. All this function does is create a new image() and append it after the existing image, this is useful to display more timing data, more on this later. loadResources() on its own is not very interesting and you should not spend time on this. What is interesting to see is that the browser continues keeping track of all the events and the information is available via JavaScript. Notice that I use a PHP script to delay the delivery of the image; the only reason is to make sure that the response time is a big enough number.

Clicking on “Display basic performance data” a table is shown. The table is printed by the function displayPerfData() which takes window.performance.getEntries() and loops through the elements to show when the browser started loading each element and how long it took to load it. The script uses the duration property to access the final timing. For basic measurements this is probably all you need to know, easy, hu?

function displayPerfData() {
 if (!!window.performance.getEntries) {
  var perf_data = loadPerfTimeData();
  document.getElementById('perf_data').innerHTML = "" + perf_data + "<br>" + "<br>";
 }
}

function loadPerfTimeData() {
 var e = window.performance.getEntries();
 var perf_data = "<table id='table_perf_data'>
  <thead><tr><td>Resource</td><td>Start time (ms)</td>
  <td>Duration (ms)</td></tr></thead>n<tbody>n";
 for (var i in e) {
  if (e[i].name == "document") {
   perf_data = perf_data+"<tr><td>HTML document</td>";
  } else {
   perf_data = perf_data+"<tr><td>"+e[i].name.replace(/^.*/|.$/g, '')+"</td>";
  }
  perf_data = perf_data+"<td>"+e[i].startTime+"</td><td>"+e[i].duration+"</td></tr>n";
 }
 perf_data = perf_data+"</tbody>n</table>n";
 return perf_data;
}

The browser provides the values for the start time and the duration in accordance to the High Resolution Time spec. The start time is particularly interesting because it shows exactly when the browser initiated the request for the object. This also means that the browser keeps track of when the user clicked on “Load another image”.

Similarly, “Display detailed performance data” loops through getEntries() to get a list of the resources the browser loaded, but instead goes into more details and breaks down the data into network-related operations (DNS lookup, open the connection, SSL, etc), request time and response time. For my example I bundled up the timings into three buckets, network, request, response, refer to the Processing model for a detail of all available properties. In my tests the request time was often close to zero when using a Wifi. For the sake of readability I have rounded the timings. This is how it works:

function displayDetailedPerfData() {
 if (!!window.performance.getEntries) {
  var d = loadResTimData();
  document.getElementById('detailed_perf_data').innerHTML = "" + d + "<br>" + "<br>";
 }
}

function loadResTimData() {
 var e = window.performance.getEntries();
 var perf_data = "<table id='table_perf_data'><thead><tr>
  <td>Resource</td><td>Network (ms)</td><td>Request (ms)</td>
  <td>Response (ms)</td></tr></thead>n<tbody>n";
 var t = [];
 for (var i in e) {
  if (e[i].name == "document") {
   // for the document refer to window.performance.timing instead,
   // we skip it for this example
   continue;
  }
  perf_data = perf_data+"<tr><td>"+e[i].name.replace(/^.*/|.$/g, '')+"</td>";
  if (e[i].requestStart==0) {
   // resource is cached, some entries are zero, 
   // we default to fetchStart instead
   perf_data = perf_data+"<td>"+Math.round(e[i].fetchStart-e[i].startTime)+"</td>";
  } else {
   perf_data = perf_data+"<td>"+Math.round(e[i].requestStart-e[i].startTime)+"</td>";
  }
  perf_data = perf_data+"<td>"+Math.round(e[i].responseStart-e[i].requestStart)+"</td>";
  perf_data = perf_data+"<td>"+Math.round(e[i].responseEnd-e[i].responseStart)+"</td>";
  perf_data = perf_data + "</tr>n";
 }
 perf_data = perf_data + "</tbody>n</table>n";
 return perf_data;
}

While I was writing the script I also thought of another example, but eventually decided to take it out as I did not find a suitable use in my specific use case. What I wanted to show was how you can also query for specific entries and filter by type looking at the initiatorType. What I initially thought was to fire an event once the extra image is loaded and show data about it specifically. I thought it was interesting to attach an onload event to a resource that I am loading at a later stage and measure its performance, but the example is so small that it did not fit. If you are curious look at resourceTiming(). First of all I get a list of objects filtering by type “resource”, this means I will get a list of objects such as images and stylesheets, then I loop through them and filter out everything but images. Calculating the time it took to load it is a matter of making a subtraction.

The source code is available on GitHub.

When to check the performance object?

In my example I decided to use a button for two reasons, I wanted to make sure it happens once all the parts of the page are loaded and I wanted to allow visitors to load the extra image and then refresh the timings.

Truth is, there isn’t a right time to access the performance object, it depends on your application. If your script makes an XHR request, loads images dynamically or anything else you can always check both when the operation started and how long it took. The “Load another image” button is an example, more complex Web apps might have much more meaningful reasons to monitor the performance tied to specific events other than onload.

Browser support

The specs described here are really new, infact they are just candidate or proposed recommendations, so we cannot expect full support across all browsers. IE10 both on Windows 8 and Windows Phone 8 is the first browser to support them. Yes, you read that right, IE10 is ahead of the pack on this one. Chrome is expected to implement it soon and Mozilla is a member of the Web Perf WG, so we can expect it there too.

Create HAR

While researching on performance I started looking into a packaging format called HAR. Both Firebug on Chrome’s developer tools can generate a HAR (basically a JSON file) of your session. The good thing about HAR is that you can store it and analyse it later and see the performance of one or more pages. For your production Web site you probably want to build something scalable, that will read a lot of data and aggregate it, you might want to do it in-house or you might want to use a third-party service. On the other hand, while you are creating a new Web app you might want to monitor specific pages or specific parts of your app and you want to do it with high level of detail. For this use case I thought that creating a HAR and posting it to a remote server could be a nice feature. What you do with the HAR posted is up to you, in my case I simply view it with an online tool called HAR online Viewer.

So let’s start from the basics: HAR 1.2 spec. What we want to do is build a JSON with all the information about the current page and store it somewhere. In my example I do this for a single page, you might want to use a session or store it somewhere (localStorage?) and then make a single POST with all the information you need.

My example is composed of two parts, the HTML and JavaScript that runs in the client and a very simple PHP script on the server that stores the file locally.

The markup

For this sort of example the markup you need is very limited, but to make it more similar to a regular Web page I added a stylesheet, an image, some text and a couple of third-party scripts (Twitter and Facebook buttons and Google analytics). Three links are available, “Generate HAR” which will fire the script to generate a HAR file and POST it to the server. The script will also check that your browser implements the Performance Timeline and Resource Timing specs, if not it will do nothing. Assuming your browser can provide the needed information, clicking on “See the dynamically generated HAR on HAR viewer” will show you the HAR of your current session. Please note that although the page works on modern mobile browsers it is designed for a big screen. If you are using a browser that does not implement the needed API’s you can click on “See a HAR file generated from a previous session with a Lumia 920 running IE10 Mobile on HAR viewer” and see a session I recorded previously.

The scripts

For this example I use two scripts, I use PPK’s browser detect and a script I created.
My script is very simple in concept, there are a number of values that I must provide for a valid HAR 1.2 JSON and I read them from either the performance object or from the window and document objects.

generateHAR() is where it all starts, I collect some basic details about the browser, the page title and initial timings.
fillEntries() is where most of the work happens. Although simple in concept there are few things to be noted. The fields that HAR supports and those provided by the Resource Timing API don’t match exactly so I had to make some adjustments. For example HAR has a field called “blocked” that represents the time the browser waited for an available slot, but this information is not available in the Resource Timing API. Similarly the Resource Timing API exposes information about redirects that are not part of HAR. I tried to bundle together what made sense.
HAR requires integer numbers, so I round all values. Because of this the generated HAR occasionally failed to validate as the numbers did not add up exactly to the total time. From my tests the rounding occasionally generated an error of 1 or 2ms, so I decided to use the “wait” field for which I had no data anyway. When the rounding makes the sum of the intervals smaller than the total I set the “wait” field to the difference in time, when it’s smaller I actually decrease the total time (I know, dirty). Here is the code:

twait = Math.round(rl[i].duration) - tdns - tconnect - tsend - treceive ;
ttime = Math.round(rl[i].duration);
if (twait<0) {
 ttime = ttime-twait;
 twait=0;
}

Probably the most interesting thing I learnt while writing the script is that some of the timings will be empty (this is actually by spec!). Imagine the previous example where I was displaying data about a page; the second image is loaded dynamically, fired by a user event (clicking the button), if you try to calculate the time it took to load, fields like responseEnd will be empty until the image has finished loading. Similarly, if there were no redirects, redirectStart and redirectEnd will be empty, so if you try to make operations with those you will get unexpected results. What is ALWAYS set is startTime, so I added a few extra checks to make sure the starting time in all my subtractions is set:

nextFrame = rl[i].startTime;
tdns = (rl[i].domainLookupEnd) ? Math.round(rl[i].domainLookupEnd-rl[i].startTime) : 0;
if (tdns) nextFrame = rl[i].connectStart;
tconnect = (rl[i].connectEnd) ? Math.round(rl[i].connectEnd-nextFrame) : 0;
if (tconnect) nextFrame = rl[i].requestStart;
tsend = (rl[i].responseStart) ? Math.round(rl[i].responseStart-nextFrame) : 0;
if (tsend) nextFrame = rl[i].responseStart;
treceive = (rl[i].responseEnd) ? Math.round(rl[i].responseEnd-nextFrame) : 0;

Internet Explorer also supports the property fileSize for documents and images. I implemented the support for it and kept the default value of -1 for the other browsers.

fileSize = -1;
httpStatus = "200";
httpText = "OK";
if (rl[i].name == "document") {
 fileSize = document.fileSize
}
// a dumb check to see if fileSize is supported, of course you 
// MUST have at least 1 image in your page
if (!!imgs[0].fileSize && rl[i].initiatorType == "img") {
 for (ii=0;ii<imgs.length;ii++) {
  if (imgs[ii].src == rl[i].name) {
   // it looks like fileSize is -1 when image is loaded from cache.
   // Not documented
   if (imgs[ii].fileSize == -1) {
    fileSize = 0;
    httpStatus = "304";
    httpText = "Not Modified";
   } else {
    fileSize = imgs[ii].fileSize;
   }
  }
 }
}

Once all is ready I generate a JSON and POST it to the server. All sessions are stored on the server, but there is a cronjob that removes them after 7 days.

If you have a computer or a device with IE10 you can try this at http://logme.mobi/jsnt/har.html or you can look at a previous session that I created with a Lumia 920. The source code is available on GitHub.

TODO

I noticed that the HTML page has a total duration that is equal to loading the entire page and all its resources. I am investigating how to get the actual data for the HTML alone. If I can, I will fix it later.

Performance and RUM tools

There are a number of tools out there that already take care of measuring the page performance on the client. Google Analytics probably gives the simplest overview and for free. If you are looking for something more advanced you should check out a couple of open source projects: Boomerang and Web Episodes are both framework for timing web pages. They offer different degrees and techniques to measure page performance, but they are both a lot more advanced than my examples and they cater for all the browsers in the market today. If you like the idea, but want to rely on a professional to take care of this, I suggest you check out lognormal (developers of Boomerang) and Torbit (who use Steve Souders’ Web Episodes).

Resources

• W3C’s Web performance WG publications (see all the specs the group is working on and their status)
  • Navigation Timing (PR as of 26 Jul 2012)
  • High Resolution Time (PR as of 23 Oct 2012)
  • Resource Timing (CR as of May 2012)
• Navigation Timing documentation on MSDN
• Performance Timeline and Resource Timing example source code
• Boomerang library
• Web Episodes library
• HAR online Viewer and other HAR adopters
• HAR generator source code

This article is a cross-post from Measuring the speed of resource loading with JavaScript and HTML5.