Testing times for APIs

It's rare these days to build a web application without some reliance on a third party hosted tool, service or API. This can have a profound affect on the reliability of your application, because no matter how stable and robust your own code and hosting environment is, it is relying on 'black box' services. The reliability of these may be good, but you have no way of knowing whether that's good design and planning, or simply luck.

Say your application A is loading data from web service B, authenticating users with web service C, and integrating advertising from ad network D. In a random month, B has 13 minutes of downtime, C has an hour, and D has 42 minutes. Assuming your own app doesn't suffer any internal downtime, and that these periods of downtime don't overlap, your own app is going to have, in total, 1 hour 55 minutes of downtime, more than any of the individual misbehaving services.

In fact, recently when I was building a site that integrated ads using OpenX hosted, they had some problems and the forums started filling up with panic stricken messages such as

None of my pages are loading! When will OpenX fix this?

One reason that third party APIs often cause problems such as this, is that when you build your website, the APIs are typically working properly, and you simply don't include outages in the APIs in your test scenarios. So you unwittingly introduce a dependency on the API that you might, if you thought about it, be able to avoid.

I've created a tool on Google App Engine that I hope might help. It allows you to simulate a range of misbehaving API scenarios, and test the ability of your application to be resilient to these.

For example:

http://badapi.trib.tv/req?wait=20

Will take 20 seconds to return output. Any integer between 1 and 30 will work.

http://badapi.trib.tv/req?resp=500

Will return a 500 Internal Server Error (any valid HTTP response code will work). If no number is given, returns a 200.

http://badapi.trib.tv/req?op=json1

Returns standard test JSON output number 1 (see index page for full list of test outputs, and ability to define your own). If not specified, returns the string 'OK'.

You can combine these directives to produce a more tailored response, eg:

http://badapi.trib.tv/req?resp=500&wait=2&op=json1&ct=js&cs=utf

This will wait 2 seconds, then give you a 500 Internal Server Error response containing standard JSON output number 1, served as text/javascript and advertised as UTF-8 encoding.

I hope others also find this useful. Let me know if you have any suggestions.

Browser-side includes (BSI)

I recently delivered a presentation to PHP London about scaling web applications. One of the most interesting things to come out of it was talk of Edge-side includes, which is a technology invented and implemented by the content delivery network Akamai to enable content publishers to set different expiry times on different portions of their web content.

For example, if you have a page with a 'hot news' section that changes every few minutes, but the rest of the page remains static for periods of days or weeks, then in order to cache the page as a whole at the CDN level, you would need to set an expiry time of just a few minutes. You then suffer an unnecessarily high number of requests from the CDN for your page.

The problem is, we don't use Akamai at Assanka, we use Edgecast, who don't support ESI, at least not yet. So how can you mitigate this problem? There are other levels of caching and includes that you could use to avoid having to regenerate your entire page in order to serve the Hot News section - on your own network you could implement a cache that supports ESI, like Varnish or Squid, so that even though you're still receiving an annoyingly large number of requests, they are not making it all the way to your application servers (except the requests for hot news). Or you could do it purely at the application level - for example if you have a Wordpress blog that implements WP-cache, you could set the cache time to an hour, and then hook in a process that runs after the cache lookup to add the Hot News section uncached. In Wordpress, this is actually quite a common thing to do when you want to print a login status message like 'Welcome Andrew'.

So how about browser side includes?

For those of us who don't have Akamai's CDN, and still want the benefits of those really long expiry times, you can always use JavaScript for that Hot News section:

<script type='text/javascript' src='/hotnews'></script>

The hotnews script then outputs nothing more complicated than this:

document.write('Hot news section html');

The key to this method is that the main page is served with Cache-Control and Expires headers that instruct caches such as CDNs, ISP caches and the end user's browser cache, to consider the page's content valid for a long time, say an hour (could be a lot longer):

Expires: Sat, 07 Mar 2009 13:50:57 GMT
Cache-Control: max-age=3600, must-revalidate, public

Whereas the hotnews script returns its javascript with headers that make it expire very quickly, say after 2 minutes:

Expires: Sat, 07 Mar 2009 12:52:57 GMT
Cache-Control: max-age=120, must-revalidate, public

Or perhaps make the JS completely uncachable (if it contains information personal to the end-user's session, for example). The effect is that while the main page will remain cached for the hour, the section of hot news will update if it becomes more than 2 mins old.

I've decided to call this browser side includes. It's not a new technique - ad networks have been doing it for years, and more recently it's been used for blog badges and other embeddable widgets. There are two big problems with BSI:

• Search engines don't load linked resources, so they won't see the content loaded by the include and people will not be able to find your site using any terms contained in your hot news content
• Browsers load SCRIPT includes synchronously, so loading of your page will stall while the browser waits for a response from the hotnews script.

Dealing with each of these in turn, first, if the included content is private - session based and personal to the user, you don't want a search engine to see it anyway. If it's public data that you just want to serve with a different expiry time to the rest of the page, then just include a version of it with the page itself, set it to be hidden with CSS, and then replace it with your BSI. Search crawlers will then see it - it's out of date but good enough for Google - while end users will see the latest data, courtesy of the BSI.

Of course this isn't ideal. It adds to your bandwidth. Ideally, we could add a behaviour attribute to our script includes that makes search crawlers download them as includes:

<script type='text/javscript' bsi='include' src='/hotnews'></script>

But clearly I'm not in a position to dictate standards to search engines. :-)

The other issue is the synchronous loading. We can get around this by lazy loading, which simply involves moving the script to the bottom of the document, and where you want the content to be included simply place a placeholder DIV. The hotnews script on your server then outputs:

document.getElementById('placeholderdiv').innerHTML = 'HTML of hot news content';

This replaces one problem with another - namely that the page will fully render before the additional content is loaded, so to avoid the layout 'rejigging' itself when your include is loaded, it's worth using this technique only for content that has a fixed size.

A good rule of thumb is that the closer to the end user you can cache content, the cheaper that content is to deliver in volume. That's the mantra of most CDN sales departments. Browser side includes are another tool in that armory.

Day 15

Last day in South Africa. Nick gets up early to do the final game drive with the dutch family and Christina and her mother, but since I will be spending 6 hours driving today I give the game drive a miss and get some extra sleep. Breakfast at 9:30 is the usual 4-course affair, followed by a dash to check out by 11 so our tent can be cleaned for the next guests. The Gualaguala guide suggests that we tip a total of about £30 a day, which seems mightily excessive to us, so we go with about £10 a day, split between the ranger, tracker and general staff.

At breakfast Ann comes to say goodbye and we mention that Paul recommended Gualaguala to us. She says she's a regular visitor to the UK and was 'finished in Aylesbury'. As we're settling up, Nadia (the lodge manager) offers us some packed lunches for the journey, which seem a far better idea than our current plan of finding somewhere to stop on the way.

For the last hour, we sit by the pool and relax until we have to go - which Nick and I have negotiated to be 12:30, so that we don't miss the flight (Nick's worry) or get to the airport hours and hours before the flight (my worry). It's going to be a long journey, but should be relatively fast, as it's highway all the way, and we have now learned the art of overtaking, South African style.

The journey is scenic, particularly at the beginning, and mostly uneventful, but we do have to keep watching out for potholes. The high quality road surface tempts you to think you're fine then suddenly there's a big pothole seemingly at random, which are particularly dangerous if you're driving at the highway speed limit. We only hit one, but the jolt sends our front left hub cap spinning across the road. Nick spots it first and we pull over to walk back for it. The verges are overgrown, but we find the hub cap within a few minutes, along with several others. It seems sensible to keep the cap in the car rather than reattach it, in case we come across another pothole.

We stop for fuel and to eat our packed lunches late afternoon with a hundred or so kilometers to go. Arriving at Johannesburg airport at about 6:30pm, the car is handed back to Avis, and we're on our way home.

Day 14

The morning is cloudy and cold as we gather in the lounge at 5:30am. Our second drive is a 'big five' drive, which means visiting a neighboring reserve, the 15,000 hectare Thornybush Private Reserve. The extra size (it's 32 times bigger than Gualaguala) means they have space for all the big five animals (lion, elephant, rhino, buffalo and leopard), but in such a massive space they can sometimes be difficult to find.

Our Thornybush land rover and guide pick us up from Gualaguala at 6am. The drive lasts about 3 hours, and we see four out of five - no leopard, which is the most difficult to find. Whenever we find the game, there are always other vehicles there as well, so the reserve seems busy, but well managed. We stop briefly for some tea, and then head back to Gualaguala.

We pull into Gualaguala at about 9am, where the other game drive is already back. Breakfast is at 9:30 in the treehouse. It consists of cereal with milk, then scrambled eggs, bacon and grilled tomatoes, then a fruit salad, followed by toast, then a cheeseboard and finally chocolate muffins. We spend a few hours reading in the tent, and at about 12:40 we go on the 'self-guided walk' shown in the Gualaguala guest handbook. It's an hour-long walk around a series of numbered trees, with a guide explaining what each tree is and what it's useful for. Unfortunately after about half an hour we have seen no numbered trees, and are starting to get the impression that we're on the wrong path. Then the path opens out and there are paths leading off everywhere. Nick decides to adopt the "let's try this way" approach to navigation. I'm consoled by the thought that this is only a 500 hectare reserve, and it has no predators in it. That doesn't mean it doesn't have snakes, biting insects and ants though.

We get hopelessly lost. We see some game in the distance - a herd of wilderbeast and some impala. Eventually we find a road, and disagree on which direction to take to get to the lodge. Finally, the lodge is found, having located zero out of twenty eight numbered trees. An incomplete success.

We've got an hour before high tea, so to the bewilderment of the South Africans, we jump in the pool. It is still just as 'refreshing' as it was yesterday.

The afternoon game drive, after a high tea of nut cake and fruit, is another 'big five' drive. This time we're with a group of dutch who seem to be avid birdwatchers, and we have to stop incessantly to take photos of and wonder over tiny birds that we can barely see. We do eventually see some buffalo and black and white rhino, but no elephant or lions.

We arrive for pre-dinner drinks to meet several new guests, including a mother and daughter from the UK, who are at the start of a garden route holiday ending in Cape Town in a couple of weeks. Dinner is fish with creamed brocolli and stir fry vegetables, followed by bread and butter pudding, and is excellent.

Day 13

We're up at 7am, and the power is still off. It comes back in time for breakfast, and with everything packed into the car we head off to the border.

At the border crossing, we get stamped out of Swaziland, and stamped (again) back into South Africa for the third time. We drive through customs with nothing to declare except a lot of Swazi dirt which is now attached to the car.

On the highway, we have to run the toll gates again. This time I manage to stop the car at the booth without stalling, but the money Nick hands over includes some Mozambican coins. This causes much confusion and I end up almost driving away without our change. We don't seem to be very good at the toll thing.

The rest of the morning is spent driving to Gwala Gwala, a game reserve on the western fringes of the Kruger National Park. At one point we realize we're driving in the wrong direction, but we get there in the end. We sign in at the gate and drive 7km within the reserve area, past numerous other reserves, until we reach the big GwalaGwala gate. We sign in again, and follow a dirt track to the edge of the actual reserve, and enter it by driving over a cattle grid. It's difficult to make much further progress, as there is a giraffe standing in the middle of the road. It moves away eventually, and we continue (slowly!) to follow the signs to 'reception'. It's another 2km before we actually reach the car park, and we've seen numerous antelope and some zebra by the time we arrive.

First impressions of Gwalaguala are that we'll probably survive. Our tent is equipped with a wooden deck floor, power, a huge standlone bathtub and a veranda, all raised up on a wooden platform built amoungst the trees. There is a bar/restuarant in a treehouse, a lounge area under a thatched roof beside the river, and a small swimming pool.

After the long drive I'm eyeing the pool longingly. After a quick change we get down there to find a Mancunian couple on honeymoon who are sitting in deckchairs, and a finnish woman in the pool. They are arguing over whether the pool is too cold, with the Brits (who are not in the water) saying it is, and the Finn (who IS in the water) saying it isn't. I dip a toe in and decide a slow entry is going to be too tortuous, so I take the plunge in one go. It's actually fine, particularly as the air is so warm.

We don't have long before 'high tea' (the lodge's late afternoon version of lunch), so we make it a quick swim. At High Tea, we meet Ann, one of the lodge's owners, who asks after Edd and forthrightly denounces him for not coming. She's friendly and very funny but I'm not sure I'd want to get into an argument with her.

The other guests comprise a gay couple, one half of which is a hilarious ex-canadian now living in Sweeden, a swiss amateur pilot who flew directly to Gualaguala (as in, he flew his plane into an airstrip walking distance from the lodge) travelling with a South African co-pilot guide, and the Manchunians.

The first game drive is straight after high tea, and is a local 'Gualaguala' drive on Gualaguala's own 500 hectare reserve. They don't have any of the 'big five' here, but because it's a small reserve, it's easy to find loads of giraffe, zebra, antelope, warthog, guinea fowl, and water buck. It's a long drive but we can barely go 100 yards without seeing some other animal. We stop at the lake for the sunset, where they have a hide - a wooden hut built n stilts on the edge of the waterhole. After sundown it's back in the land rover for some night driving. We find some more giraffe - now sitting down, but fail to find any hyena.

The South African pilot suddenly pipes up "Hey, back up we got sex going on here". Sure enough there's a giraffe making concerted efforts to mate with a female, but she doesn't seem to be in the mood.

Dinner is a brie, and a very smoky one, but the food is delicious - five meats to choose from, potato salad, and sadza/pap. We have to be up at 5am for the morning game drive so we make it an early night.

Day 12

Breakfast is included, which is good as it's a full english, and relatively good, especially compared to dinner yesterday. It's served in silence by our semi-mute host. We decide to stay another night but to get dinner elsewhere.

We drive into town for provisions and find a 'U-Save' supermarket, where everything is half the price of Mozambique. They also have a fridge full of bottled water but it's 'flavoured' water, and Nick is unimpressed. He seems to get through several gallons a day.

Thirty kilometres south of Pigg's Peak is the M___________ Nature Reserve. The road to get there covers some spectacular scenery, and frequently passes cattle simply standing still by the side of the road. You'll round a bend and see a cow just stood in the centre of the road, absolutely oblivious to the obstruction it's causing.

We come to a bridge curving over a picturesque valley, and a man is standing in the road waving at us. I slow down, never sure in these situations whether the frantic waving means 'Help! My children are trapped in an abandoned mine shaft', or 'Please visit my shop'. To be honest it has always been the latter so far. In this case y slowing down is greeted by even more enthusiastic gesturing and pointing at the side of the road, where two small children, dressed in costumes covered in leaves, start bobbing up and down in a bizarre dance.

I drive around him. Nick refuses to take a photo on the grounds that they'll want paying for it.

At the other end of the bridge, another pair of identical performers await, this time with a less enthusiastic chaperone, and start bobbing again as we approach. Nick hasn't got his camera ready.

Getting into the park, we're on Africa time again. It involves making a hand drawn copy of the park map (which they've run out of), and signing the visitors register (last visitor yesterday afternoon). The roads within the park are ingenious, consisting of concrete ridges embedded in the grassland, making the road surface green and well camoflaged against the landscape. The weather is perfect, with a few white clouds in an otherwise clear sky.

Following Nick's map leads us to the first picnic spot, where there are fixed tables, toilets and the start of a marked trail. I decide to sit under a tree and chill out for a while, while Nick tackles the trail. Ninety minutes later he returns complaining that he has become unfit in the last two weeks.

We drive on to the next point on our map - a trailhead for four trails, which are marked with meaningless symbols like triangle, square, trapezium, parallel curves. On the way we suddenly spy an antelope of some kind standing on a hill. Nick spies his first opportunity to use the SLR for a wildlife shot, and starts the lengthy process of getting the camera bag out, extracting camera, strap around neck, turn camera on.... fortunately the antelope has clearly been to the Nick Ambrose school of wildlife photography, and stands absolutely still while he faffs. We follow the signs to the parallel curves trail, which is supposed to go past a waterfall. The trails are well marked with red pumpkin-like stones painted with symbols and icons in white. Following the trail for around half an hour, we start hearing water bubbling, and discover a stream. Nick, as ever, wants to dam it. Considering the environmental impact of excessive whimsical damming of rivers, and the lack of suitable building materials, we agree that a bridge would be a better idea. I sit in the shade to apply another layer of sunscreen while Nick finds a stick, lays it across the stream and declares his bridge open to traffic. Insect traffic, presumably.

The trail splits into the triangle trail and the curves trail, though the triangle trail is closed due to a Bald Ibis nesting. I'm impressed that the Swazi parks service are this attentive to the maintenance of the park and the wildlife. After all, if we're the first visitors today, then they're making a pittance from the entrance fees at £2 a go.

It takes another 40 minutes to get to the end of the trail, which meanders through open grassland and over and round several hills. The last few hundred metres are particularly steep, first up, and then down. Overall we've descended more than we've climbed, and yet we end up at the top of the waterfall. A French couple are already there, having overtaken us while Nick was planning and building his bridge.

On our way back, black clouds start congregating in a corner of the sky, and we head rumbling in the distance. We're still 20 mins from the car when the sky is mostly black overhead and the thunder is coming quite distinctly from all directions. The temperature drops ten degrees, and the wind picks up suddenly. This gradually gets worse and as we scramble over the top of the hill to see the car in front of us, the few drops of rain is starting to turn serious. We get a few spectacular shots of the storm, with the clear sunny weather clearly visible on the other side, before fleeing the park before it becomes impassable by a weedy two-wheel-drive hire car.

We get to the exit gate at about quarter to six (park closes at 6pm). For dinner we've decided to try the Hawene Lodge, an upmarket hotel village about 5km further along the road. We arrive to find an immaculate complex of 'beehive' cottages build in a combination of Swazi and western style, set around a circular thatched restaurant. We enter to find it deserted, and ask the barman if we can book for dinner. He tells us it's fully booked - a big disappointment considering the distance to the next decent restaurant. At least it's still early. We leave the - still deserted - restaurant and head for the exit, and are briefly acknowledged by a friendly but important-looking chap who asks us if we're guests. I explain that we had simply intended to come to dinner but had discovered that the restaurant is fully booked, and this causes him some confusion. He insists that it is not fully booked. Following him back into the restaurant the staff get a firm dressing down from the chap who is presumably the owner, and we sit in the bar area waiting for dinner to be served.

The bar area has a TV tuned to CNN, and in our 20 minutes of being back in touch with the 'real world' we discover that Hillary Clinton has NOT said what her role is in the new Obama administration. This is CNN's lead story for half an hour.

Nick disappears saying something about taking photos of the sunset, but I am reading the paper and not concentrating. He returns with some spectacular pictures and I'm kicking myself for not going out to see it.

Dinner is a 'brie' - African word for barbeque. Three different meats, 'pap' (which I know as Sadza from my time in Zimbabwe) and salad. It's fantastic, and cheaper than the dodgy dinner we had last night. The restaurant does fill up a bit more, but gets nowhere near full.

It's about 40km back to our hotel, where we find that the power is out. In fact it's hard to spot the hotel at all with no lights on. Since we're only going to bed anyway it doesn't matter much.

Day 11

A taxi takes us to the bus station for 6:30am. It's raining again, this time with more enthusiasm.

When we arrive there is a bus already there, and people milling around. We check in and wait. The bus leaves and another arrives, which turns out to be ours. we get our bags stowed and board in time to leave just after 7am.

The bus driver makes his way down the aisle, offering tea, coffee and 'juice', which turns out to be something pretty vile and not really the juice of anything fruit-like.

The journey is uneventful until about 8:30am when we get to the South African border. We all have to get off and go inside the Mozambican departures building to get our passports stamped, then walk about half a mile across no man's land to the actual border, where a very substantial sliding gate is open. We follow the pedestrian channel and enter the SA arrivals building where the whole process repeats itself. We already have an SA visa, and one entry and departure stamp (entry at Jo'burg airport, departure at Lanseria airport), so we just get another entry stamp.

Back on the bus.

Seems everyone made it into South Africa. Another two hours later, we're in Neilspruit. The only thing going for this outback SA town seems to be the motor industry, which is everywhere. It's like the South African Detroit. This seems promising since we want to rent a car. We are deposited by the bus at a petrol station on the corner of a busy intersection, and the sun is now shining strongly so make for some shade to apply sunscreen. An armoured car suddenly appears and two men with large guns get out and encircle a nearby cashpoint. One eyes us suspiciously and asks us to move away.

We go to the Europcar office, which turns out to be a desk in a hotel, and find it unmanned. The helpful receptionist phones a few companies for us and we end up with Avis, who also come and pick us up from the hotel.

Renting a car in Africa seems, like many administrative processes, to run on Africa time. It takes a full hour to complete all the paperwork, but at the end we're on the road at the wheel of a marvellous grey Toyota Yaris - not quite the same model I have at home, but close.

First priority is food. It may only be 1pm, but we've been up since 6am and only had bananas for breakfast. Nick spots a Wimpey on the highway, and I manage to successfully negotiate a dual carriageway to get us there in one piece. One very irritating difference between this Yaris and the one I drive at home is that the windscreen wiper and indicator controls are reversed. I curse this fact for about the fourth or fifth time as I turn the windscreen wipers on in order to turn left. Other drivers may not know my intentions but at least the windscreen is being cleaned regularly.

I haven't been to Wimpey for about 15 years. After a brief lunch of burgers, I reason that 15 years was about right, and could probably go another 15 years without visiting Wimpey again.

Finally at about 2pm, we're on the highway heading for Swaziland. It's back along the road towards Mozambique, and it takes about 90 minutes to get to the border. On the way we go through the toll gates that we came through in the other direction on the bus this morning. "I'm really getting to grips with the car" I declare as I roll to a gentle stop at the toll booth - and stall - having forgotten that I'm driving a manual. Seems it only takes a week driving an automatic to cause me to forget what the clutch is for.

We discuss the Swazi border and hope that it might be more efficient than the Moz/SA border we did earlier. Maybe even a drive-through operation.

We're waved through the first gate - a good sign, but then we're invited to park and enter the SA immigration building for our departure stamp. Back in the car we drive 100m and have to do the same again for Swaziland, including paying a £2 road tax for our car.

A billboard proclaims "Welcome to Swaziland". And we've arrived.

The first town on the road leading away from the border is Pigg's Peak, but before that we pass a sign for the Ph__________ Nature Reserve, which is on our map so we decide to stop for a visit. The turnoff is a dirt road that winds 2.5km though some spectacular scenery to the gates of the reserve, where we are charged £2 to enter.

A lodge sits at the entrance, with luxury cottages perched on the edge of views across the valley. We get a trail map from the reception guy and set off on one of the marked routes, through the densely wooded sides of the valley. Water runs everywhere, mainly through man-made channels next to the paths, and pours into natural ponds and eventually flows into the valley.

The trails are well marked. We reach the valley floor and a couple of bridges over the narrow parts of the river - only a couple of metres wide, but flowing fast and rough over huge piles of rocks.

We loop round back to the lodge. I want to know their prices for rooms but Nick prefers to press on into town. They're full anyway - seems there's a poker tournament down the road and the lodge is providing spillover accomodation.

As we drive on into town, we pass several large trucks that are carrying what appear to be sticks. They're not logs, by any means, and it's a jumbled collection of small branches piled high in a huge lorry. Just random bits of twig, by the lorryload. We thought Swiziland exported timber, but not sticks. Maybe it's for the pet market.

In town the first B&B is full, and in driving up to their intercom to establish this fact, I have stopped the car on a steep downward slope. The Yaris strains in reverse but simply can't make it back up to the road. We're considering buzzing again to ask them to open the gate simply so I can turn around and leave again, but I give it one last go with Nick out of the car, and manage to heave the Yaris back up to the road. There is, however, a definite whiff of burning rubber in the air. I make a mental note to not mention this to Avis.

The final option is the Highlands Inn, which the Lonely Planet advises to leave as a last resort. Unfortunately it is in fact the only one left. Luckily they have rooms, and they actually seem to be perfect. We get two double rooms for £30 in total. The proprietor is a very softly spoken african woman who talks so quietly you can barely hear what she's saying. The rooms are large, clean, have massses of storage and very comfy beds, so we're actually quite impressed, but the restaurant is awful, so we resolve to find somewhere better to eat tomorrow.

Nick looks up the times for breakfast, and notes that they start serving at 7:30am, which is "quite early". I disagree, pointing out that most hotels serve breakfast from 7am at the latest. "Ah yes", contends Nick, "But many don't serve breakfast at all". Right.