The first meeting I had about the London Olympics was June 22nd 2010. It was with a colleague from RTÉ Television, who was about to head to an EBU (European Broadcasting Union) meeting in London to discuss broadcast television feeds, and we discussed implications for RTÉ’s online requirements. In September 2010 the first large round table meetings kicked off, headed up by RTÉ Sport but with people there representing all areas of RTÉ. I almost felt late to the party, as the discussions going on even at that stage almost 2 years out were so detailed that it was obvious this was an extraordinarily well planned endeavour.
In 2008 for Beijing we had access to 5 “raw” feeds from a selection of 12 which the EBU made available to RTÉ, and we put as much content on those 5 streams as we could. This time the message was clear from the outset. We must get all 12 incoming EBU satellite feeds online. And we must have them available to stream in High Definition. At that stage RTÉ2 hadn’t even moved to HD in broadcast, never mind online, so it seemed a tall order.
We looked into buying all of the equipment necessary to take in the feeds and encode them ourselves, however a significant sum of money was involved that couldn’t be justified for a 3 week event. The next option was to try and find a partner who had access to the same 12 EBU feeds, and a bank of video encoders in place to create the necessary web streams. In the end we partnered with iStreamPlanet who have a specialist facility in London with broadcast feed access, and Haivision who provide class leading HD stream encoding solutions.
It’s not enough to simply make 12 video streams available, we also needed the web site to integrate with the EBU’s schedule of what’s on when, so our users can clearly see what sports they can watch live at any time. Haivision put together a solution which ingested the EBU schedule data feeds, and matched them up with real time thumbnail screenshots of the stream encodes, so users are presented with a video wall showing what’s available. You can see this in action at http://www.rte.ie/live/olympics/. The other positive feature of this page is that it will work for iPhone/iPad users.
We’ve also made RTE2 available in HD everywhere you can stream it on the web site. We have been testing this for a while, and some of you may have tried it out during Euro 2012. We had some great feedback about the quality then, so we were keen to make sure the Olympics benefitted too.
So, how exactly is all this stuff wired up?
RTÉ Olympics Video Streaming Overview
As I said above there are two main aspects to this, the video streaming, and the scheduling. The schedule data is relatively straightforward; both RTÉ and Haivision consume XML feeds from the EBU, which we load into our respective scheduling databases. On RTÉ’s side, this schedule data is used to populate various areas of the web site where you will see a list of what’s coming up next. On Haivision’s side, this is used to populate the web page where you can view the 12 EBU raw feeds, and served from Amazon EC2 Cloud servers.
On the video side of things, RTÉ are encoding the RTÉ2 HD streams, while Haivision are using their encoders in iStreamPlanet London to produce a high quality encode of the 12 EBU Satellite streams. This high quality encode is sent to the Haivision Hyperstream Cloud, which transcodes it in real time into 6 different bitrates (ranging from 300K up to 2.2Mbps 720p). From there Haivision send these 72 encodes (12 channels at 6 bitrates each) onto the Akamai HD Network platform. This is a global streaming platform which provides high capacity delivery of the streams to both web and iOS users. RTÉ are also sending an RTE2 encode onto this platform so it can be viewed by iOS users too.
RTÉ are streaming RTÉ2 HD on the web using our in house streaming infrastructure. This is made up of Haivision Kulabyte HD Encoders which send streams onto Adobe Flash Media Servers (FMS). These FMS servers take the incoming streams and convert them into HTTP fragmented streams for playout. This is done for both Web using Adobe’s HDS protocol, and for RTÉ2 in RTÉ Player on iOS using Apple’s HLS protocol. A farm of Varnish HTTP Servers provides a high capacity caching layer at the edge of our network. If we reach capacity on those servers, or on our INEX links, we have the capacity to offload traffic to a third party CDN (in this case, Level3).
Always conscious of the need for a backup plan, both RTÉ and Haivision are also sending the full suite of encodes onto a set of Amazon EC2 FMS servers, so in the unlikely event of any issues with Akamai, we have a completely separate path. In the event there are any problems with the Haivision encoders in iStreamPlanet, RTÉ have the ability to encode up to 5 of the EBU feeds using existing encoding infrastructure. If the EBU feeds themselves run into trouble, all bets are off and viewers of TV stations across Europe will be missing out! (In all seriousness, the EBU and all host broadcasting systems will have multiple layers of redundancy for this event).
A lot of effort has gone into giving Irish viewers the best possible experience of the London Games. Sometimes in spite of detailed planning, layers of redundancy, backup plans etc., there are bumps in the road. And unfortunately we haven’t been able to avoid some bumps.
Some of you may have experienced issues on our page which gave access to the 12 EBU Feeds. You will have seen a spinning circle with a “Live Streams Loading” message, even though they never load. Very frustrating, we agree, and we’d like to apologise to any users that have been unable to access the streams for reasons beyond their control. An unusual and hard-to-recreate problem was causing this, and we worked tirelessly through yesterday to pin down the problem and resolve it.
In the end, problems with the Schedule Service hosted on Amazon’s EC2 Cloud in Dublin turned out to be the culprit. Simply migrating those particuar EC2 instances to Amazon EC2 instances in the USA resolved it. We are still not sure why, and our investigations continue. The most important thing for us is to get the service working and stable for as many people as possible. Some things are beyond our control (performance of users’ PCs or Broadband), but where there are issues on our end be assured that we do acknowledge it and do everything we can to resolve.
We can already see that thousands of you are enjoying the streams at any given time. We hope this continues and that we can give you streams of some very memorable moments for Team Ireland. Feedback both good and bad is always welcome, you can comment here or find me on twitter @Jonathan_RTE_ie.
Excellent post sir – very informative and interesting.
Thanks very much Mr. Lundberg! Very interesting!
Very good post.