In the wake of Gogo’s media flights on its Boeing 737-500 testbed (the “Jimmy Ray”) to test its 2Ku system — which CEO Michael Small quipped was “extreme focus group testing” — it was clear that the generally positive real-world experience of 2Ku conflicted with some of the results from the frequently used Ookla speedtest, which itself was showing wildly different results for the same user, varying from under 0.5 Mbps to over 20 Mbps.
— Paul Thompson (@FlyingPhotog) November 12, 2015
Many of the results journalists experienced varied significantly as the aircraft flew across the US midwest with a load of passengers attempting to make the system fall over.
— Seth Miller (@WandrMe) November 12, 2015
A valuable piece of information can be gleaned from the latency results (measured by ping time). The range was 600-1200ms – not necessarily a relative constant performance but certainly useful information for management as it works to fine-tune the system.
— Jason Rabinowitz (@AirlineFlyer) November 12, 2015
The Ookla speedtest is used by many journalists, RGN contributors included, as a way to use a common tool to compare speeds. With well-known iOS and Android apps, as well as a Flash version for browsers — based on Ookla’s enterprise testing and benchmarking tools — it’s certainly a reasonable first take at speeds.
But as with any kind of industry benchmark, the Ookla speedtest does have limitations, and the frequency of its use (by the industry, users, and journalists) means that companies have an incentive to game the system by prioritising Ookla speedtest traffic. This is, again, common to any benchmarking tool, and RGN uses a number of factors (including real-world performance) when discussing how a particular system is working.
Following the 12 November flights, RGN put a number of the questions raised by the 2Ku Ookla speedtest results and the on-the-plane experience – which ranged from “great” to “inconsistent” among RGN contributors – to Gogo CEO Michael Small. Specifically, RGN asked whether there was a technical reason that the speedtest results journalists were seeing were very variable on the downstream speeds, from below 1 Mbps to over 14, while at the same time some were seeing promising results from real-world streaming [on the flight, some 50 devices pulled in close to 11 GBs of data]. Was there, for instance, some kind of prioritisation of traffic to video over the speedtest protocols?
Small stressed that Gogo specifically avoided any kind of traffic shaping or prioritization for the test flight. And he noted that there “was no prioritization of video over speedtests, but your question also points out one inherent flaw in speedtests. Flip that question around and you can see why we put almost no stock in speedtests. The speedtests themselves can be prioritized and gamed enough through prioritization.”
Small previously told RGN contributing editor Seth Miller flatly that data was capped at 25 Mbps to the plane to mirror a production network. Thus, he re-upped on what he deems a successful real-world experience on the test flight, and emphasised the global nature of the Ku satellite constellations 2Ku uses. “We understand speedtests are what journalists use as the standard to measure a network. But, with 2Ku, in real world scenarios, we’d expect passengers to be able to speedtest anything from a few Mbps to 20+Mbps anywhere in the world. They’ll be able to stream videos and enjoy multiple channels of live television through Gogo TV. In sum, they’ll be able to do pretty much what they can do on the ground and our airline partners will be able to customize the experience in any way they want.”
The iterative developments coming both in terms of the airborne terminal, the Ku satellite constellations that 2Ku uses, and the Gogo backend were also high on Small’s list of talking points. “Can it be configured in a way that will deliver even more? Yes. If the goal was to win the speedtest battle, we would have waited a few months when our next generation modem is on the Jimmy Ray and we could’ve shown journalists speedtests of over 70 Mbps, and even more when HTS satellites arrive next year. But, that’s a fool’s errand because we know from our experience it won’t scale that way globally.”
Back on the real-world experience, point, Small noted that “we are focused on the passenger experience and managing a network at global scale because we are confident very few passengers are spending their time in flight running Ookla speedtests every few minutes. We are not focused on or concerned with comparing speedtests on 2Ku vs. competitors. We know from our experience that 2Ku is capable of matching or beating the bandwidth of any technology solution on the market.” However, RGN notes that Small also did not address the “real world” inconsistencies in performance which were experienced.
As high-throughput Ku-band satellites come on stream, as Inmarsat’s GlobalXpress system is installed, and as the hybrid S-band system in Europe develops further, RGN will continue to monitor the performance of these (and existing) systems. Gogo’s 2Ku seems to have passed its first test — but the real test comes in revenue service.