Best Practices for High-Availability Streaming at Scale
How do you maintain five-nines uptime for tentpole event streams reaching hundreds of thousands or millions of users? Steve Nathans-Kelly, VP and Editor-in-Chief of Streaming Media and Video Publishing Director at Information Today Inc., discusses this topic with Corey Smith, Sr. Director, Advanced Production Technology, CBS Sports Digital, Paramount, Joshua Johnson, Sr. Director, Solution Architects, EdgeNext, and John Petrocelli, CEO/Founder, Bulldog DM, in this clip from Streaming Media Connect 2023.
Nathans-Kelly begins by asking Corey Smith, “What would you say in your experience are the keys to creating high availability service with large-scale streams and tentpole events?”
Smith emphasizes that developing detailed customer analytics is a major key for providing seamless high-availability streaming at scale. “You have to evolve your analytics to figure out the customer experience,” he says. “Because if you don't understand what the customer's going through, you can never improve, right? If you never analyze the data, your customer base is constantly in this turmoil of, ‘Should I even bother with this thing anymore?’ Like if I go to a streaming service for live concerts and it's constantly falling over, why would I go back? Why would I spend money to go watch a streaming concert when I could easily go down the street and watch it at Five Point Amphitheater here in Orange County or go somewhere else.”
“I agree,” Johnson says. He points out that pulling real-time data is especially valuable. “It used to be that the data was so slow to get to us that acting on it was always detrimental. But now, actually, it is literally being fed in, and we're being able to make dynamic changes and choices as the situation goes on. But at the same time, if you look at the foundation of where you're at and the people…actually behind the scenes, that's key to it. Because if you don't have that foundational piece and they're not solid enough to actually provide the infrastructure or the inbound, then it doesn't matter how good the end is. You're not getting the initial piece in a qualified manner in any type of quantity [or] any type of quality.”
Smith says, “Oh, a hundred percent.” He also highlights that a wide view of audience location is essential to choosing the correct providers in the first place for large-scale streaming events. For the continental US and North America, he says, “You're looking at the typical providers, Akamai Limelight, etc. if you're going to do your CDN distribution. “But what do you do for a global international audience? What partnerships do you make there? How is the client telemetry influencing making decisions on constantly connecting the customer to the best stream possible? It's really the things around streaming delivery and load balancing across multiple CDM providers in different areas of the world that really hasn't changed much, from the three-tier architecture model of how an application is built in a data center somewhere…you have a load balancer. You have to spread out your application servers, you have to spread your middle-tier business logic, and your databases. It's basically the same thing: origin, origin, shield, edge…”
Johnson concurs with Smith about the importance of knowing the dominant players in a specific geographic region. “Which provider is the dominant player in that general area, who's got the infrastructure?” he says. “And are you actually relying upon them, or are you relying upon them and their partners, and do you actually know who their partners are? And then it becomes a question [of] do you see that partner as a value?”
Smith notes that, “A lot of those partnerships are only as good as their peering agreements, so at some point, it's still a business conversation on paper. Theoretically, you can get bits to anybody in the world. It's how you best optimize those routes.”
John Petrocelli says that while all of these aforementioned factors are crucial, many players who initially swept into the live music streaming realm during the pandemic were ill-prepared for the various technical hurdles that come with international-scale live streaming events. “I think about 100 to 200 companies went out and got venture funding and kind of ran into that market thinking, ‘Oh, this is no problem, I can pull this off,’” he says. “But what we saw was this is not an easy thing to do. And because you got some funding, I don't think you're going to figure it out in 90 days or even six months. And people weren't contemplating things like redundancy and serving to, first of all a big domestic audience, but potentially given who the talent was, it could be a pretty massive international audience. And it was a rude awakening for a lot of these companies.”
Petrocelli says that this reckoning ultimately was a test for who could be adaptable and wide-viewed enough to learn from mistakes and adjust quickly, and that this period served as a positive catalyst for improved tech and growth in the live streaming sphere. “I think that's consolidated back to reality, now that the people remaining in that space kind of understand the undertaking and what a best practice approach is to pull the experience off, but then to do it in real-time, and to scale it to a potentially pretty significant simultaneous audience.”
Learn more about high-availability streaming at scale at Streaming Media East 2023.