Challenge - In 2020, during the Covid-19 pandemic, demand for 8x8’s encrypted video conferencing service increased exponentially worldwide – jumping from 20 virtual servers to 8,000 in a matter of weeks. Although the work an 8x8 video router does is simple enough to explain, it’s actually quite CPU-intensive.
Scaling this system out meant deploying more servers in the cloud. 8x8 servers were suffering through the same number of network performance bottlenecks in a matter of hours as their engineers would have expected to have dealt with over the course of an entire year.
“Usually when something goes wrong, a machine would get overwhelmed, there would be too much demand on the CPU, and there would be not enough work done to take packets out of queues as they arrived,” explained Emil Ivov of 8x8. The result of these issues was increased latency that became visible to users and, more urgently, a huge increase in cloud service cost.
Solution - In order to bring costs down, 8x8 invested in Oracle Cloud Infrastructure (OCI). OCI offered 8x8 a less expensive Arm64 OCI instance powered by Ampere® Altra® CPUs. With little time to spare, 8x8 took a gamble and began refactoring Jitsi for the move to Arm64 microarchitecture — a move it feared would be arduous and ridden with incidents. Most of the critical code for Jitsi is written in Java, requiring JNI interfaces to native machine-code encryption libraries and other libraries that needed to be recompiled.
Fortunately for 8x8, due to already existing Arm64 libraries, relocating all of its Jitsi code to Arm64 required minimum work from its developers, all of which was completed in one day. In addition, automation scripts for launching new virtual machine instances had to be re-tuned for Arm64 instances — equating to a few more days.
Results - After relocating to Ampere-based Arm64 instances, 8x8 met its goal of Jitsi meeting or exceeding a P95 of 1 ms. This meant everyday performance should see packet transactions consuming less than one millisecond and a P99 of 5 ms.
From a cost perspective, OCI’s Arm64 instances were up to 30 percent less expensive per instance than the instances that they were using before. On top of that, the Ampere-based Arm64 instances delivered up to 30 percent greater performance, again according to Ivov — “a very substantial improvement.” Each Ampere instance was doing 20-30% more work, at the same size, compared to before the migration. Said, Ivov, “There is no reticence to adopting Ampere instances. They are always the first option we look at.”
Developer Story
After COVID increased demand for online communication tools to unfathomed extremes, one of the world’s biggest open-source telecommunications projects moved its services to Ampere-powered Arm64 instances on OCI. That decision is paying big dividends.
In 2020, when the COVID-19 pandemic transformed the culture of work worldwide, video conferencing solutions provider 8x8 experienced colossal traffic spikes for its encrypted video communication service.
Before the start of the pandemic, video routers for 8x8 videoconferencing services ran on 20 virtual servers. One week later, they were consuming 80 instances. Six weeks later, that number grew to 8,000. With the global economy shaken to its core, 8x8 found itself suddenly serving as a global telecommunications backbone.
At the heart of 8x8’s video conferencing services was an open-source project built on the scaffolding of a University of Strasbourg student project called SIP Communicator. It had since been rechristened Jitsi, after the Bulgarian word for “wires.” Jitsi is now absorbing enough energy-consuming cloud computing resources to power a small city.
Anatomy of an Overload
Although the work an 8x8 video router does is simple enough to explain, it’s actually quite CPU-intensive. Like a telephone switchboard, a server reads data packets from one socket, decrypts them, encrypts them again, and copies the encrypted packets to other sockets to be forwarded. Scaling this system out meant deploying more servers in the cloud. That seemed like a simple enough scalability plan, at least at the outset.
“Every week, we were doubling the infrastructure necessary [to host Jitsi],” said Emil Ivov, Jitsi’s creator and Vice President of Product at 8x8. “And it’s crazy how fast things grow when things are exponential.”
8x8 servers were suffering through the same number of network performance bottlenecks in a matter of hours as their engineers would have expected to have dealt with over the course of an entire year. Ivov likened the feeling to finding oneself cast in a submarine combat movie, where all the depth charges come at once and all the valves in the vessel’s bridge start exploding and spewing water.
“Usually when something goes wrong, a machine would get overwhelmed, there would be too much demand on the CPU, and there would be not enough work done to take packets out of queues as they arrived,” explained Ivov.
“These queues would grow longer. It takes time for a packet to leave that queue, be processed, be put on an outbound queue, and be sent. That time increases, and that increase is visible to users.
That time manifests itself as latency [and then] packet loss, because as you start sending too many packets down the road, you may overwhelm the queues of various devices on the road to your end device. You could see bad video or noise in your audio, interruptions. This becomes very visible not only in metrics graphs but also for users.”
One moment, the HAProxy load balancer would encounter thread delegation deadlocks. Then, because HAProxy relies on syslog to generate logs rather than write log files for itself, the log system would become overloaded.
But the technological challenges weren’t even the biggest problem. “There was a huge financial challenge that came into it. Our bill grew two orders of magnitude, getting into millions a month.”
Payoff at the 99th Percentile
8x8 needed to bring its costs down immediately. First, it invested in Oracle Cloud Infrastructure (OCI). Next, its developers came up with optimizations that enabled workloads to consume fewer OCI server instances. Following that, Oracle approached 8x8 with the news that it was launching a less expensive Arm64 OCI instance powered by Ampere® Altra® CPUs.
With little time to spare, 8x8 took a gamble and began refactoring Jitsi for the move to a completely different microarchitecture — a move it feared would be arduous and ridden with incidents. Most of the critical code for Jitsi is written in Java, requiring JNI interfaces to native machine-code encryption libraries and other libraries that needed to be recompiled. The libraries themselves, however, already had Arm64 versions.
“It’s an incredibly well-supported architecture,” noted 8x8’s Ivov. “Every meaningful project out there already has binaries for Arm64, so we’ve never had any issue with that.”
According to Ivov’s estimate, OCI’s Arm64 instances were up to 30 percent less expensive per instance than the instances that they were using before. On top of that, the Ampere-based Arm64 instances delivered up to 30 percent greater performance, again according to Ivov — “a very substantial improvement.” Each Ampere instance was doing 20-30% more work, at the same size, compared to before the migration.
Latency is measured in telecommunications networks by stacking the response times for all transactions together in a graph. The metric reflecting the expected average latency value for the 95th percentile of all transactions is referred to as the P95, which preferably should be very low.
At the 99th percentile (P99), average latency estimates are expected to be higher, because the four percent of transactions between the 95th and 99th percentiles will likely include slower transactions. After relocating to Ampere-based Arm64 instances, 8x8 met its goal of Jitsi meeting or exceeding a P95 of 1 ms. This meant everyday performance should see packet transactions consuming less than one millisecond and a P99 of 5 ms.
Relocating all of Jitsi to Arm64 required minimum work from its developers, all of which was completed in one day. In addition, automation scripts for launching new virtual machine instances had to be re-tuned for Arm64 instances — equating to a few more days. Jitsi engineers reported no serious issues adapting their existing monitoring tools for collecting performance metrics for Arm64, nor for reconfiguring orchestration platforms.
A low cost, easy win
8x8’s Emil Ivov admits he and his team had some trepidation about what they felt would be an exodus from x86 to Ampere. Would Jitsi suffer from performance hits, and would the cost of mitigating those hits exceed whatever savings they’d reaped from running on lower-cost platforms? Would 8x8 find itself split down the middle, with the communications platform hosted by a completely different architecture from the CX platform?
“At first, I thought, ‘Ah, gosh, who knows what exotic thing that is,’ “ remarked Ivov, “and how many things are going to not run there on this Arm64 base architecture. I was scared, that unforeseen things would happen. But then paying less money is a great incentive to go and try things.”
To their surprise, the entire itinerary for migrating Jitsi from x86 to Arm64 using Ampere-based OCI consumed just two weeks. The next step was the shift from staging to production, and the results exceeded everyone’s expectations.
“We turned it on, and machines that were roughly 20% to 30% cheaper were giving us 20 to 30% better performance,” said Ivov. Asked what 8x8’s plans are for moving other platforms to Ampere and OCI, Ivov declared it’s the easiest path to cost reduction. “There is no reticence to adopting Ampere instances,” he said. “They are always the first option we look at.”