Nov 22, 2021
Installing a Game Server on a Pacific island... How hard could it be?
As it turns out, it’s not easy.
First up, why would you put a server on a Pacific island, anyway?
Imagine this: you’re a gamer living in Sydney.
You want to play games with your friends Alice and Bob, who live in Singapore and Japan respectively — but there’s a problem:
Choosing a game server location that works for everyone is hard.
If you put a game server in someone’s home city, the other two players would have a delayed connection, and it wouldn’t be as fun for them.
So what’s the solution? Easy! Put a server on an island somewhere in the middle. That way, everyone can play together on a level playing field.
How hard could it be? That’s what we asked when Ben Cooper (whose idea it was) brought it up.
If the game plan worked, it could finally unlock cross-regional gaming in the Asia-Pacific (APAC) region.
As a team of gamers — we couldn’t resist.
The easiest part: choosing a Pacific island
First thing’s first, the island had to be well connected, and, when you open up a submarine cable map, one place in Asia-Pacific lights up like a Christmas tree:
A casual browse of SubmarineCableMap.com
The Pacific island of Guam.
On the surface, it looks like the perfect spot for an international server hub:
it’s highly connected
it’s a submarine cable hub for the regionit’s right in the middle of the Asia-Pacific region
so a perfect spot for a cross-regional server facilitylatency to nearby countries is excellent
internet packets can do a return trip to nearby countries faster than the blink of an eyeit’s a US territory
with a familiar legal environmentit’s a beautiful place to work
So why hasn’t it been done before?
We figured most companies hadn’t even thought about doing it. Those who knew it was possible, didn’t know there was a need for it.
And the people who needed it, didn’t know it was possible.
Plus, creating something new takes time and money.
So we figured we’d give it a shot.
Over the last 18 months, we’ve been building a pool of servers on the island of Guam. We’ve got the scars, the tears, and a high-performance cloud platform to prove it.
Here’s what we learned on the way.
Getting space & submarine connectivity is hard
The first thing we found was that there aren’t many places in Guam where you can put servers.
There are plenty of buildings, but most of them aren’t in reach of the cables that carry internet traffic across the Pacific.
After many dead-ends, we found a potential location in a new cable landing station that was being built.
And then we struck gold.
An undersea cable company, RTI, was finalising a new subsea cable which went from Japan to Australia (where we are) — with a stop-off halfway in the exact same building in Guam.
An RTI submarine cable coming ashore • Watch on YouTube
Even better, instead of only landing in Sydney (where cables on the east coast of Australia usually land), it was also landing in the Sunshine Coast, Queensland — our backyard.
Normally, subsea operators don’t deal with small startups (which we were).
But RTI were no normal company and Russ Matulich (CEO) was no normal leader. He loved the idea of what OneQode was doing and took the time to help us through the process of getting capacity on their cables.
Thanks to them, we were soon preparing to deploy our equipment in the Sunshine Coast, Tokyo and Guam to leverage our new capacity.
What does “preparing to deploy” mean anyway?
To turn the lights on at a location, there are a few things we need to do:
negotiate legal agreements
arrange data centre space
hire staff
get transit connectivity
(internet for data centres)organise peering
(connections to last-mile internet providers)order and set up equipment
routers, switches, servers, and DDoS protection hardware
All this means mountains of paperwork, jurisdictional legal headaches, language translations and endless email threads — before we even got to hiring staff.
There aren’t many people around
It’s hard to find qualified network engineers in Guam — and there’s good reason:
With a population of ~170,000, 3 internet providers, and only a handful of companies employing IT professionals, there aren’t a lot of technical specialists looking for work.
We put some feelers out and luckily managed to snag the excellent Matt Ziobro, an IT specialist (and keen pilot), to take charge of our operations in Guam. Without him, it’s unlikely our Guam project would have taken off the ground at all.
So — we now had a place to put servers and the beginnings of a team.
Next, we needed to order the equipment.
Shipping takes forever
When you decide to send a 100kg server to Guam, it’s most likely getting there via boat — which takes 25 days at the best of times.
So if something is broken or missing, that’s at least another month on the clock.
We found ourselves up late at night praying that our suppliers had triple-checked orders before shipping them out — and included the right cables. There aren’t many places on the island where you can buy replacements.
Decent shipping is like a lottery. Sometimes, the stars and your wallet align, and 2-day air freight can be available. It’s rare, but when it happens, it’s a great day.
Red tape plays a role, too. As a U.S. territory, the Jones Act applies. It effectively bans non-US vessels from carrying cargo to Guam. That means you can order an item from Japan, and if there’s no U.S. ship to take it to Guam, it’s probably going via Alaska.
Don’t ask.
Even when your shipment gets to Guam, tragedy can strike.
More than once, we’ve had large shipments arrive on the island only to be sent back because the delivery company didn’t think the data centre existed. Maybe they went to the beach instead of delivering our equipment.
One last thing — if you’re using anyone except USPS for delivery, there’s a 4% “Use tax” on the value of your delivery. At this point in the process, you just sigh and pay up.
Finally, your delivery has arrived — so you plug things in.
Electricity is expensive
On the island, electricity costs at least twice as much as it does in the United States.
Why so pricey? Like most things in Guam, the energy sector is at the mercy of the shipping industry too. With 90% of the power coming from fossil fuels and limited alternatives to choose from, electricity is precious.
Not only is it more expensive, but power distribution to the data centres we use is also precarious, so there are strict limits to how much power we can draw.
As if that’s not enough, because we play life on hard mode and operate high-performance servers, we’ve had to dance on the knife-edge between power and performance. That means choosing very specific components and making trade-offs where we can. It’s a delicate balancing act.
Can’t they just build more power plants?
Well, they did — but it blew up back in 2015.
How? Best to ask an islander.
The good news though, is that a new power plant is on its way, due for completion in 2024 💪
You can’t do it from behind a computer screen
Doing this during a pandemic was odd, to say the least.
Every single step of the project was affected — from factory shutdowns and global supply shortages to customs delays and interstate travel.
Thing is, you can’t build a global network from behind a computer screen. You need boots on the ground to build servers, package and ship things, install equipment and plug stuff in, as well as replace broken hardware in the middle of the night.
With travel restrictions preventing key staff from travelling interstate and overseas, we had to improvise.
Remote telesurgery: OneQode CEO Matt Shearing installs new hardware in a data centre, guided 500 miles away by Chief Architect Ben Cooper via TeamViewer’s augmented reality app
Things would have been 10 times more difficult without the last 5 years of technology.
Augmented reality support apps like Teamviewer Pilot have made remote data centre operations much easier. You can have a colleague assist you from the other side of the world, drawing arrows and annotations in real-time over objects in front of you.
A game-changer • TeamViewer Pilot demonstration video • Watch on YouTube
Teamwork tools have improved, too.
Most of our team have never met in person — we’re spread across 10 cities and 6 time zones.
We used to live on Microsoft Teams and Zoom, but they didn’t scratch our itch. Sometimes you want the spontaneity of being in an office instead.
So we switched to Gather. And it is good.
We’re lucky. We haven’t encountered a typhoon... yet
One thing we haven’t come across — thankfully — is a typhoon.
Guam is often rocked by tropical storms, and occasionally a typhoon. Some have been severe category 4 and 5 storms with winds reaching close to 300 km/h, and they leave devastation in their wake.
Typhoon Mangkhut passes through Guam in 2018 [Source: NOAA]
Much of the island’s infrastructure (including the data centres we’re in) is built to withstand the most extreme conditions. But, up against nature, anything can happen.All we can do is be prepared and build resilience into our systems. 🤞
In summary: it’s harder than it looks — but worth it
Bringing it all together, we’ve encountered delays, travel restrictions, chip shortages, hardware failures, electricity calculations, import duties and a truckload of paperwork.
But we did it.
We were the first company to put servers on the island designed to bring the region together, and while it’s been a hard road, it was worth it.
What would that look like for gaming with your friends Alice & Bob?
A marked improvement, with a very playable latency for everyone.
And how did it turn out in the real world?
Earlier this year we hosted a $17,000 cross-regional CS:GO tournament to put our brand new network to the test.
96 players from 9 countries took part over 5 days to fight for the prize pool, streamed live to a global audience on Twitch.
While we had a rocky start (solving scenic internet provider routing in real-time… story for another day), the tournament ended on a high — with an epic final between two excellent teams.
During the tournament, there was a defining moment when we knew we’d done something special.
We watched live as an Australian team battled a Mongolian team 5,000 km away — with less than 90ms of latency.
It was huge. Unthinkable before we put servers there.
We could hardly believe it.
Guam (APAC Central) is a brand new hub for cross-regional gaming. Players across 3 continents can now connect, introducing completely new strategies and levels of competition that haven’t been possible until now.
This is beyond exciting — and we are proud to declare Guam as the first region for our high-performance cloud that’s built for game studios, developers, publishers, and tournament operators.
We’re grateful to our families, teammates, suppliers and friends for supporting us up till now. Without your motivation, we would have given up long ago.
Finally, we’ve gained a new respect for the people who pave the way for their industry — in spite of the naysayers.
If that’s you, carry on 👏
