This blog post is a companion to a talk that Jonathan gave at Flock to Fedora 2024: From Zero to Everywhere: AlmaLinux’s High-Speed Mirrorlist Evolution
In August of 2021 we first launched our new and improved mirror system, including the geo-location feature that allows users to automatically use the closest mirror to them. This had the incredible benefit of making sure users had the fastest access to updates possible. In the time since we launched, we’ve grown from around 100 mirror servers to over 400. The majority of these mirrors are hosted at universities, businesses, and institutions that believe in our mission and want to show their support.
In the almost three years since then, we’ve made a lot of improvements to the mirrorlist which supports over 1 million systems worldwide. Let’s talk about them!
Map showing the over 400 active AlmaLinux mirrors as of August 5, 2024
Dedicated AWS Mirroring
AWS is the most used cloud platform by AlmaLinux users which use our public mirrorlist system so it was only appropriate for us to work closely with Amazon to deploy a dedicated mirroring infrastructure across AWS. This means that for all AWS users, update traffic to your VMs is local within the AWS network. Not only does this save on ingress charges, it means update downloads are FAST. A side effect of this is update traffic from AWS is no longer hitting the rest of our mirrorlist which is great for mirror owners, and other non-AWS AlmaLinux users since it leaves more bandwidth available for everyone else.
Thanks to AWS for making this possible and helping us improve the user experience for AlmaLinux users on AWS!
Improved Mirrorlist Caching
By adjusting the way we cache mirrorlist requests - now just per IP, instead of per IP, version, and repo - we have improved our cache hit rates by over 50%. This means when running dnf upgrade it hits https://mirrors.almalinux.org to fetch a list of mirrors for updates the transaction is up to 240ms faster - 80ms per repository after the first.
As a side effect of this, a race condition we’ve seen in the past is now resolved that was caused by hitting 3 different mirrors (or more depending on what repos are enabled) for the various repos when they weren’t all perfectly updated and there were dependencies between the repos. This sometimes yielded transaction errors about missing dependencies.
We also moved away from a per-instance status checker and redis cache to a centralized redis cache with a separate, single status checker. This means increased redis cache hits and faster response times. It also means the list of mirrors at https://mirrors.almalinux.org will always be the same - previously multiple page refreshes could yield a slightly different list due to the individual caches on each of the mirrorlist service VMs.
Speed and Functionality
We have found a ton of ways to decrease latency and make it less likely for someone to hit a mirror that isn’t actually online or up to date for whatever reason. To help make sure that you’re always being served a valid mirror, we have increased the frequency of mirror status checks to every 15 minutes instead of 30.
We have also upgraded the VMs that serve the mirror list itself to AWS c7a instances (from c6a). That means more resources are available for each request, and that alone has yielded a ~20ms per request speedup on average.
New and More Rsync Tier 0 Mirrors
All updates are initially served from one of our Tier 0 mirrors, and last week we brought on a fourth one - this time in Japan. We now have 130Gbps of capacity across our 4 Tier 0 mirrors. These are provided by Hivelocity, Ziply Fiber, and Macarne. These organizations (two of which are Sponsor members of the AlmaLinux OS Foundation) believe in our vision and see the value we provide, and we cannot thank them enough.
Starting out we had a single tier 0 mirror with a 10Gbps uplink in Atlanta, GA. Now our tier 0s exist in Atlanta, GA (Hivelocity, 10Gbps); Seattle, WA (Ziply Fiber, 50Gbps); Frankfurt, Germany (Macarne, 50Gbps); and Tokyo, Japan (Macarne, 20Gbps) and are accessed via geo-steering DNS. This helps us get out new content to mirrors at lightning speed!
“Alma had the highest performance tier0 of literally any project mirrored by FCIX/MicroMirror, then they went and upgraded it”
- Kenneth Finnegan, Micro Mirror Project / FCIX
IPinfo Geolocation Database
The most significant change from a user experience perspective is our swap from MaxMind’s GeoIP database to IPinfo. In our use thus far IPinfo has proven to be a much more accurate and comprehensive geolocation database which allows us to serve better (closer) mirrors to users.
The improvement in this area will impact users in geographically-large countries the most. The reason for this is MaxMind reported users as being in the geographical center of a country when it lacked state/province and city data for an address. In the case of the US, this means in the middle of a lake in Kansas…not a major internet hub by far. I have to give a big shout out to our four mirrors near this magical location as they’ve been really putting in some work over the past year when ~20% of our mirrorlist traffic for the US was hitting them due to this incomplete data.
Additionally, MaxMind’s data was incorrect even on a country level in many cases with the result being, for example, European users were hitting North American mirrors and vice versa. We’ve spot checked hundreds if not thousands of IPs and have found IPinfo to be incredibly accurate and as a result anticipate these users to have significantly better (read faster) update experiences now.
They say a picture is worth 1000 words so let’s have a look:
MaxMind’s interpretation of US-based mirrorlist requests for the week of Oct 15-21, 2023
IPinfo’s interpretation of US-based mirrorlist requests for the week of Oct 15-21, 2023
The images above make it very apparent just how much traffic was being poorly directed due to the inaccurate geolocation data. The next three most-impacted countries are Canada, China, and Australia.
Huge shout out to the amazing team at IPInfo who have been phenomenal to work with and became an official sponsor of AlmaLinux!
Up Next
We have so many plans for more improvements to the mirror system! In a quick list, here are just some of our ideas.
- Better statistics
- What mirrors are getting sent the most traffic?
- Public heatmaps of traffic patterns
- Interactive AlmaLinux usage statistics from countme data.
- Improved development environment to ease onboarding of contributors
- Protocol filtering
- Ability to receive only https-enabled mirrors.
- Partial mirroring
- Some organizations may not have the capacity to mirror the entire AlmaLinux repo but want to contribute. We need to support mirroring only specific architectures and data sets (ISOs vs packages, etc.)
- Better mirror validation
- We can make further improvements to how we validate mirror status to ensure only up-to-date and online mirrors are served to users.
If you are interested in helping us get these done, come join us in ~mirrors on chat.almalinux.org!
Becoming a mirror sponsor at AlmaLinux
If you’re one of the more than 400 AlmaLinux mirror hosts, it’s easy to become an AlmaLinux OS Foundation mirror sponsor member. There’s a quick form to fill out, and then the membership committee reviews and approves your application! Joining the foundation as a mirror sponsor gives you the right to vote in elections as long as your mirror remains active, and gets you listed as an official AlmaLinux mirror sponsor anywhere we list mirror sponsors. If you’re interested, get started now!
