Performance Boost: VALORANT’s Global Invalidation

Hello! I’m Aaron Cheney, a software engineer on VALORANT’s Performance team. Performance is a key component to maintaining VALORANT’s competitive integrity, and our team is the one responsible for monitoring, maintaining, and improving both server and client performance.

We’re excited to provide an update on a feature that has been in development for several months: Global Invalidation. We’ll get into the details of the feature shortly, but first let’s spend some time looking at how this feature has impacted client performance since patch 4.03.

Spoiler alert: it’s pretty awesome.

Global Invalidation has provided significant improvements for a large part of our player base. In fact, it’s the single largest performance gain for clients since launch.

Although these charts are exciting—and we’re thrilled with the results—it’s important to understand exactly what you’re looking at. We work with large, complex data sets; organizing, filtering, and controlling the data helps us understand the player experience. Here’s what you need to keep in mind to understand the full picture:

• The chart plots “patch” vs. “average FPS”; higher numbers are better.
• Each line represents a common hardware configuration (CPU+GPU pairing) from among our player base. When analyzing performance data, we consider this pairing to be the most important predictor for expected performance. Machines with the same CPU+GPU pairing are aggregated together in these charts.
• The data samples are taken from two queues: Unrated and Competitive. Since these are the most popular game modes in VALORANT, we spend a great deal of effort to understand and improve performance in these areas.
• We’ve excluded games that don’t have exactly 10 players. This ensures outliers don’t skew the data (games with fewer players perform better).

SUMMARY OF GLOBAL INVALIDATION

Global Invalidation delivers up to 15% gains for CPU-bound clients (generally mid to high spec machines). Realizing these gains was an effort that spanned multiple teams and took many months to complete. Our process for identifying areas of the game ripe for optimization paid off, and managing risk along the way helped ensure a stable experience for players.

Who Benefits, and Calibrating Expectations

Based on our live metrics, Global Invalidation has delivered up to 15% improvement for client configurations that are CPU-bound (generally mid to high spec machines).

While the upward trend is noticeable in the aggregate, it doesn’t represent moment-to-moment gameplay. Additionally, it doesn’t guarantee that every machine with matching hardware will see identical results.

This means the baseline performance for VALORANT on CPU-bound machines has generally increased, but exactly how your specific machine will perform is dependent on a number of other factors unique to you.

UNDERSTANDING GLOBAL INVALIDATION

Before we can properly give an overview of Global Invalidation, we first need to explain a bit about UI elements in the Unreal Engine.

Widgets and Tree Structures

UI elements (also known as Widgets) are created from smaller building blocks using a tree structure. The tree structure is analogous to the file system on your computer. A Widget can have any number of children (just like a folder can have any number of files).

These building blocks can be combined to create complex Widgets. Our ammo counter, for example, consists of many parts, and the tree looks something like the following:

Putting it all together, the ammo counter building blocks look like the following:

(The red outlines above are exaggerated for clarity. In practice, many of the outlines would overlap.)

Whenever one or more Widgets change within the tree structure, it can affect multiple other Widgets. For example, if a Widget moves to a new position on screen, all Widgets underneath need to recalculate their position as well. These changes are managed by a system called “Invalidation,” which we’ll talk about in the next section.

Invalidation

Invalidation is the mechanism the Unreal Engine uses to indicate when a particular Widget has changed and needs to be updated.

A Widget needs to be invalidated for a variety of reasons: animation, color, opacity, size, ordering, text, images, and many other properties can change in response to something that happens in the game. When these changes happen to a Widget, it’s “Invalidated,” signaling that it needs to be updated.

Complicating this process a bit more, a Widget can have multiple types of invalidation. A few types include:

• Layout - When the size of a Widget changes (very expensive).
• Paint - When the look of the Widget has changed, but hasn’t changed size.
• Child Order - When the order of Widgets has changed within the tree (also implies Layout, and is therefore expensive).
• Visibility - When the visibility of a Widget has changed, either becoming invisible or becoming visible (also implies Layout, and is therefore expensive).

These types of invalidation are used to signify what kinds of operations must be performed to draw it properly.

It gets even more complicated when one Widget depends on another Widget. Widgets are organized into hierarchies, and their layout is dependent on a number of factors. Invalidating a single Widget may require invalidating a number of related Widgets to properly draw. For example, if multiple Widgets are organized into a vertical layout (e.g. the Social Panel with your friends list), and the order of Widgets changes (e.g. a friend comes online), all Widgets within the vertical layout need to be updated.

There are several goals with a system like this:

• Invalidate as few Widgets as possible. This reduces the number of Widgets that have to be updated in order to draw properly.
• Only invalidate a Widget when necessary. Improperly invalidating a Widget wastes precious CPU cycles.
• When a Widget isn’t invalidated, cache the result to quickly draw it every frame. If nothing changes, then CPU cycles can be saved.

That’s enough of a look under-the-hood to understand how Widgets update and what kinds of things can cause invalidation. Next, we’ll look at how developers put this into practice.

Invalidation Boxes

Unreal Engine provides a component—called an Invalidation Box—to group multiple Widgets together. All Widgets contained within a single Invalidation Box are prevented from getting pre-passed, ticked, or painted. Instead, the result is cached into a vertex buffer.

Whenever one of the Widgets within the Invalidation Box is invalidated, the cached data is discarded and the Widget gets updated and painted again. While refreshing the cache may be expensive for a single frame, the amortized result is far better in the long run.

Invalidation Boxes are a key part of making VALORANT’s UI performant, especially as we worked toward launch. However, they’re not entirely free:

• Developers need to understand which Widgets are good candidates for grouping together with an Invalidation Box. Any Widget that updates regularly isn’t suited for this.
• Placing Widgets within an Invalidation Box takes manual work on the part of a developer. It’s not feasible to do for every Widget in the game, and therefore developers must also understand which Widgets are worth putting into an Invalidation Box.

To read more about Invalidation Boxes, check out Epic’s documentation.

Now we have enough context to talk about Global Invalidation!

Enter Global Invalidation

By this point you may be thinking to yourself, “Why not just put every UI element into a global Invalidation Box?” Well, that’s precisely what Global Invalidation does (more or less).

Global Invalidation aims to significantly improve UI performance across the whole game while also reducing manual work required of developers to place Widgets in individual Invalidation Boxes. It’s the best of all worlds.

However, as of UE4.25 (the version of the Unreal Engine that VALORANT uses), Global Invalidation isn’t universally supported for all Widget types. Later versions of the Unreal Engine have made improvements, but VALORANT couldn’t take advantage of that right away. Additionally, we didn’t have a great understanding for how much faster Global Invalidation would make VALORANT.

Here’s where our work began.

WHY DID WE DECIDE TO DO THIS WORK?

In late July of 2021, the team decided to take Global Invalidation for a testrun during an internal playtest. Minor changes were made to fix a few bugs so the playtest could successfully complete. However, we knew bugs would surface during the playtest… and we certainly found bugs.

By the end of the playtest, we’d collected around 20 bugs—and those were just the obvious ones. It was likely more subtle, insidious bugs were waiting to be found, not to mention several edge cases that hadn’t been specifically tested.

But… did Global Invalidation deliver performance gains? It absolutely did.

Analyzing the data generated from that single playtest, it was determined that UI was ~35% faster. (Note: UI is only part of the cost for a single frame.)

However, we still had many open questions:

• How much time is needed to fix all of the bugs?
• Which team(s) should be responsible for the work?
• Does this take priority over other planned work? To meet regular content releases, our schedules are often set months in advance and emergent work–even something as exciting as this–is hard to fit in the schedule.
• Would fixing the bugs take away from the performance gains? Fixing all of the bugs would require many code changes, each of which has the potential to cause UI to become more expensive.
• When should we do this work? Knowing that Global Invalidation was being actively developed in later Unreal Engine versions meant we needed to consider our timeline for integrating changes from Epic.

Ultimately, we decided the work was worthwhile for a couple reasons.

Unreal Engine Integrations and Timing Considerations

Although Unreal Engine 4.26 and 4.27 have made significant progress on Global Invalidation, VALORANT works on a delayed integration schedule. The reason we don’t work on the “bleeding edge” is to manage risk and ensure stability for our players.

Since our schedule meant we would stay on Unreal Engine 4.25 for many more months, this meant players wouldn’t get to benefit from these performance gains for well over a year. That didn’t sit well with us.

For more details on how VALORANT thinks about Unreal Engine upgrades, check out this Twitter thread from VALORANT’s Tech Lead, Marcus Reid.

Known Performance Gains

Global Invalidation represented something really unique in terms of performance work: measured value. We measured the potential gains during an internal playtest, and the path to deliver that value was (mostly) clear.

Performance work is difficult. It’s a game of inches, not miles; incremental changes help generally improve performance over time, and it’s rare to find a single change capable of delivering double-digit gains. This was too juicy of an optimization to leave on the table.

Even after accounting for diminished gains due to bug fixes, Global Invalidation was the best chance for us to deliver significant gains for players within a reasonable amount of time.

HOW DID WE ACCOMPLISH THIS WORK?

Although initial experimentation around Global Invalidation began in late July 2021, earnest effort on stabilizing the feature didn’t start until late September 2021.

Selectively Integrating Epic’s Changes

Fully integrating Unreal Engine 4.26 and 4.27 was off the table, but since we knew Epic had actively worked on Global Invalidation, we decided to dig through thousands of changes and identify which changes were likely to get us closer to a stable, fully-functioning Global Invalidation.

Partially integrating changes was a challenging task. It was important to modify as few core engine features as possible to maintain stability while pulling in the right changes from Epic. All of this work was done in a separate branch from the main VALORANT branch to prevent impacting other developers.

After selectively integrating Epic’s changes into our engine, we spent several more weeks fixing as many bugs as possible while we prepared to introduce Global Invalidation into the main VALORANT branch. Along the way a toggle was created to let us quickly enable and disable the feature (should anything catastrophic occur).

With many bugs fixed, and with many of Epic’s changes from 4.26 and 4.27, we merged from our isolated branch back into the main branch.

Finding Commonalities Between Bugs

Although many of the bugs with Global Invalidation manifested in different ways, the root cause could often be traced back to a single issue. These were the most valuable bugs to fix, as doing so could eliminate multiple issues with a single change. One change, for example, fixed 10+ bugs that were scattered throughout the game. Careful analysis of the root problem led to robust solutions that improved the reliability and stability of Global Invalidation.

Identify Bugs, Fix Bugs, Playtest, Rinse and Repeat

The following weeks and months involved a regular cycle of enabling Global Invalidation before a playtest, identifying a series of bugs, disabling Global Invalidation after the playtest, and fixing those bugs.

Fewer bugs were reported with each iteration of the loop. We continued doing this until the steady stream of bugs turned into a slow drip, and eventually stopped.

By the end of November 2021, all of the major issues were resolved and Global Invalidation was largely stable.

Notable Bugs

• Astra crashes the game - At one point, every Astra player was greeted with a crash upon loading into the game. This bug was fixed after integrating changes from Epic.
• Multiple inheritance crash - Multiple inheritance in C++ is a tricky subject. Without getting too far into the weeds, the order of destructors in a particular class did not execute in the right order, causing a crash. Simply swapping 2 lines of code to change the inheritance order fixed the problem. To read more about multiple inheritance, check out this page.
• Infinite chat audio - While in the menus, the chat bar plays a sound when you hover over it with the mouse. To the annoyance of everyone, a bug caused that audio to play multiple times a second. Fixing this bug involved understanding the timing of how Widgets receive mouse events multiple times a frame.

Testing Methodologies

One element of Global Invalidation that made us particularly cautious (and therefore thorough with our testing) is that it affects every aspect of the game. Literally.

Your friends list? Absolutely. The button you press to queue up? That too. The settings menu? You betcha. My headshot percentage? Well…

The point is that UI elements exist throughout the game, and they often convey critical information to players. Breaking even one of those UI elements was unacceptable.

To that end, our QA department created a testing plan with multiple strategies to build confidence that Global Invalidation was working as intended.

Vertical Slice Testing

A “vertical slice” of VALORANT represents the main path players generally take, from launching the client, to queuing up for a match, to playing a full game, to interacting with the end-of-game screen. By focusing on the critical elements of the game, QA could quickly test the most used elements of the game and identify problems early.

Destructive Testing

Where vertical slice testing leaves off, destructive testing picks up. This type of testing is intended to identify off-nominal issues, often by modifying external factors (such as network ping, frame rate, alt-tabbing, etc.). Armed with a set of internal tools, QA spent several weeks doing destructive testing.

Edge Case Testing

Many parts of VALORANT are only experienced by a small percentage of the player base. Some parts of the game are only ever experienced once (e.g. the New Player Experience). Just because those parts of the game are less traversed, it doesn’t make them less important. Identifying and testing all of our edge cases helped catch hidden errors.

PBE (Public Beta Environment) Testing

PBE was a major milestone for Global Invalidation.

• It was the first time outside players could test the feature. This meant that Global Invalidation could be tested under “real world” conditions.
• PBE represents a variety of hardware specifications. PBE intentionally covers the range from low to high specs, and testing with such a large range of player devices helped us build confidence in how well Global Invalidation would perform across the wider player base.

After testing on PBE during the weekend of Jan. 22, 2022 - Jan. 23, 2022, we had confidence that Global Invalidation didn’t interfere with integrity and that performance gains were in line with our predictions.

Releasing Global Invalidation

After launching Global Invalidation with Patch 4.03, we closely monitored player reports for bugs. We also kept a close eye on the performance data to confirm our estimations matched the results. In the end, Global Invalidation was a huge success for players, and we hope you enjoy the improved frame time.

Now that Global Invalidation is out in the world, the Performance team is going back to work on delivering even more gains. Until next time, happy fragging!