Coding isn’t dead for people who know how to write programs and know how hardware actually runs those programs. The engineer who knows both will be the last one or never kicked out of this well paid, fun industry. Even if people on X declare it dead every week. If you want to get better at the hardware side, this post is for you, and you start today with cache line consequences for concurrent code.
I will show examples in Go, but if you’ve ever written code in another backend language, you won’t have a problem following along.
Let’s first get to the basics.
What is cache line? A cache line is the smallest unit of data that moves between RAM and the CPU cache. It’s typically 64 bytes, so even if you read one byte, the CPU fetches the whole 64 bytes line.
What is cache coherence? Cache coherence makes sure that when a CPU core writes to a cache line, the other cores’ copies of that line are invalidated, so no core can read the old value. Note that RAM can stay stale while the dirty line lives in a core’s cache. It gets updated later, when the line is written back.
Those have concrete consequences for concurrent code. Most notably false sharing, which can cause a significant performance drop. Let’s look at what false sharing is with an example.
Say you’re building analytics for a YouTube-like platform. You have a list of videos, and you want two numbers: total views and total likes across all of them. Easy win for parallelism, right? In Go, you can spin up two goroutines, one sums the views and the other sums the likes. They never touch the same variable, so no locks and it should be twice as fast.
type Video struct {
views int64
likes int64
}
type Stats struct {
totalViews int64
totalLikes int64
}
func aggregate(videos []Video) Stats {
var wg sync.WaitGroup
wg.Add(2)
stats := Stats{}
go func() {
defer wg.Done()
for i := 0; i < len(videos); i++ {
stats.totalViews += videos[i].views
}
}()
go func() {
defer wg.Done()
for i := 0; i < len(videos); i++ {
stats.totalLikes += videos[i].likes
}
}()
wg.Wait()
return stats
}Let’s have a look what happens in the RAM. Stats’s object fields are allocated contiguously to the same memory block.
If you have two cores CPU, the goroutines can be scheduled in two separate threads. Each goroutine updates its own field of Stats.
Remember the cache coherence? Since we have CPU cache coherence when the goroutine 1 writes to total views, the core’s 2 copy of that line is invalidated. And when the goroutine 2 writes to total likes, the core’s 1 copy of that line is invalidated. Two threads write to two independent fields, but those fields sit on the same cache line. Every write kicks the line out of the other core’s cache, so that core has to get it again updated. Then it writes, and kicks it back. The line keeps bouncing between cores. Sometimes it comes from the other core’s cache or L3, and in the worst case it goes all the way to RAM. Each fetch costs way more than a normal cache hit. In CPU time, that’s ages. Nothing is shared logically but the CPU acts like it is.
So, we need to make sure those independent fields live on separate cache lines. For such case, there is a technique called “padding“. We allocate extra memory between independent data.
type Stats struct {
totalViews int64 // int64 needs 8 bytes
_ [56]byte // padding
totalLikes int64
}
Using the padding now let’s see what it improves.
The updates are independent. It doesn’t cause the cache coherence and back-and-forth fetch overhead.
Conclusion
I hope this all makes sense to you. I tried to explain it as simple as possible. There are some items that I didn’t address in this article like how CPU mechanism keeps data consistent across many cores while updates happen. And of course, I simplified a little cores structures and data flow inside to get the main point.
If this was helpful, send it to an engineer friend. Less AI doomerism, more leveling up.
Thanks for reading,
Adlet Balzhanov
Also, big news on my side. I’m joining a leading AI infra company as a software engineer. There’s a lot I need to learn like inference, GPUs, low level stuff.
So, I created an account on X. I’ll be learning in public and following all the frontier people there as I go. I’ll share what I pick up, plus any AI infra, inference, frontier software engineering insights worth passing on.
If that sounds useful, come follow me on X.
Let’s learn together. The button’s right below.