Read more

Now that we’ve seen how a debugger modifies a debuggee (using the kernel and CPU features), let’s put these pieces together and see how we might implement both breakpoints and instruction-level stepping in a basic debugger.

The Control Loop

Recall in Part 3 that we wrote a simple Windows “debugger”, which launched and attached to a process, and logged the debug events that the debugger received.

We launched and attached to the process through CreateProcessA:

  // launch process, attach
  char *cmd_line = arguments[1];
  STARTUPINFOA startup_info = {sizeof(startup_info)};
  PROCESS_INFORMATION process_info = {0};
  CreateProcessA(0, cmd_line, 0, 0, 0, DEBUG_PROCESS, 0, 0, &startup_info, 
                 &process_info);

We gathered debug events using WaitForDebugEvent and ContinueDebugEvent:

// gather events from debuggee
for(DEBUG_EVENT evt = {0};
    WaitForDebugEvent(&evt, INFINITE);
    ContinueDebugEvent(evt.dwProcessId, evt.dwThreadId, DBG_CONTINUE))
{
  // use `evt`
  if(evt.dwDebugEventCode == EXIT_PROCESS_DEBUG_EVENT)
  {
    break;
  }
}

This “debugger” does nothing but follow along the debuggee as it executes. When it receives an event, it immediately resumes the debuggee, and doesn’t modify it at all. But an actually usable debugger, of course, allows the user to choose when the debuggee resumes, and what modifications to make before it does, through a user interface.

For the purposes of this post, those debuggee modifications will include instruction-level stepping and instruction-level breakpoints.

To support this, let’s enclose this event gathering loop with a control loop, which will receive commands to either place breakpoints, step threads, or to resume the debuggee.

for(B32 is_running = 1; is_running;)
{
  // read commands from user
  for(B32 need_commands = 1; need_commands;)
  {
    CommandKind kind = ...; // unpack the kind of command somehow
    switch(kind)
    {
      default:                        {...}break;
      case CommandKind_Resume:        {...}break;
      case CommandKind_Quit:          {...}break;
      case CommandKind_InstStep:      {...}break;
      case CommandKind_SetBreakpoint: {...}break;
    }
  }

  // gather events
  for(DEBUG_EVENT evt = {0};
      WaitForDebugEvent(&evt, INFINITE);
      ContinueDebugEvent(evt.dwProcessId, evt.dwThreadId, DBG_CONTINUE))
  {
    // ...
  }
}

First, for the basic commands, our work is simple.

For CommandKind_Resume, we simply set need_commands to 0. Execution flows to the event loop, which resumes the debuggee until an event occurs which the debugger would need to know about.

For CommandKind_Quit, we simply set need_commands to 0, then is_running to 0, and then we kill the debuggee—for example, on Windows, using the TerminateProcess API:

TerminateProcess(process_handle, 0);

We also need to adjust our debug event loop to terminate (thus allowing for more command execution) under certain conditions. This is a debugger design decision, although many choices are obvious—for example, should the debuggee pause if the debugger is notified that a debuggee thread is created? Probably not. If the debuggee completes an instruction-level step, or hits a trap instruction? Then yes. If the debuggee encounters an exception? Then also yes.

// gather events
for(DEBUG_EVENT evt = {0};
    WaitForDebugEvent(&evt, INFINITE);
    ContinueDebugEvent(evt.dwProcessId, evt.dwThreadId, DBG_CONTINUE))
{
  B32 should_take_more_commands = 0;
  // ...
  // thread creation? should_take_more_commands -> 0
  // hit exception? should_take_more_commands -> 1
  // ...
  if(should_take_more_commands)
  {
    break;
  }
}

Now, let’s consider how we might implement the instruction-level stepping and breakpoint commands.

Read more

I covered in Part 3 that debuggers help analyze programs—“debuggees”—both by reading from, and writing to, them. The kernel exposes features which facilitate this bi-directional flow of information. A debugger can receive debug events, read memory, read registers, and write memory, and write registers.

But what, exactly, does a debugger write into a debuggee’s registers or memory to modify it?

CPUs have a number of mechanisms—exposed by their instruction set architecture definitions—that are built for debugging. By modifying registers or memory, a debugger can make use of these mechanisms.

A debugger can use specific registers to set up the CPU such that it behaves differently when executing debuggee code. A debugger can also dynamically modify instruction memory, to dynamically adjust which instructions are executed by debuggee threads, at certain points in codepaths.

In this post, I’ll cover the basics of the following CPU mechanisms, and how they can be used by a debugger:

• Instruction pointer register — A register which is used to store at which address the CPU will next execute an instruction.

• Interruption instructions — Instructions that can be written into an instruction stream, which cause the CPU core to interrupt execution of that instruction stream.

• Data breakpoint registers — Registers which are used to implement “data breakpoints”, where an exception will occur if some number of bytes at specific addresses are written to, read from, or executed.

• Single-stepping mode — A mode which causes the CPU core to immediately interrupt after executing a single instruction.

• Return instructions — Instructions which return from a called procedure, replacing the instruction pointer with one stored on a thread’s call stack.

Read more

Usenet is a decentralized message board network created in 1980. Predating the World Wide Web’s ubiquity, Usenet was inhabited by a small and exclusive population from universities, laboratories, or other prestigious institutions. The general public originally didn’t know what it was, have the means to access it, or know why it mattered.

Even in simple text-based communication, humans build culture, norms, shared ideas, and etiquette. These are social technologies, developed to find and apply common purpose, maintain a standard of discussion quality, and share information efficiently. Usenet was no exception.

Throughout the ‘80s and ‘90s, at the beginning of each academic year, new university students would have their first opportunity to join Usenet. So, every September, existing Usenet users would notice an influx of “noobs”.

Noobs joined and disrupted culture. Training a noob to be familiar with a board’s shared ideas, etiquette, norms—and to be respectful of them—was not free. But given sufficient time, noobs would either assimilate or leave. Given the still-limited access—and given that it was limited to university students, an already exclusive (at the time) demographic—it was manageable to maintain the spirit of many boards, despite the yearly September influx.

This changed in 1993. Eternal September began when easy Usenet access was first offered to the general public by Internet Service Providers. Usenet boards—originally frequented, again, by university students, professors, engineers, scientists—were flooded by a large and seemingly permanent wave of noobs, sourced from a much larger and less exclusive demographic.

The masses of noobs were so large, and so culturally far, and in many cases so cognitively far from those who originally frequented boards, that moderation—and thus the preservation of board culture—became non-viable. Old-timers became outnumbered by noobs—their boards were no longer theirs. What was once a decentralized network of communication hubs for small, elite circles became much more representative of modern Internet boards—what you might see if you have the misfortune of taking a wrong turn while surfing the web.

I don’t remember Eternal September because I wasn’t born when it happened, let alone using a computer. But when I first learned about the shift it brought, I was intrigued, as I’d experienced something strikingly similar.

Other examples of similar cultural disruption beyond the Internet are not scarce. The CGDC of the 1990s is not the same as today’s GDC. The Bell Labs of the 1950s is not the same as today’s Nokia Bell Labs. The Apple of the 1990s is not the same as today’s Apple. The Microsoft of the 1990s is not the same as today’s Microsoft. The Google of the 1990s is not the same as today’s Google.

The Death of Google

In each case, what began as a selective, elite, high quality, productive, and revolutionary organization, ultimately sacrificed health for growth. This is not to dismiss the still brilliant individuals within any given organization—I mean to merely describe patterns at the granularity of the organizations themselves.

A microcosm of the same phenomenon can be seen in the conflict between early Google programmer Ben Gomes—who joined Google in 1999, and worked as a pivotal contributor to the early success of Google’s flagship Search product—and others, notably Prabhakar Raghavan—the “noob” who joined Google in 2012.

There was a fierce conflict between Gomes and others like Raghavan over the decision to intentionally degrade Search quality in favor of serving more advertisements to users—or, in other words, what users didn’t ask for—to provoke a larger number of queries.

There is perhaps no clearer example of the rise—in Gomes—and fall—in Raghavan—of the Google “Don’t Be Evil” motto. Gomes seemed to truly believe in the original spirit of Google, and felt that it was being corrupted by a lust for growth:

I've been thinking a bit about what Shashi says and I tend to agree that we are getting too close to the money.

[…]

I think it is good for us to aspire to query growth and to aspire to more users. But I think we are getting too involved with ads for the good of the product and company. We need to think of other issues like DuckDuckGo and the privacy challenge and our innovation narrative. We need to retain users for the long run.

[…]

I am getting concerned that growth is all we are thinking about.

Raghavan, on the other hand, was a noob—far removed from the original mission and spirit of Google, and far too focused on second-order spreadsheet metrics like “number of queries”—too depraved and soulless to understand that such metrics are downstream from spirit.

People Are Not Fungible

For any given example, it’s not difficult to find anecdotes from old-timers, where they nostalgically recall the old times, and lament what they feel was lost. The reason is clear: this is an undesirable phenomenon. More is not always better.

Individuals within larger and less exclusive populations have less in common with one another; there is less to bind them together. Reasons to form strong social bonds within the community have become informal, whereas in a highly selective group, the selection itself makes those reasons formal.

In the younger versions of these organizations, due to their more exclusive nature, meeting another person is refreshing. Upon meeting, you instantly know many deeply personal things you can bond over—shared passions, interests, hobbies, goals, philosophies. This makes friendships, partnerships, and information transfer both pleasant and efficient. In the aged versions of these organizations, one’s null hypothesis about commonalities between any two people change such that this is not true.

This phenomenon is driven by an organization’s age, and it seems to necessarily flow in one direction—towards the preference of growth, at the cost of exclusivity and cultural preservation. It’s unsurprising that companies, specifically, exhibit this effect—and ultimately regress towards the mean—because it reduces risk, and doesn’t immediately compromise profit. The longer a company is alive, the more incentive there will be to stop betting on the initial “magic” which made the company function to begin with (since it’ll inevitably die out regardless).

The more aged an organization, the more generic the bonds, the less exclusive the members, the larger the membership—and thus, a much higher social cost to change. It’s not popular to make an inclusive group exclusive. In many cases, it’s unclear how to do it at all. It destabilizes relationships, causes controversy, and poisons the air with conflict.

Put simply, in the long term, the quality of an organization can only diminish with time.

Therefore, the initial formulation of an organization—who is in it—is critical. Beyond that, the degree to which both exclusive selection and “noob training” continue to occur determines the organization’s age—or at least as long as it has until it deteriorates beyond recognition.

This is all for a simple reason: people are not fungible.

The Better Software Conference

The first Better Software Conference took place in July 2025. I had the privilege of giving a talk.

I arrived in Stockholm the day before conferencegoers were to meet. This gave me a chance to briefly explore the beautiful, historic city.

The next day, conferencegoers met up at the airport. We found ourselves on a four hour train ride to what seemed like the middle of Swedish nowhere. For many attendees including myself, this came after a long international flight to Sweden, so the extra travel might seem like it’d be exhausting.

Physically I was tired, without a doubt. But mentally, nobody seemed to be exhausted. Despite being comprised almost entirely of strangers, the whole four hours were filled by fun and fascinating conversations—which didn’t subside until the conference ended almost a week later.

We arrived in a charming small town, and were led by one of the organizers on a short walk from the train to the somewhat Twin-Peaks-like hotel.

After getting settled in, the organizers had the hotel generously arrange free food and drink in the hotel’s conference room, and the aforementioned conversation resumed for hours.

The organizers had worked with the town’s local government to arrange the conference, so the talks all took place in the town’s beautiful theater, which was a short walk from the hotel. Upon entering the theater on the first morning of talks, attendees were greeted with coffee, snacks, and music that lifted the spirits. It brilliantly created an atmosphere that supported continuing conversation and camaraderie.

Even before any talks had occurred, the conference had all of the properties I described of those fresh, exciting, and revolutionary circles. Everyone I spoke to was passionate about their craft. Everyone wanted to build something beautiful. Everyone loved what they did—at least, enough to fly to Sweden from all across the globe, take a four hour train ride to a rural town, and spend several days surrounded by a bunch of (mostly) strangers.

But nobody felt like strangers. The morning after arriving in the town, there was friendly Brazilian jiu-jitsu sparring, lake swims, and many-hour-long conversations.

This is partly because of Internet familiarity, but it was also because of a completely unique null hypothesis about shared passions, ideas, and knowledge. It was safe to assume that a “stranger” knew about—for instance—the early game engine work of John Carmack, or the game technology work at RAD, or Casey Muratori’s Handmade Hero, or the work and streams of Jonathan Blow. And it was safe to assume that they had a passion about writing high-quality games, engines, tools, or other software largely from scratch, to push the boundaries of what is capable in software, and to produce results superior to those elsewhere in industry.

And because they had passion, they had spent thousands of hours of their own time practicing and refining their abilities. Many attendees were highly successful in shipping their own games or other software—turn around, and you might just happen to run into the creator of Teardown, or a programmer who works on the Jai compiler, or the creator of File Pilot!

This made communication pleasant, valuable for all parties, and highly efficient. The conference was socially energizing, rather than exhausting—even for a population as naturally introverted as programmers. It was unfortunate to go to bed each night, because you’d have to cut exhilarating conversations with great, talented, and personable friends short.

High Signal, Low Noise

On top of the excellent atmosphere and practicalities, the talks at Better Software Conference were stellar. The signal-to-noise ratio was unlike almost any conference I’d seen (the only comparable equivalents were Casey’s HandmadeCon in 2015 and 2016).

The conference kicked off with a phenomenal talk from Casey himself, breaking down the history of a nearly ubiquitous—perhaps mistakenly—paradigm in the software industry. The subsequent talks in the day featured insights from Dennis Gustafsson (creator of Teardown) on physics engine parallelization, Bill Hall (gingerBill, creator of Odin and programmer at JangaFX) on learning mathematical tools, and Vjekoslav Krajačić on the codebase of File Pilot. This was only the first day.

The selection of speakers and the talk quality was no happy accident—the organizers didn’t simply get lucky. If your speakers regress to the mean, then your content regresses to the mean.

What do high quality speakers want? First, to be surrounded by other high quality people—or in other words, high quality attendees. Second, they want freedom. Third, they want a platform. The conference organizers provided all three. First, by carefully inviting attendees. Second, by keeping the conference schedule highly flexible, without any talk time limits or required topics. Third, by streaming all talks freely online to the public, with excellent professional audio and video.

Most importantly, because high quality speakers want an excellent in-person audience, and to be surrounded by others they’d be interested in speaking with, if your attendees regress to the mean, then your speakers regress to the mean.

Euphemisms For Power

This invite-only structure is an unpopular way to execute a conference, here at the tail end of an era of “inclusivity”. Look no further than—and I’m sorry to do this—many online comments about the conference:

This is a small selection, but plenty similar comments exist. Of course, the conference was invite-only, and thus “exclusive”, but it wasn’t selecting for particular demographics, it was selecting for—and exclusively for—people who care about better software. It’s in the title. It’s why the conference wasn’t—despite the characterization—demographically homogeneous. Unsurprisingly, those more concerned with demographics than the topic of the conference wouldn’t typically make the cut.

Such rants about “sameyness”, or the conference looking like a “frat party”, or “dudes duding it up dudely”, are resentful, deranged tantrums that the demographics of a conference—a computer programming conference—attended by primarily Swedes, other Europeans, and Americans—do not match those that they’d prefer, which suggests a rather nefarious undertone.

The stories I’ve told in this post—which are a small sample of the same pattern which has repeated time and time again—teach an important lesson: obsession with growth will kill the spirit of an organization. Growth can manifest in a number of ways, but most importantly, it can manifest financially and influentially. Increasing growth in one of those two ways requires expanding a particular population—a userbase, a conference’s attendees, a follower count—which necessarily requires appeal to a broader population. In other words, the appeal must become less particular—more generic—to include a broader population.

Is this not inclusivity?

And so it comes full circle—the executive boardroom meeting discussing the maximization of growth—and the Internet “discussion” board “discussing” the maximization of inclusivity—are, fundamentally, discussing their desire for exactly the same outcome. The characteristic difference is that the boardroom is discussing it from inside an organization, and the Internet board is discussing it from outside an organization—but both desire, ultimately, growth. The terms they use may differ, yes—both groups use the appropriate language to maximally signal virtue to those from whom they desire social approval—but in substance, it is the same.

Furthermore, to desire financial growth and influential growth is to desire power—to desire that most of all is to have lust for power.

But for someone who is primarily concerned with craft, or the creation of beauty, and would rather financial gain and influence come as second order effects of that, this lust for power is detrimental to their whole purpose. It kills their communities, their organizations, their companies, their teams, and thus detracts from their already-difficult work.

This is why careful selection of people is critical, and this is why it works. In many cases—for example, in that of Usenet—this selection is informal and organic. In other cases, like that of the invite-only Better Software Conference, it was formalized. Both work, though formalization can aid the longevity of an organization as it was intended.

The reason I chose to write this post is that this conference was one of the best experiences of my life, but it was also an extremely unique experience. My experience is that the overwhelming number of companies, communities, conferences, and meetups do not fit this description. Most of them are Usenet—after Eternal September. I find that highly undesirable; the Better Software Conference provides a blueprint to build companies, communities, conferences, or other organizations which are highly fruitful and desirable.

The organizers of the Better Software Conference—beyond doing an excellent job planning practical necessities, like travel, lodging, and food—and doing a phenomenal job providing high quality audio & video and an excellent presentation space for speakers—understood that people are not fungible. They understood that—to put on a great conference—you need great people, together, with a platform, space, and ample free time.

When put like that, it sounds simple. And yet almost nobody does it. After all, that’s the philosophy of better software: Do the simple thing that solves the problem, that nobody else is thinking about.

If you enjoyed this post, please consider subscribing. Thanks for reading.

-Ryan

Subscribe now

We decided to record the discussion, since we felt it might be fruitful for others to hear. I enjoyed the discussion, and I hope you do too.

-Ryan

Subscribe now

Learning to program a single CPU core is difficult. There is an enormous number of techniques, amount of information, and number of hours to spend in order to learn to do it effectively. Learning to program multiple CPU cores to do work in parallel, all while these cores cooperate in accomplishing some overarching task, to me seemed like the anvil that broke the camel’s back—so to speak—there is already so much to wrangle when doing single-core programming, that for me, it was much more convenient to ignore multi-core programming for a long time.

But in the modern computer hardware era, there emerges an elephant in the room. With modern CPU core counts far exceeding 1—and instead reaching numbers like 8, 16, 32, 64—programmers leave an enormous amount of performance on the table by ignoring the fundamentally multi-core reality of their machines.

I’m not a “performance programmer”. Like Casey Muratori (which is partly what made me follow him to begin with), I have always wanted reasonable performance (though this might appear like “performance programming” to a concerning proportion of the software industry), but historically I’ve worked in domains where I control the data involved, like my own games and engines, where I am either doing the art, design, and levels myself, or heavily involved in the process. Thus, I’ve often been able to use my own programming constraints to inform artistic constraints.

All of that went out the window over the past few years, when in my work on debuggers, I’ve needed to work with data which is not only not under my control, but is almost exactly identical to the opposite of what I’d want—it’s dramatically bigger, unfathomably poorly structured, extraordinarily complicated, and not to mention unpredictable and highly variable. This is because, as I’ve written about, debuggers are at a “busy intersection”. They deal with unknowns from the external computing world on almost all fronts. And if one wanted a debugger to be useful for—for instance—extraordinarily large codebases that highly successful companies use to ship real things, those unknowns include unfortunate details about those codebases too.

As such, in my work, making more effective use of the hardware has been far more important than it ever has been for me in the past. As such, I was forced to address the “elephant in the room” that is CPU core counts, and actually doing multi-core programming.

I’ve learned a lot about the multi-core aspect of programming in the past few years, and I’ve written about lessons I’ve learned during that time, like those on basic mental building blocks I used to plan for multithreaded architecture, and carefully organizing mutations such that multiple threads require little-to-no synchronization.

I still find those ideas useful, and my past writing still captures my thoughts on the first principles of multi-core programming. But recently, thanks to some lessons I learned after a few discussions with Casey, my abilities in concretely applying those first principles have “leveled up”. I’m writing this post now to capture and share those lessons.

The Parallel for (And Its Flaws)

Because every programmer learns single-core programming first, it’s common—after one first learns multi-core programming techniques—to apply those techniques conservatively within otherwise single-core code.

To make this more concrete, consider the following simple example:

S64 *values = ...;
S64 values_count = ...;
S64 sum = 0;
for(S64 idx = 0; idx < values_count; idx += 1)
{
  sum += values[idx];
}

In this example, we compute a sum of all elements in the values array. Let’s now consider a few properties of sums:

• a + b + c + d = (a + b) + (c + d)

• a + b + c + d = d + c + b + a

• (a + b) + (c + d) = (c + d) + (a + b)

Because we can reconsider a sum of elements as a sum of sums of groups of those elements, and because the order in which we sum elements does not impact the final computation, the original code can be rewritten like:

S64 *values = ...;
S64 values_count = ...;

S64 sum0 = 0;
for(S64 idx = 0; idx < values_count/4; idx += 1)
{
  sum0 += values[idx];
}

S64 sum1 = 0;
for(S64 idx = values_count/4; idx < (2*values_count)/4; idx += 1)
{
  sum1 += values[idx];
}

S64 sum2 = 0;
for(S64 idx = (2*values_count)/4; idx < (3*values_count)/4; idx += 1)
{
  sum2 += values[idx];
}

S64 sum3 = 0;
for(S64 idx = (3*values_count)/4; idx < (4*values_count)/4 && idx < values_count; idx += 1)
{
  sum3 += values[idx];
}

S64 sum = sum0 + sum1 + sum2 + sum3;

That obviously doesn’t win us anything—but what this means is that we can obtain the same result by subdividing the computation into several, smaller, independent computations.

Because several independent computations do not require writing to the same memory, they fit nicely with multi-core programming—each core does not need to synchronize at all with any other. This not only greatly simplifies the multi-core programming, but improves its performance—or, more precisely, it doesn’t eat away from the natural performance obtained by executing in parallel.

For cases like this, we can implement what’s known as a “parallel for”. The idea is that we’d like to specify our original for loop…

for(S64 idx = 0; idx < values_count; idx += 1)
{
  sum += values[idx];
}

…but we’d like to also express that the loop can be subdivided into independent computations (the results of which we can join into a single result later).

In other words, we begin with normal, single-core code. But, for some computation, we want to “go wide”, and compute something in parallel. Then, we want to “join” this wide, parallel work, and go back to more single-core code, which can use the results of the work done in parallel.

This is a widely known and used concept. In many real codebases written in modern programming languages which offer many tools for abstraction building, you’ll find a number of impressive gymnastics to succinctly express this.

One of the reasons I prefer working in a simpler language is that, if what my code ultimately generates to facilitate some abstraction is complicated, that being reflected directly in the source code helps keep me honest about how “clean” some construct actually is.

If, on the other hand, some higher level utility can be provided by a simple and straightforward concrete implementation, that is a sign of a superior design—one that does not compromise on its implementation, but also does not compromise on its higher level utility.

Many people behave as though this is impossible—that higher level utility necessarily incurs substantial tradeoffs at the low level, or vice versa, that low level properties like performance necessitate undesirable high level design. This is simply not universally true. By hunting for tradeoffs, many programmers train themselves to ignore cases when they can both have, and eat, their cake.

So, if we consider our options for implementing a “parallel for” without a lot of modern language machinery, we might start with something like this:

struct SumParams
{
  S64 *values;
  S64 count;
  S64 sum;
};

void SumTask(SumParams *p)
{
  for(S64 idx = 0; idx < p->count; idx += 1)
  {
    p->sum += p->values[idx];
  }
}

S64 ComputeSum(S64 *values, S64 count)
{
  S64 count_per_core = count / NUMBER_OF_CORES;
  SumParams params[NUMBER_OF_CORES] = {0};
  Thread threads[NUMBER_OF_CORES] = {0};
  for(S64 core_idx = 0; core_idx < NUMBER_OF_CORES; core_idx += 1)
  {
    params[core_idx].values = values + core_idx*count_per_core;
    params[core_idx].count = count_per_core;
    S64 overkill = ((core_idx+1)*count_per_core - count);
    if(overkill > 0)
    {
      params[core_idx].count -= overkill;
    }
    threads[core_idx] = LaunchThread(SumTask, &params[core_idx]);
  }

  S64 sum = 0;
  for(S64 core_idx = 0; core_idx < NUMBER_OF_CORES; core_idx += 1)
  {
    JoinThread(threads[core_idx]);
    sum += params[core_idx].sum;
  }

  return sum;
}

There are a number of unfortunate realities about this mechanism:

1. In something like LaunchThread and JoinThread, we interact with the kernel to create and destroy kernel resources (threads) every time we perform a sum.

2. The actual case-specific code we needed (for the sum, in this case), and the number of particular details we had to specify and get right—like the work subdivision—has exploded. What used to be a simple for loop has been spread around to different, more intricate parts, all implementing different details of the mechanism we wanted—the work preparation, the work kickoff, and the joining and combination of work results. All parts must be maintained and changed together, every time we want a parallel for.

3. The solution’s control flow has been scattered across threads, CPU cores, and time. We can no longer trivially step through the sum in a debugger. If we encounter a bug in some iterations in a parallel for, we need to correlate the launching of that particular work, and that actual work. For example, if we stop the program in the debugger and find ourselves within a thread performing some iterations of the parallel for, we have lost context about who launched that work (in single-core code, this information is universally provided with call stacks).

The first problem can be partly addressed with a new underlying layer which our code uses instead of the underlying kernel primitives. In many codebases, this layer is called a “job system”, or a “worker thread pool”. In those cases, the program prepares a set of threads once, and these threads simply wait for work, and execute it once they receive it:

void JobThread(void *p)
{
  for(;;)
  {
    Job job = GetNextJob(...);
    job.Function(job.params);
  }
}

void SumJob(SumParams *p)
{
  ...
}

S64 ComputeSum(S64 *values, S64 count)
{
  Job jobs[NUMBER_OF_CORES] = {0};
  for(S64 core_idx = 0; core_idx < NUMBER_OF_CORES; core_idx += 1)
  {
    ...
    jobs[core_idx] = LaunchJob(SumJob, &params[core_idx]);
  }

  S64 sum = 0;
  for(S64 core_idx = 0; core_idx < NUMBER_OF_CORES; core_idx += 1)
  {
    WaitForJob(jobs[core_idx]);
    sum += params[core_idx].sum;
  }

  return sum;
}

In this case, there is still some overhead incurred by sending to and receiving information from the job threads, but it is significantly lighter than interacting with the kernel.

But it hasn’t improved the higher level code very much at all—we’ve simply replaced “threads” with “jobs”. The latter two problems hold. We still need to perform an entire dance in order to set up a “wide loop”—a “parallel for”, which scatters control flow for a problem across both source code, and coherent contexts (CPU cores, call stacks) at runtime.

In this concrete case—computing a sum in parallel—this is not a huge concern. Will it compute a sum in parallel? Yes. Does it have very few shared data writes? Yes. Can you parallelize all similarly parallelizable problems this way? Yes. But, we pay the costs of these problems every time we use this mechanism. If we have to pay that cost very frequently throughout a problem, it can become onerous to write, debug, and maintain all of this machinery.

The Job System (And Its Flaws)

One desirable property of the parallel for is that all jobs—which execute at roughly the same time, across some number of cores—are identical in their “shape”. Each job thread participating in the problem is executing exactly the same code—we simply parameterize each job slightly differently, to distribute different subproblems to different cores. This makes understanding, predicting, profiling, and debugging such code much simpler.

Furthermore, within a parallel for, each job’s lifetime is scoped by the originating single-core code’s lifetime. Each job begins and ends within some scope—the scope responsible for launching, then joining, all of the jobs. This means no substantial lifetime management complexity occurs—allocations for a parallel for are as simple as for normal single-core code.

But in practice, the mechanism often used to implement parallel fors—the job system—is rarely only used in this way, which is understandable, given its highly generic structure. For example, it’s also often used to launch a number of heterogeneous jobs. In these cases, it becomes even more difficult to understand the context of a particular job—who launched it, and in what context? It also becomes more difficult to comprehensively understand a system—because there is such a large number of possible configurations of thread states, it can be difficult to ensure a threaded system is robust in all cases.

These jobs are also often not bounded by their launcher scope—as such, more engineering must be spent on managing resources, like memory allocations, whose lifetimes are now defined by what happens across multiple threads in multiple contexts.

And this is, really, the tip of the iceberg. In more sophisticated systems, one might observe that there are dependencies between jobs, and jobs ought to be implicitly launched when their dependency jobs complete, creating an even longer (and more difficult to inspect) chain of context related to some independent through line of work.

Ultimately, this presents recurring writing, reading, debugging, and maintenance costs that don’t exist in normal single-core code. All of the costs incurred by this job system design—whether used in a parallel for or otherwise—are paid any time new parallel work is introduced, or any time parallel work is maintained.

Now, if we have few parts of our code that can be parallelized in this way, then this is not a significant cost.

…But that if is doing a lot of heavy lifting.

In practice, I’ve found that an enormous number of systems are riddled with opportunities for parallelization, because of a lack of serial dependence between many of their parts. But, if taking advantage of every instance of serial independence requires significantly more engineering than just accepting single-core performance, then in many cases, programmers will opt for the latter.

Again—does this mean that a job system cannot be used to do such parallelization in these systems? No. But, it also means that we pay the costs of using this job system—the more moving parts; the extra code and concepts to write, read, and debug—much more frequently, if we’d like to take advantage of this widespread serial independence, or if we’d like any algorithm in particular to scale its performance with the number of cores.

Single-Core By Default

The critical insight I learned from speaking with Casey on this topic was that a significant reason why these costs arise is because of the careful organization a system needs in order to switch from single-core to multi-core code. Mechanisms like job systems and their special case usage in parallel fors represent, in some sense, the most conservative application of multi-core code. The vast majority of code is written as single-core, and a few carveouts are made when multi-core is critically important. In other words, code remains single-core by default, and in a few special cases, work is done to briefly hand work off to a multi-core system.

Because the context of code execution changes across time—because work is handed off from one system to another—it necessarily requires more code to set up, and it is more difficult to debug and understand the full context at any point in time.

But is this the best approach? Perhaps, instead of writing single-core code (which sometimes goes wide) by default, we can write multi-core code (which sometimes goes narrow) by default.

What does this look like in practice?

There’s a good chance that you’ve already experienced this style in other areas of programming—notably, in GPU shader programming.

GPU shaders—like vertex or pixel shaders, used in a traditional GPU rendering pipeline—are written such that they are multi-core by default. You author a single function (the entry point of the shader), but this function is executed on many cores, always, implicitly. The language constructs and rules are arranged in such a way that data reads and writes are always scoped by whatever core happens to be executing the code. A single execution of a vertex shader is scoped to a vertex—a pixel shader to a pixel—and so on.

Because the fundamental, underlying architecture is always multi-core by default, and because there is little involvement of each specific shader in how the multi-core parallelism is achieved, GPU programming enjoys enormous performance benefits, and yet as the shader programmer, it feels that there are few costs to pay for it. So few, in fact, that it feels more like artistic scripting, to the degree that someone can build an entire website—Shadertoy—built around rapid-iteration, high-performance, visual GPU scripting.

Wait a minute… “high performance”, “rapid-iteration scripting”? It seems like many believe that these are mutually exclusive!

Why does CPU programming feel so different?

Contrast the GPU programming model to the usual CPU programming model—you author a single function (the entry point of your program), which is scheduled onto a single core only, normally by a kernel scheduler, using a single thread state. This model is, in contrast, single-core by default.

Long story short: it doesn’t have to be!

Multi-Core By Default

Let’s begin by exactly inverting the approach. Instead of having a single thread which kicks off work to many threads, let’s just have many threads, all running the same code, by default. In a sense, let’s have just one big parallel for:

void BootstrapEntryPoint(void)
{
  Thread threads[NUMBER_OF_CORES] = {0};
  for(S64 thread_idx = 0; thread_idx < NUMBER_OF_CORES; thread_idx += 1)
  {
    threads[thread_idx] = LaunchThread(EntryPoint, (void *)thread_idx);
  }
  for(S64 thread_idx = 0; thread_idx < NUMBER_OF_CORES; thread_idx += 1)
  {
    JoinThread(threads[thread_idx]);
  }
}

void EntryPoint(void *params)
{
  S64 thread_idx = (S64)params;
  // program's actual work occurs here!
}

To click into an architecture which assumes a single-threaded entry point, we start with a BootstrapEntryPoint. But the only work this function actually does is launch all of the threads executing the actual entry point, EntryPoint.

Let’s consider the earlier summation example. First, let’s just take the original single-threaded code, and put it into EntryPoint, and see how we can continue from there.

void EntryPoint(void *params)
{
  S64 thread_idx = (S64)params;

  // we obtain these somehow:
  S64 *values = ...;
  S64 values_count = ...;

  // compute the sum
  S64 sum = 0;
  for(S64 idx = 0; idx < values_count; idx += 1)
  {
    sum += values[idx];
  }
}

What is actually happening? Well, we’re “computing the sum across many cores”. That is… technically true! Ship it!

There’s just one little problem… This is just as fast as the single-core version, except it also uses enormously more energy, and steals time from other tasks the CPU could be doing, because it is simply duplicating all work on each core.

But, if we were to measure this, and consider the real costs, and profile the actual code, the profile would look something like this:

Duplication itself is not, in principle, a problem, and it is sometimes not to be avoided, because deduplication can sometimes be more expensive than duplication. For instance, communicating the result of a single add instruction across many threads—to deduplicate the work of that add—would be vastly more expensive than simply duplicating the add itself. We do want deduplication, but only when necessary, or when it actually helps.

So, where does it help? Unsurprisingly in this case, the dominating cost—the reason we are using multiple cores at all—is the sum across all elements in values. We want to distribute different parts of the sum across cores. To start, instead of computing the full sum, we can instead compute a per-thread sum. After each per-thread sum is computed, we can then combine them:

void EntryPoint(void *params)
{
  S64 thread_idx = (S64)params;

  // we obtain these somehow:
  S64 *values = ...;
  S64 values_count = ...;

  // decide this thread's subset of the sum
  S64 thread_first_value_idx = ???;
  S64 thread_opl_value_idx = ???; // one past last

  // compute the thread sum
  S64 thread_sum = 0;
  for(S64 idx = thread_first_value_idx;
      idx < thread_opl_value_idx;
      idx += 1)
  {
    thread_sum += values[idx];
  }

  // combine the thread sums
  S64 sum = ???;
}

We have two blanks to fill in:

1. How do we decide each thread’s subset of work?

2. How do we combine all thread sums?

Let’s tackle each.

1. Deciding Per-Thread Work

Currently, the only input I’ve provided each thread is its index, which would be in [0, N), where N is the number of threads. This is stored in the local variable thread_idx, which will have a different value in [0, N) for each thread. This is an easy example, because a good way to distribute the sum work across all threads is to uniformly distribute the number of values to sum amongst the threads. This means we are simply mapping [0, M) to [0, N), where M is the number of values—values_count—and N is the number of threads.

We can almost compute this as follows:

S64 values_count = ...;
S64 thread_idx = ...;
S64 thread_count = NUMBER_OF_CORES;

S64 values_per_thread = values_count / thread_count;
S64 thread_first_value_idx = values_per_thread * thread_idx;
S64 thread_opl_value_idx = thread_first_value_idx + values_per_thread;

This is almost right, but only almost, because we also need to account for the case where values_count is not cleanly subdivided by thread_count. Because our values_per_thread will truncate to the next lowest integer, this current distribution will underestimate the number of values we need to compute per thread, by anywhere from 0 (if it divides cleanly) to thread_count-1 values—or in other words, the remainder of the division.

Thus, the number of values this division underestimates by—the “leftovers”—can be computed as follows:

S64 leftover_values_count = values_count % thread_count;

We can then distribute these leftovers amongst the first leftover_values_count threads:

// compute the values-per-thread, & number of leftovers
S64 values_per_thread = values_count / thread_count;
S64 leftover_values_count = values_count % thread_count;

// determine if the current thread gets a leftover
// (we distribute them amongst the first threads in the group)
B32 thread_has_leftover = (thread_idx < leftover_values_count);

// decide on how many leftovers have been distributed before this
// thread's range (just the thread index, clamped by the number of
// leftovers)
S64 leftovers_before_this_thread_idx = 0;
if(thread_has_leftover)
{
  leftovers_before_this_thread_idx = thread_idx;
}
else
{
  leftovers_before_this_thread_idx = leftover_values_count;
}

// decide on the [first, opl) range:
// we shift `first` by the number of leftovers we've placed earlier,
// and we shift `opl` by 1 if we have a leftover.
S64 thread_first_value_idx = (values_per_thread * thread_idx +
                              leftovers_before_this_thread_idx);
S64 thread_opl_value_idx = thread_first_value_idx + values_per_thread;
if(thread_has_leftover)
{
  thread_opl_value_idx += 1;
}

Or more succinctly:

S64 values_per_thread = values_count / thread_count;
S64 leftover_values_count = values_count % thread_count;
B32 thread_has_leftover = (thread_idx < leftover_values_count);
S64 leftovers_before_this_thread_idx = (thread_has_leftover
                                        ? thread_idx
                                        : leftover_values_count);
S64 thread_first_value_idx = (values_per_thread * thread_idx +
                              leftovers_before_this_thread_idx);
S64 thread_opl_value_idx = (thread_first_value_idx + values_per_thread + 
                            !!thread_has_leftover);

Now, using this [first, opl) calculation, we can arrange each thread to only loop over its associated range, thus not duplicating all sum work done by other threads.

2. Combining All Thread Sums

Now, how might we combine each thread’s sum to form the total sum? There are two simple options available: (a) we can define a global sum counter to which each thread atomically adds (using atomic add intrinsics) its per-thread sum, or (b) we can define global storage which stores all thread sums, and each thread can duplicate the work of computing the total sum.

For (a), we just need to define sum as static, and atomically add each thread_sum to it:

static S64 sum = 0;

void EntryPoint(void *params)
{
  // ...
  // compute `thread_sum`
  // ...
  AtomicAddEval64(&sum, thread_sum);
}

Note: This has a downside in that only one thread group can be executing this codepath at once. This is sometimes not a practical concern, since if we are going wide at all, we are often using all available cores to do so, and it is likely not beneficial to also have some other thread group executing the same codepath for a different purpose. That said, it’s now a new hidden restriction of this code, and it can be a critical problem. There are some techniques we can use to solve this problem, which I will cover later—for now, the important concept is that the data is shared across participating threads.

For (b), we’d instead have a global table, and duplicate the work of summing across all thread sums. But we can only do that after we know that each thread has completed its summation work—otherwise we’d potentially add some other thread’s sum before it was actually computed!

static S64 thread_sums[NUMBER_OF_CORES] = {0};

void EntryPoint(void *params)
{
  // ...
  // compute `thread_sum`
  // ...
  thread_sums[thread_idx] = thread_sum;

  // ??? need to wait here for all threads to finish!

  S64 sum = 0;
  for(S64 t_idx = 0; t_idx < NUMBER_OF_CORES; t_idx += 1)
  {
    sum += thread_sums[t_idx];
  }
}

That extra waiting requirement might seem like an argument in favor of (a), but we’d actually need the same mechanism if we did (a) once we wanted to actually use the sum—we’d need to wait for all threads to reach some point, so that we’d know that they’d all atomically updated sum.

We can use a barrier to do this. In (a):

static S64 sum = 0;
static Barrier barrier = {0};

void EntryPoint(void *params)
{
  // ...
  // compute `thread_sum`
  // ...
  AtomicAddEval64(&sum, thread_sum);
  BarrierSync(barrier);
  // `sum` is now fully computed!
}

And in (b):

static S64 thread_sums[NUMBER_OF_CORES] = {0};
static Barrier barrier = {0};

void EntryPoint(void *params)
{
  // ...
  // compute `thread_sum`
  // ...
  thread_sums[thread_idx] = thread_sum;

  BarrierSync(barrier);

  S64 sum = 0;
  for(S64 t_idx = 0; t_idx < NUMBER_OF_CORES; t_idx += 1)
  {
    sum += thread_sums[t_idx];
  }
  // `sum` is now fully computed!
}

At this point, we have everything we need for both (a) and (b). Both are simple, and likely negligibly different. (a) requires atomic summation across all the threads, which implies hardware-level synchronization, whereas (b) duplicates the sum of all per-thread sums—these likely subtly differ in their costs, but not by much when compared to the actual values summation.

Going Narrow

Now, while I hope this summation example has been a useful introduction, I know it’s a bit contrived, and incomplete. Specifically, it’s missing two key parts of any program: inputs and outputs. What are we doing with this sum, and how do we use that in producing some form of output, and how do obtain the inputs, and store them in values and values_count?

Let’s barely extend the summation example with stories for the inputs and outputs. For the inputs, let’s say that we read values out of a binary file, which just contains the whole array stored as it will be in memory. For the outputs, let’s say that we just print the sum to stdout with printf.

Printing out the sum will be the easiest part, so let’s begin with that.

In single-core code, after computing the sum, we’d simply call printf:

S64 sum = ...;
// ...
printf("Sum: %I64d", sum);

We can start by just doing the same in our “multi-core by default” code. What we’ll find is that our output looks something like this:

Sum: 12345678
Sum: 12345678
Sum: 12345678
Sum: 12345678
Sum: 12345678
Sum: 12345678
Sum: 12345678
Sum: 12345678

And obviously, we only want our many cores to be involved with the majority of the computation, but we only need one thread to do the actual printf. In other words, we need to go narrow. Luckily, going narrow from wide code is much simpler than going wide from narrow code:

S64 sum = ...;
// ...
if(thread_idx == 0)
{
  printf("Sum: %I64d", sum);
}

We simply need to mask away the work from all threads except one.

Now, let’s consider the input problem. We need to compute values_count based on the size of some input file, allocate storage for values, and then fill values by reading all data from the file.

Single-threaded code to do that might look something like this:

char *input_path = ...;
File file = FileOpen(input_path);
S64 size = SizeFromFile(file);
S64 values_count = (size / sizeof(S64));
S64 *values = (S64 *)Allocate(values_count * sizeof(values[0]));
FileRead(file, 0, values_count * sizeof(values[0]), values);
FileClose(file);

So, naturally, one option is to simply do this narrow:

if(thread_idx == 0)
{
  char *input_path = ...;
  File file = FileOpen(input_path);
  S64 size = SizeFromFile(file);
  S64 values_count = (size / sizeof(S64));
  S64 *values = (S64 *)Allocate(values_count * sizeof(values[0]));
  FileRead(file, 0, values_count * sizeof(values[0]), values);
  FileClose(file);
}
BarrierSync(barrier); // `values` and `values_count` ready after this point

This will work, but we somehow need to broadcast the computed values of values and values_count across all threads. One easy way to do this is simply to pull them out as static, like we did for shared data earlier:

static S64 values_count = 0;
static S64 *values = 0;
if(thread_idx == 0)
{
  char *input_path = ...;
  File file = FileOpen(input_path);
  S64 size = SizeFromFile(file);
  values_count = (size / sizeof(S64));
  values = (S64 *)Allocate(values_count * sizeof(values[0]));
  FileRead(file, 0, values_count * sizeof(values[0]), values);
  FileClose(file);
}
BarrierSync(barrier);

But consider that we might not want to do this completely single-core. It might be the case that it’s more efficient to issue FileReads from many threads, rather than just one. In practice, this is partly true (although, depending on the full stack—the kernel, the storage drive hardware, and so on—it may not be beneficial past some number of threads, and for certain read sizes).

So let’s say we’d like to do the FileReads wide now also. We need to still allocate values on a single thread, but once that is done, we can distribute the rest of the work trivially:

// we can open the file on all threads (though for some reasons
// we may want to deduplicate this too - for simplicity I am
// keeping it on all threads)
File file = FileOpen(input_path);

// calculate number of values and allocate (only single thread)
static S64 values_count = 0;
static S64 *values = 0;
if(thread_idx == 0)
{
  S64 size = SizeFromFile(file);
  values_count = (size / sizeof(S64));
  values = (S64 *)Allocate(values_count * sizeof(values[0]));
}
BarrierSync(barrier);

// compute thread's range of values (same calculation as before)
S64 thread_first_value_idx = ...;
S64 thread_opl_value_idx = ...;

// do read of this thread's portion
S64 num_values_this_thread = (thread_opl_value_idx - thread_first_value_idx);
FileRead(file,
         thread_first_value_idx*sizeof(values[0]),
         num_values_this_thread*sizeof(values[0]),
         values + thread_first_value_idx);

// close file on all threads
FileClose(file);

It’s much simpler, now—compared to, say, the original parallel for case—to simply take another part of the problem like this, and to also distribute it amongst threads, simply because wide is the default shape of the program.

Instead of spending most programming time acting like we’re on a single-core machine, we simply assume our actual circumstances, which is that we have several cores, and sometimes we need to tie it all together with a few serial dependencies.

Non-Uniform Work Distributions

Let’s take a look at our earlier calculations to distribute portions of the values array:

S64 values_per_thread = values_count / thread_count;
S64 leftover_values_count = values_count % thread_count;
B32 thread_has_leftover = (thread_idx < leftover_values_count);
S64 leftovers_before_this_thread_idx = (thread_has_leftover
                                        ? thread_idx
                                        : leftover_values_count);
S64 thread_first_value_idx = (values_per_thread * thread_idx +
                              leftovers_before_this_thread_idx);
S64 thread_opl_value_idx = (thread_first_value_idx + values_per_thread + 
                            !!thread_has_leftover);

This was an easy case, because uniformly dividing portions of values produces nearly uniform work across all cores.

If, in a different scenario, we don’t produce nearly uniform work across all cores, we have a problem: some cores will finish their work in some section long before others, and they’ll be stuck at the next barrier synchronization point while the other cores finish. This diminishes the returns we obtain from going wide in the first place.

Thus, it’s always important to uniformly distribute work whenever it’s possible. The exact strategy for doing so will vary by problem. But I’ve noticed three common strategies:

1. Uniformly distributing inputs produces uniformly distributed work (the case with the sum). So, we can decide the work distribution upfront.

2. Each portion of an input requires a variable amount of per-core work. The work is relatively bounded, and there are many portions of input (larger than the core count). So, we can dynamically grab work on each core, so cores which complete smaller work first receive more, whereas cores that are stuck on longer work leave more units of work for other cores.

3. Each portion of an input requires a variable amount of per-core work, but there is a small number (lower than the core count) of potentially very long sequences of work. We can attempt to redesign this algorithm such that it can be distributed more uniformly instead.

We’ve already covered the first strategy with the sum example—let’s look at the latter two.

Dynamically Assigning Many Variable-Work Tasks

Let’s consider a case where we have many units of work—“tasks”—and we’d like to distribute these tasks across cores. We may start by distributing the tasks in the same way that we distributed values to sum in the earlier example:

Task *tasks = ...;
S64 tasks_count = ...;
S64 thread_first_task_idx = ...;
S64 thread_opl_task_idx = ...;
for(S64 task_idx = thread_first_task_idx;
    task_idx < thread_last_task_idx;
    task_idx += 1)
{
  // do task
}

If each task requires a variable amount of work, then a profile of the program might look something like this:

Instead of deciding the task division upfront, we can dynamically assign tasks, such that the threads which are occupied (performing larger tasks) are not assigned more tasks until they’re done, and threads which complete shorter tasks earlier are quickly assigned more tasks, if available.

We can do that simply with a shared atomic counter, which each thread increments:

Task *tasks = ...;
S64 tasks_count = ...;

// set up the counter
static S64 task_take_counter = 0;
task_take_counter = 0;
BarrierSync(barrier);

// loop on all threads - take tasks as long as we can
for(;;)
{
  S64 task_idx = AtomicIncEval64(&task_take_counter) - 1;
  if(task_idx >= tasks_count)
  {
    break;
  }
  // do task
}

This will dynamically distribute tasks across the cores, so that a profile of the program will look more like this:

Redesigning Algorithms For Uniform Work Distribution

Dynamically assigning tasks to cores will help in many cases, but it gets less effective if tasks are highly variable, to the point of sometimes being exceedingly long (e.g. many times more expensive than smaller tasks), or if there are fewer tasks than the number of cores.

In these cases, it can often be helpful to reconsider the serial independencies within a single task, or whether the same effect as a highly serially-dependent algorithm can be provided by an alternative highly serially-independent algorithm. Can a single task be subdivided further? Can it be performed in a different way? Can serially-dependent work be untangled from heavier work which can be done in a serially-independent way?

The answers to such questions are highly problem-specific, so it’s impossible to offer substantially more useful advice while staying similarly generic. But to illustrate that it’s sometimes possible—even when counterintuitive—I have an example problem from my recent work, in which finding more uniform work distribution required switching from a single-threaded comparison sort to a highly parallelizable radix sort.

In this problem, I had a small number of arrays that needed to be sorted, but these arrays were potentially very large, thus requiring a fairly expensive sorting pass.

My first approach was to simply distribute the comparison sort tasks themselves, so I would sort one array on a single core, while other cores would be sorting other arrays. But as I’ve said, there were a relatively small number of arrays, and the arrays were large, so sorting was fairly expensive—thus, most cores were doing nothing, and simply waiting for the small number of cores performing sorts to finish.

This approach would’ve worked fine if I had a larger number of smaller tasks. In fact, another part of the same program does distribute single-threaded comparison sort tasks in this way, because in that part of the problem, there are a larger number of smaller tasks.

In this case, I needed to sort array elements based on 64-bit integer keys. After sorting, the elements needed to be ordered such that their associated keys were ascending in value.

Conveniently, this can be done with a radix sort. I won’t cover the full details of the algorithm here (although I briefly covered it during a stream recently, which I recorded and uploaded here), but the important detail is that a radix sort requires a fixed number of O(N) passes over the array, and huge portions of work in each pass can be distributed uniformly across cores (in the same way that we distributed the sum work earlier).

Now, all cores participate in every larger sorting task, but they only perform a nearly uniform fraction of the work in each sort. This results in a much more uniform work distribution, and thus a much shorter total time spend sorting:

This is just one concrete example a larger pattern I’ve noticed: In many problems, upon close examination, some serial dependencies can either vanish, or they can be untangled from heavier work.

In some problems, serially-dependent parts of the algorithm can be isolated, such that they prepare data which allows the rest of the algorithm to be done in a serially-independent fashion. Imagine a program which walks a linked list early, on a single core, to compute a layout in a serially-dependent way. This layout can then allow subsequent work to execute just using the full layout, rather than forcing that subsequent work to also include the serially-dependent pointer chasing.

Single-Threaded, Just Better

Code which is multi-core by default feels like normal single-threaded code, just with a few extra constructs that express the missing information needed to execute on multiple cores. This style has some useful and interesting properties, which make it preferable in many contexts to many of the popular styles of multi-core code found in the wild.

Single-Core as a Parameterization

One interesting implication of code written in this way—to be multi-core by default—is that it offers a strict superset of functionality than code which is written to be single-core, because “multi-core” in this case includes “single-core”, as one possible case. We can use the same code to execute on only a single core, simply by instead executing our entry point on a single thread, and parameterizing that thread with thread_idx = 0 and thread_count = 1.

In that case, one core necessarily receives all of the work. BarrierSyncs turn into no-ops, since there is only one thread (there are no other threads to wait for). Thus, it is equivalent to single-core functionality.

Simpler Debugging

This style of multi-core programming requires far less busywork and machinery in order to use multiple cores for some codepath. But one of the problems I mentioned with job systems and parallel fors earlier was not only that they require more busywork and machinery, but that they’re also more difficult to debug.

In this case, debugging is much simpler—in fact, it doesn’t look all that different from single-core debugging. At every point, you have access to a full call stack, and all contextual data which led to whatever point in time that you happen to be inspecting in a debugger.

Furthermore, because all threads involved are nearly homogeneous (rather than the generic job system, where all threads are heterogeneous at all times), debugging a single thread is a lot like debugging all threads. This is especially true because—between barrier synchronization points—the threads are all executing the same code. In other words, the context and state on one thread is likely to be highly informative of the context and state on all threads.

Access To The Full Stack

Because the context for some through line of computation frequently changes in traditional job systems, extra machinery must be involved to pipe data from one context to another—across jobs and threads—and maintain any associated allocations and lifetimes. But in this style, resources and lifetimes are kept as simple as they are in single-threaded code.

The stack, containing all contextual state at any point, becomes a single bucket for useful thread-local storage. In a job system, the stack is useful multi-core thread-local storage, but only for the duration of the job. The job is equivalent to the inner body of a for—this is a tiny, fragmentary scope. With this style, the entire stack is available, at any point.

Codebase Support

I’ve found some useful patterns which can be extracted and widely used in code which is multi-core by default. These patterns seem as widely applicable as arenas—as such, they can be a useful addition to a codebase’s base layer.

Thread-Local Group Data

LaneIdx(), LaneCount(), LaneSync()

The earlier example code frequently uses the thread_idx, thread_count, and barrier variables. Passing these to every codepath which might need them is redundant and cumbersome. As such, they are good candidates for thread-local storage.

In my code, I’ve bundled these into the base layer’s “thread context”, which is a thread-local structure which is universally accessible—it’s where, for example, thread-local scratch arenas are stored.

This provides all code the ability to read its index within a thread group (thread_idx), or the number of threads in its group (thread_count), and to synchronize with other lanes (BarrierSync).

As I suggested earlier, any code’s caller can choose “how wide”—how many cores—they’d like to execute that code, by configuring this per-thread storage. In general, shallow parts of a call stack can decide how wide deeper parts of a call stack are executed. If some work is expected to be small (to the point where it doesn’t benefit from being executed on many cores), and other cores can be doing other useful work, then before doing that work, the calling code can simply set thread_idx = 0, thread_count = 1, and barrier = {0}.

This means that a single thread may participate in many different thread groups—in other words, thread_idx and thread_count are not static within the execution of a single thread. Therefore, I found it appropriate to introduce another disambiguating term: lane. A lane is distinct from a thread in that a lane is simply one thread within a potentially-temporary group of threads, all executing the same code.

As such, in my terminology, thread_idx is exposed as LaneIdx(), and thread_count is exposed as LaneCount(). To synchronize with other lanes, a helper LaneSync() is available, which just waits on the thread context’s currently selected barrier.

Uniformly Distributing Ranges Amongst Lanes

LaneRange(count)

I’ve mentioned the following computation multiple times:

S64 values_per_thread = values_count / thread_count;
S64 leftover_values_count = values_count % thread_count;
B32 thread_has_leftover = (thread_idx < leftover_values_count);
S64 leftovers_before_this_thread_idx = (thread_has_leftover
                                        ? thread_idx
                                        : leftover_values_count);
S64 thread_first_value_idx = (values_per_thread * thread_idx +
                              leftovers_before_this_thread_idx);
S64 thread_opl_value_idx = (thread_first_value_idx + values_per_thread + 
                            !!thread_has_leftover);

This is useful whenever a uniformly distributed range corresponds to uniformly distributed work amongst cores. As I mentioned, this is sometimes not desirable. But nevertheless, it’s an extremely common case. As such, I found it useful to expose this as LaneRange(count):

Rng1U64 range = LaneRange(count);
for(U64 idx = range.min; idx < range.max; idx += 1)
{
  // ...
}

Broadcasting Data Across Lanes

LaneSyncU64(value_ptr, source_lane_idx)

Earlier, we saw that when a variable needs to be shared across lanes, it can simply be marked as static. I mentioned that this has the unfortunate downside that only a single group can be executing the code at one time, since one group of lanes could trample over the static variable while another group is still using it. As I mentioned, this is sometimes not a concern (since it’s desirable to only have a single lane group executing some code), but it invisibly makes code inapplicable for some cases.

For example, let’s suppose I have some code which is written to be multi-core by default. Depending on the inputs to this codepath, I may want this to be executed—on the same inputs—with all of my cores. But in other cases, I may want this to be executed with only a single core—I may still want to execute this codepath on other cores, but for different inputs. That requires many lane groups to be executing the code at the same time, thus disqualifying the use of static to share data amongst lanes within the same group.

To address this, I also created a simple mechanism to broadcast small amounts of data across lanes.

Each thread context also stores—in addition to a lane index, lane count, and lane group barrier—a pointer to a shared buffer, which is the same value for all lanes in the same group.

If one lane has a value which it needs to be broadcasted to other lanes—for instance, if it allocated a buffer that the other lanes are about to fill—then that value can be communicated in the following way:

U64 broadcast_size = ...;         // the number of bytes to broadcast
U64 broadcast_src_lane_idx = ...; // the index of the broadcasting lane
void *lane_local_storage = ...;   // unique for each lane
void *lane_shared_storage = ...;  // same for all lanes

// copy from broadcaster -> shared
if(LaneIdx() == broadcast_src_lane_idx)
{
  MemoryCopy(lane_shared_storage, lane_local_storage, broadcast_size);
}
LaneSync();

// copy from shared -> broadcastees
if(LaneIdx() != broadcast_src_lane_idx)
{
  MemoryCopy(lane_local_storage, lane_shared_storage, broadcast_size);
}
LaneSync();

I’ve found that this shared buffer just needs to be big enough to broadcast 8 bytes, given that most small data can be broadcasted with a small number of 8 byte broadcasts, and larger data can be broadcasted with a single pointer broadcast.

I expose this mechanism with the following API:

U64 some_value = 0;
U64 src_lane_idx = 0;
LaneSyncU64(&some_value, src_lane_idx);
// after this line, all lanes share the same value for `some_value`

It might be used in the following way:

// set `values_count`, allocate for `values`, on lane 0, then
// broadcast their values to all other lanes:
S64 values_count = 0;
S64 *values = 0;
if(LaneIdx() == 0)
{
  values_count = ...;
  values = Allocate(sizeof(values[0]) * values_count);
}
LaneSyncU64(&values_count, 0);
LaneSyncU64(&values, 0);

Revisiting The Summation Example

With the above mechanisms, we can program the original summation example with the following steps.

First, we load the values from the file:

U64 values_count = 0;
S64 *values = 0;
{
  File file = FileOpen(input_path);
  values_count = SizeFromFile(file) / sizeof(values[0]);
  if(LaneIdx() == 0)
  {
    values = (S64 *)Allocate(values_count * sizeof(values[0]));
  }
  LaneSyncU64(&values);
  Rng1U64 value_range = LaneRange(values_count);
  Rng1U64 byte_range = R1U64(value_range.min * sizeof(values[0]),
                             value_range.max * sizeof(values[0]));
  FileRead(file, byte_range, values + value_range.min);
  FileClose(file);
}
LaneSync();

Then, we perform the sum across all lanes:

// grab the shared counter
S64 sum = 0;
S64 *sum_ptr = ∑
LaneSyncU64(&sum_ptr, 0);

// calculate lane's sum
S64 lane_sum = 0;
Rng1U64 range = LaneRange(values_count);
for(U64 idx = range.min; idx < range.max; idx += 1)
{
  lane_sum += values[idx];
}

// contribute this lane's sum to the total sum
AtomicAddEval64(sum_ptr, lane_sum);
LaneSync();
LaneSyncU64(&sum, 0);

And finally, we output the sum value:

if(LaneIdx() == 0)
{
  printf(”Sum: %I64d\n”);
}

Closing Thoughts

The concepts I’ve shared in this post represent what I feel is a fundamental shift in how CPU code can be expressed, compared to the normal single-core code all programmers are familiar with. Through small, additional annotations to code—basic concepts like LaneIdx(), LaneCount(), and LaneSync()—all code can contain the information necessary to be executed wide, using multiple cores to better take advantage of serial independence.

The same exact code can also be executed on a single core, meaning through these extra annotations, that code becomes strictly more flexible—at the low level—than its single-core equivalent which does not have these annotations.

Note that this is still not a comprehensive family of multithreading techniques, because it is strictly zooming in on one unique timeline of work, and how a single timeline can be accelerated using the fundamental multi-core reality of modern machines. But consider that programs often require multiple heterogeneous timelines of work, where one lane group is not in lockstep with others, and thus should not prohibit others from making progress.

But what I appreciate about the ideas in this post is that they do not unnecessarily introduce extra timelines. Communication between two heterogeneous timelines has intrinsic, relativity-related complexity. Those will always be necessary. But why pay that complexity cost everywhere, to accomplish simple multi-core execution?

I’m aware that, for many, these ideas are old news—indeed, everyone learns different things at different times. But in my own past programming, and when I look at the programming of many others, it seems that there is an awful lot of overengineering to do what seems trivial, and indeed what is trivial in other domains (like shader programming). So, for at least many people, these concepts do not seem well-known or old (even if they are in some circles and domains).

In any case, the concepts I’ve shared in this post have been dramatically helpful in improving my ability to structure multi-core code without overcomplication, and it seemed like an important-enough shift to carefully document it here.

I hope it was similarly helpful to you, if you didn’t know the concepts, or if you did, I hope it was nonetheless interesting.

If you enjoyed this post, please consider subscribing. Thanks for reading.

-Ryan

Subscribe now

Read more

My talk focused on some of my work on the RAD Debugger over the past few years—specifically, that on the architecture of the debugger’s evaluation and visualization systems.

In my view, visualization is near the top of the priority list for debuggers, but most debugger material focuses on the implementation of core building blocks of debugger-debuggee interaction, like process control, stepping, breakpoints, and so on. This is understandable, because debuggers are an enormously vast problem space. But, seeing this as one of a debugger’s topmost priorities, and as something unique about debuggers that I’d spent a lot of time on, I tried to cover as much as possible about debugger visualization engine architecture within a ~1.5 hour window.

As it turns out, there’s a lot of ground to cover, so in the talk I walk through the basics of process control, debug information, and the compilation and interpretation pipeline a debugger contains for evaluation, and I follow that by digging a bit more deeply into the pipeline for visualizing evaluations in a variety of ways, from building an “infinite” watch window tree, to graphical visualizers:

I hope everyone enjoys the talk, and finds it interesting or useful!

-Ryan

Subscribe now

As I stated in part 1, “debugger” wouldn’t have been my name choice for, well, debuggers, because it understates the full scope of their functionality. Debuggers are programs for interactive program runtime analysis. Now that I’ve unpacked what a program actually is, I can dig into what “analysis” means.

Debug Events

Debuggers execute as processes which receive information about another executing process, from an operating system’s kernel. In one way or another, a kernel will associate a debugger’s process with some other process. The debugger is said to be “attached” to this other process. This other process can be called the “debuggee” or “target” process.

When a debugger is attached to a process, it can receive information about notable events in that process’ execution, like:

• When a process is created, and details about that process.

• When a thread is created, and details about that thread.

• When a module is loaded, and details about that module.

• When a thread is named, which thread was named, and the contents of that name.

• When a thread encounters an exception, like a “trap”, or memory violation, which thread encountered this exception, and at which instruction address this exception occurred.

• When a thread logs a debug string, which thread logged it, and the contents of that string.

• When a process exits.

• When a thread exits.

• When a module is unloaded.

These are called debug events.

The kernel is able to report these events to a debugger because these events are first reported to the kernel. This is either because: (a) these events are caused by a program’s direct interaction with the kernel, like its calling of LoadLibrary on Windows causing a module to be loaded, or (b) because the kernel configures the hardware to interrupt execution and report information to the kernel when certain events occur.

In the latter case, this is done through mechanisms like x86’s interrupt descriptor table, which encodes a table of code addresses—the beginning addresses of a number of “interrupt handlers”. The CPU—upon encountering specific error conditions (or “exceptions”—not to be confused with exceptions in high-level languages)—will execute code at one of these addresses. It selects an entry in the table using a numeric code, which represents whatever error condition was encountered.

This system is used to implement virtual address spaces, as I previously described.

When a virtual address fails to map to a physical address using a page table, a page fault is raised. On x86, the code for this fault happens to be 0x0E—this code is used to select a specific interrupt handler from the interrupt descriptor table. The CPU will jump execution to the associated interrupt handler, which is supplied by the kernel. Thus, the fact that the code accessed a non-physically-mapped address is first reported to the kernel.

In this case, one possibility is that the kernel swapped the physical storage for some address to disk, but the virtual address is completely legal for the program to access.

Another possibility is that the virtual address was not legal for the program to access. This is a situation in which this information would be reported to a debugger, if one is attached to the thread’s owner process. If a debugger is not attached, then the kernel has a number of other options—for example, it might simply kill the process, or it might want to launch a debugger and attach it to the process (a feature known as “Just-In-Time Debugging”), so that the state of the process can be inspected.

There are a number of design decisions a kernel might make in how it chooses to report this information to a debugger. For example, if one thread encounters an exception, do the process’ other threads continue executing? Or, if that thread is simply reporting a debug event which doesn’t necessarily halt execution—like a new module being loaded, or a debug string being logged—does that thread (and, still, the others in the process) continue executing?

The model used by Windows is to enable the debugger to interleave its logic with the debuggee process. Thus, when an event is reported to a debugger, all threads in the associated process will stop being scheduled, until the debugger has signaled to the kernel that the process should resume.

The model used by Linux is similar, in that the debugger (in Linux terminology, the “tracer”) can interleave its work with the debuggee (the “tracee”), although attachment happens at thread-granularity, rather than process-granularity.

While not literally the case, it’s as if the debuggee calls into the debugger, in order to understand how to proceed. The kernel facilitates this by interrupting the debuggee, transferring execution to the debugger, and then when the debugger says the word, execution is transferred back to the debuggee.

There is a further layer of minutiae to this problem, in that the kernel can also decide whether or not to allow a single debugger to be attached to multiple processes, rather than a single process. Windows does allow this, for example. And thus, extra design decisions apply. If one thread in one process reports a debug events, we know that process halts, but do the threads within other debuggee processes also halt?

While working on the RAD Debugger, I deduced from experimentation that Windows does not, in fact, automatically suspend other debuggee processes, when one debuggee reports a debug event. This is idiosyncratic with the behavior at thread granularity, which suspends all threads in a process, if one reports a debug event. For the RAD Debugger, I decided to implement the process-granularity suspension myself, such that threads across processes behaved identically to threads within processes. This simplifies the debugger design (and thus the user interface and experience), as well, because there is only a single state which determines whether or not debuggees are executing, rather than a per-process state. Other designs, however, are possible.

Debug Event APIs

Because the information provided to the debugger about the debuggee(s) is, in essence, a sequence of events, it’s natural to expose it through a blocking event loop API, as Windows does, through its WaitForDebugEvent and ContinueDebugEvent APIs:

for(DEBUG_EVENT evt = {0};
    WaitForDebugEvent(&evt, INFINITE);
    ContinueDebugEvent(evt.dwProcessId, evt.dwThreadId, DBG_CONTINUE))
{
  // process `evt`
}

Linux’s primary debugger API—ptrace— is more granular. It’s not built directly for this event-loop structure, and as I mentioned earlier, it is a thread-granularity API rather than a process-granularity API, but it can be used to implement the same concept. It’s also used for a number of other operations—like many other Linux APIs, it’s like a Swiss Army knife.

With the above Windows API usage, either the debugger’s event loop is executing, or the debuggee process is executing. They do not simultaneously execute.

Note that, in a full debugger, there are codepaths which should execute simultaneously with the debuggee—for example, that which repeatedly builds and renders a user interface, and allows visualization of previously-received debug events, or any information which can be collected while the debuggee executes. In practice, these codepaths are moved to separate threads:

But I’ll cover more about that in a later post.

Reading Non-Event Debuggee Information

While debug events help inform the debugger of changes to a debuggee across time, they do not contain the full set of information that a debugger needs to read about a debuggee.

A debugger may also need to read memory from arbitrary addresses within a debuggee’s address space. Remember that virtual address spaces exist per-process—in other words, a debuggee’s address space is not a debugger’s address space, so it isn’t as simple as reading from an arbitrary pointer.

Given our understanding of virtual address spaces, it would easily be doable for an operating system to map the same physical storage into different address spaces—in this case, being the debugger’s and debuggee’s address spaces.

In practice, this is a much heavier weight operation than necessary, as it requires a virtual address space allocation, whereas debugger’s often simply want to read sub-page-size regions of memory, at one moment in time (rather than holding a persistent mapping to a region of memory in a child process across stretches of time).

Windows provides the means to do this with its ReadProcessMemory function:

BOOL ReadProcessMemory(
  [in]  HANDLE  hProcess,
  [in]  LPCVOID lpBaseAddress,
  [out] LPVOID  lpBuffer,
  [in]  SIZE_T  nSize,
  [out] SIZE_T  *lpNumberOfBytesRead
);

We can notice that—confusingly—the lpBaseAddress parameter, which encodes the address within hProcess from which data should be copied into lpBuffer, is a pointer. But pointers only refer to addresses within the virtual address space of the process in which they’re used. So this pointer can never be dereferenced, actually—it is merely an address-sized package, for encoding a virtual address in hProcess. Super weird! But, aside from that, this is fairly straightforward.

Linux provides the same functionality through pread, although that API less-confusingly uses an integer offset.

A debugger may also need to read thread register values—on Windows, this is provided through, mainly, the GetThreadContext API, although the exact APIs and how they’re used vary by underlying architecture. On Linux, this is provided through the ptrace API also (remember how I said it’s like a Swiss Army knife?).

A Simple Debugger Event Loop

To concretize this information, let’s build a simple Windows debugger program which launches a single process, attaches to it, and logs information about debug events that it sees.

We can do this with the following APIs:

• CreateProcessA — to launch a process, and automatically attach to it.

• WaitForDebugEvent — to sleep our debugger loop, until a debug event occurs.

• ContinueDebugEvent — to continue the attached debuggee after we’re done processing a debug event’s data, and to signify whether or not the debugger has handled an exception.

First, we can use CreateProcessA to launch a process—if we use the DEBUG_PROCESS flag in our argument for dwCreationFlags, Windows will automatically attach our process as the debugger for the process we create.

// sample_debugger.c

#include <windows.h>

int main(int argument_count, char **arguments)
{
  // launch process, attach
  char *cmd_line = arguments[1];
  STARTUPINFOA startup_info = {sizeof(startup_info)};
  PROCESS_INFORMATION process_info = {0};
  CreateProcessA(0, cmd_line, 0, 0, 0, DEBUG_PROCESS, 0, 0, &startup_info, 
                 &process_info);
  // ...
  return 0;
}

Then, using WaitForDebugEvent and ContinueDebugEvent, we can build our debug event loop:

// loop for debug events
for(DEBUG_EVENT evt = {0};
    WaitForDebugEvent(&evt, INFINITE);
    ContinueDebugEvent(evt.dwProcessId, evt.dwThreadId, DBG_CONTINUE))
{
  // ...
}

Finally, if our debuggee process terminates, we’ll want to exit our loop early, before calling ContinueDebugEvent:

// loop for debug events
for(DEBUG_EVENT evt = {0};
    WaitForDebugEvent(&evt, INFINITE);
    ContinueDebugEvent(evt.dwProcessId, evt.dwThreadId, DBG_CONTINUE))
{
  // ...
  if(evt.dwDebugEventCode == EXIT_PROCESS_DEBUG_EVENT)
  {
    break;
  }
}

Next, all we need to do is dig into the structure of DEBUG_EVENT, log some information for each DEBUG_EVENT our loop iterates over, and hook this up to an actual program.

Let’s begin by getting everything up and running. First, I’ll just stub out the logging code:

for(DEBUG_EVENT evt = {0};
    WaitForDebugEvent(&evt, INFINITE);
    ContinueDebugEvent(evt.dwProcessId, evt.dwThreadId, DBG_CONTINUE))
{
  printf("Received a DEBUG_EVENT\n");
  fflush(stdout);
  if(evt.dwDebugEventCode == EXIT_PROCESS_DEBUG_EVENT)
  {
    break;
  }
}

And I’ll hook it up to a trivial program:

// sample_debuggee.c

#include <windows.h>

int main(int argument_count, char **arguments)
{
  OutputDebugStringA("Hello, Debugger!\n");
  return 0;
}

Both of these programs can be built with MSVC, with the following commands:

cl /nologo /Zi sample_debugger.c
cl /nologo /Zi sample_debuggee.c

I can then run sample_debugger.exe, parameterizing it with the sample debuggee program:

sample_debugger.exe sample_debuggee.exe

And the output is the following:

Received a DEBUG_EVENT
Received a DEBUG_EVENT
Received a DEBUG_EVENT
Received a DEBUG_EVENT
Received a DEBUG_EVENT
Received a DEBUG_EVENT
Received a DEBUG_EVENT
Received a DEBUG_EVENT
Received a DEBUG_EVENT
Received a DEBUG_EVENT
Received a DEBUG_EVENT
Received a DEBUG_EVENT

Not very useful! But now we can fill in the details by reading through the documentation for DEBUG_EVENT. I won’t exhaustively do so for the purposes of this post, but let’s fill in some basics. First, let’s log a string encoding which dwDebugEventCode each event contains:

// kind code -> name
char *evt_kind_name = "[unknown event kind]";
switch(evt.dwDebugEventCode)
{
  default:{}break;
  case CREATE_PROCESS_DEBUG_EVENT:{evt_kind_name = "CREATE_PROCESS";}break;
  case EXIT_PROCESS_DEBUG_EVENT:  {evt_kind_name = "EXIT_PROCESS";}break;
  case CREATE_THREAD_DEBUG_EVENT: {evt_kind_name = "CREATE_THREAD";}break;
  case EXIT_THREAD_DEBUG_EVENT:   {evt_kind_name = "EXIT_THREAD";}break;
  case LOAD_DLL_DEBUG_EVENT:      {evt_kind_name = "LOAD_DLL";}break;
  case UNLOAD_DLL_DEBUG_EVENT:    {evt_kind_name = "UNLOAD_DLL";}break;
  case EXCEPTION_DEBUG_EVENT:     {evt_kind_name = "EXCEPTION";}break;
  case OUTPUT_DEBUG_STRING_EVENT: {evt_kind_name = "OUTPUT_DEBUG_STRING";}break;
}

// log
printf("Received a %s DEBUG_EVENT\n", evt_kind_name);
fflush(stdout);        

If we run this, we’ll see the following:

Received a CREATE_PROCESS DEBUG_EVENT
Received a LOAD_DLL DEBUG_EVENT
Received a LOAD_DLL DEBUG_EVENT
Received a LOAD_DLL DEBUG_EVENT
Received a EXCEPTION DEBUG_EVENT
Received a OUTPUT_DEBUG_STRING DEBUG_EVENT
Received a LOAD_DLL DEBUG_EVENT
Received a LOAD_DLL DEBUG_EVENT
Received a CREATE_THREAD DEBUG_EVENT
Received a LOAD_DLL DEBUG_EVENT
Received a EXIT_THREAD DEBUG_EVENT
Received a EXIT_PROCESS DEBUG_EVENT

Many of these are self-explanatory—the initial CREATE_PROCESS, the EXIT_PROCESS, and surely that OUTPUT_DEBUG_STRING corresponds with our sample debuggee’s call to OutputDebugStringA. But let’s clear up some perhaps confusing details (to the degree that we can). My knowledge is limited, since not all of this behavior is documented (or the documentation hasn’t been easy for me to find), so some of my answers will be unsatisfying, but I’ll do my best.

What is that EXCEPTION event?

This is a mystery to me, but I can conject. Through experimentation, it seems that—no matter what—Windows generates this exception after a process is launched, seemingly always after all initial modules (those required by the executable’s imports) are loaded. If I had to guess, this is to mark the end of the loader’s work—to signify to debuggers that no further modules will be loaded before the program’s actual code is executed. I’m not sure if this is the true explanation, but marking this exception as processed (by passing DBG_CONTINUE to ContinueDebugEvent) reliably works.

How is the only CREATE_THREAD event seen after our OUTPUT_DEBUG_STRING event, which was caused by the main thread?

This is due to some unfortunate idiosyncrasies and asymmetries in the Windows debug event API. Almost all threads generate CREATE_THREAD events, and almost all modules generate LOAD_DLL events—except the main thread and main module. These—the creation of the main thread and the loading of the main module—are implied by the CREATE_PROCESS event. So, before the OUTPUT_DEBUG_STRING event is encountered by the debugger event loop, the main thread was actually created.

This API choice unnecessarily bifurcates codepaths which must apply to all modules, or to all threads. This is why, in the RAD Debugger codebase, we designed our own event structure, which we produce by converting the event information within DEBUG_EVENTs—with this small extra conversion cost, we can simulate extra module-load or thread-creation events, given a CREATE_PROCESS event, such that all threads and all modules are represented by their own events. Thus, all per-module and all per-thread code can be trivially unified.

If that CREATE_THREAD event has nothing to do with the main thread, why is a thread created, then exited, before the program exits?

This seems to have something to do with the C runtime implementation. If we adjust our program to control its own low level entry point, and to remove its dependency on the C runtime, then our debug event log becomes much simpler.

Adjusting our debuggee program to this:

void WinMainCRTStartup(void){}

Results in our debug event log being:

Received a CREATE_PROCESS DEBUG_EVENT
Received a LOAD_DLL DEBUG_EVENT
Received a LOAD_DLL DEBUG_EVENT
Received a LOAD_DLL DEBUG_EVENT
Received a EXCEPTION DEBUG_EVENT
Received a EXIT_PROCESS DEBUG_EVENT

Which cleans up three of the DLLs, and that strange thread spawned when our debuggee is about to exit. Thus, we can deduce that the thread—for one reason or another—is introduced to do something for the MSVC C runtime implementation.

Which DLLs are being loaded?

To do this, let’s continue inspecting the definition of DEBUG_EVENT. The documentation for LOAD_DLL_DEBUG_EVENT informs us that, if a DEBUG_EVENT’s dwDebugEventCode member matches LOAD_DLL_DEBUG_EVENT, then the union within DEBUG_EVENT is to be interpreted as a LOAD_DLL_DEBUG_INFO structure. That structure contains an hFile member, which is a HANDLE to the executable image file. We can use Windows’ GetFinalPathNameByHandleA API to produce the executable image’s full path. If we log that, we’ll know which DLLs are being loaded.

// extract module path
char module_path_buffer[256] = {0};
char *module_path = 0;
if(evt.dwDebugEventCode == LOAD_DLL_DEBUG_EVENT)
{
  GetFinalPathNameByHandleA(evt.u.LoadDll.hFile, module_path_buffer, sizeof(module_path_buffer), 0);
  module_path = module_path_buffer;
}

// log
printf("Received a %s DEBUG_EVENT", evt_kind_name);
if(module_path) { printf(" (%s)", module_path); }
printf("\n");
fflush(stdout);

This produces:

Received a CREATE_PROCESS DEBUG_EVENT
Received a LOAD_DLL DEBUG_EVENT (\\?\C:\Windows\System32\ntdll.dll)
Received a LOAD_DLL DEBUG_EVENT (\\?\C:\Windows\System32\kernel32.dll)
Received a LOAD_DLL DEBUG_EVENT (\\?\C:\Windows\System32\KernelBase.dll)
Received a EXCEPTION DEBUG_EVENT
Received a EXIT_PROCESS DEBUG_EVENT

Recall that, in part 2, I covered the sort of work that an operating system’s loader might do, when loading modules:

Every LOAD_DLL event also encodes at which address a module is loaded. We can log that too:

// extract base address
U64 base_vaddr = 0;
if(evt.dwDebugEventCode == LOAD_DLL_DEBUG_EVENT)
{
  base_vaddr = (U64)evt.u.LoadDll.lpBaseOfDll;
}

// log
printf("Received a %s DEBUG_EVENT", evt_kind_name);
if(module_path) { printf(" (%s)", module_path); }
if(base_vaddr)  { printf(" (0x%I64x)", base_vaddr); }
printf("\n");
fflush(stdout);

Received a CREATE_PROCESS DEBUG_EVENT
Received a LOAD_DLL DEBUG_EVENT (\\?\C:\Windows\System32\ntdll.dll) (0x7ffff2af0000)
Received a LOAD_DLL DEBUG_EVENT (\\?\C:\Windows\System32\kernel32.dll) (0x7ffff1970000)
Received a LOAD_DLL DEBUG_EVENT (\\?\C:\Windows\System32\KernelBase.dll) (0x7ffff0520000)
Received a EXCEPTION DEBUG_EVENT
Received a EXIT_PROCESS DEBUG_EVENT

I recommend studying the documentation of DEBUG_EVENT, to see what further information it contains, and how it might be used. For instance, with this basic structure and very few extensions, we can start answering questions like:

• My program crashed while using a pointer—was it reading from, or writing to that pointer? What was the pointer’s value?

• A thread hit a trap instruction—at what address was that trap instruction?

• What name has the program assigned to each of the created threads?

The next step in building useful debugger functionality lies in accumulating and using information from a history of received debug events. A debugger can treat debug events as deltas to apply to its own data structure, which mirrors a debuggee’s process structure. This way, a debugger can keep track of handles, names, or addresses for every thread and module.

Such a data structure comes in handy—for example, in implementing features like:

• Given an address referenced by an EXCEPTION debug event, and a history of LOAD_DLL events, determine which module that exception occurred within, and correlate it with its name.

• Use ReadProcessMemory to parse information out of a loaded executable image—since we know its address in the debuggee’s address space—like the path to its debug information file.

• Given a thread, read the value of its instruction pointer register, and determine from which module it’s executing code.

Debugger-to-Debuggee Interaction

Debug events are used to send information, debuggee-to-debugger. But I opened this post by stating that debuggers are for interactive runtime analysis. And true interactivity requires both debuggee-to-debugger, and debugger-to-debuggee information flow.

Kernels provide a few extra mechanisms for this. I stated earlier that they support reading memory from debuggees, and reading thread registers from debuggees. They also provide mechanisms for writing memory into debuggees (on Windows, WriteProcessMemory, on Linux, pwrite), and writing thread registers into debuggees (on Windows, SetThreadContext, on Linux, ptrace).

An additional mechanism available to debuggers is the selection of which debuggee threads will be scheduled. This is often exposed through a “thread freezing” feature in debuggers:

In the case of Linux, because ptrace is a thread-granularity API, this is a natural extension of ptrace functionality—the debugger can simply decide to not resume execution of specific threads. On Windows, this functionality is provided through SuspendThread and ResumeThread, which control a per-thread counter. When this counter is nonzero, a thread will not be scheduled. When it is zero, the thread is legal to be scheduled by the kernel.

The kernel just providing the ability to write memory, write registers, and suspend or resume threads leads to a massive explosion of potential features, of which debuggers can take advantage. But we’ll dig into exactly how a debugger can take advantage of these features in a later post.

On Current Operating System Debugger APIs

The peculiarities—of which there are many—of the Windows or Linux debugger APIs are not particularly relevant, which is why I am not exhaustively exploring them in these posts. The more important part is the set of operations that they facilitate—the effects which they may be used to create.

I encourage readers to not be too concerned about these peculiarities. For the RAD Debugger, because we’ve built the debugger to be easily portable, and thus did not want to couple our debugger functionality tightly to specific peculiarities of—say—the Windows debugging API, we built our own debugging API abstraction, which is much simpler for us to use in implementing the debugger. This API is implemented as needed on each target platform.

Furthermore, I encourage readers to not be too intimidated by these APIs. There is no magic here—at some level, the following API must be implementable on each platform, in order for debuggers (as we know them) to work at all:

DebugEvent GetNextDebugEvent(...)
U64 ReadProcessMemory(Handle process, U64 addr, void *out, U64 size)
U64 WriteProcessMemory(Handle process, U64 vaddr, void *in, U64 size)
U64 ReadThreadRegs(Handle thread, void *out)
U64 WriteThreadRegs(Handle thread, void *in)

As a concrete example, here are the associated APIs in the RAD Debugger codebase:

DMN_EventList dmn_ctrl_run(Arena *arena, DMN_CtrlCtx *ctx, DMN_RunCtrls *ctrls)
U64 dmn_process_read(DMN_Handle process, Rng1U64 range, void *dst)
B32 dmn_process_write(DMN_Handle process, Rng1U64 range, void *src)
B32 dmn_thread_read_reg_block(DMN_Handle handle, void *reg_block)
B32 dmn_thread_write_reg_block(DMN_Handle handle, void *reg_block)

The details can get a bit messy, but I encourage a spirit of resolve, because there are some amazing possibilities unlocked when the implementation is right!

I’ve now covered all of the basic building blocks that kernels provide for debuggers. As surprising as it may seem, this is all of the machinery we need from the kernel for the vast majority of common debugger features. But we’ll dig more into how a debugger can take advantage of this machinery next time!

If you enjoyed this post, please consider subscribing. Thanks for reading.

-Ryan

Subscribe now

From day one using a modern home computer, users are exposed to the concept of a program. Support for separate programs is, after all, the main value-add of multitasking operating systems. But—if we take a peek under the hood—a program is a high-level term which refers to many lower level mechanisms and concepts, and it isn’t obvious from the outset how they’re all arranged.

To unpack debuggers—programs which analyze the execution of other programs—it’s important that we first unpack the concept of a program, so that we’re familiar with the details of programs that a debugger must contend with.

Programs are the virtualized equivalent of cartridges for an old video game console, like the Nintendo Entertainment System. The NES didn’t have a multitasking operating system, and it only executed a single program while it was turned on—whatever one was stored on the cartridge that the player installed.

In this context, the program executing on the system had full availability to all of the system’s resources. There was no code running of which the program couldn’t be aware.

Programs, in the context of a multitasking operating system, are a bundle of mechanisms used to approximately provide the same thing virtually as the NES provided to the program stored on the cartridge physically. Of course, multitasking operating systems also provide ways for these programs to communicate and interact (that is indeed the point), but at some level they must still exist independently, as different physical cartridges do.

Because programs, unlike cartridges, can be executing on the same chip at the same time, and thus contend for the same resources, there are many additional software concepts that operating systems use to virtualize independent program execution:

• A virtual address space — A range of virtual addresses, for which the platform provides a mapping to physical addresses. Programs are built to interact with virtual addresses, which are entirely independent from addresses in other virtual address spaces. Virtual address spaces can be much larger than, for example, physical RAM limitations.

• A thread of execution — A bundle of state which is used to initialize the CPU to coherently execute a sequence of instructions. Threads of execution are scheduled by the platform, such that many threads can execute on a small, fixed number of cores.

• An executable image — A sequence of bytes encoding data in a platform-defined format, to encode executable machine instructions, as well as relevant headers and metadata. An independent code package’s non-live representation—a blueprint for execution.

• A loader — The part of an operating system responsible for parsing executable images—blueprints for execution—and instantiating them, so that the code encoded in the images may be actually executed.

• A module — The loaded equivalent of an executable image. One process can load several modules, although a process is always initialized by the loading of one specific module (the initial executable image). Modules can be both dynamically loaded and unloaded.

• A process — An instance of a live, running program. Instantiated by the platform’s loader using the initial executable image to determine how it’s initialized, and what code is initially loaded. The granularity at which operating systems assign virtual address spaces. The container of several modules, and threads of execution.

Let’s unpack all of this.

Virtual Address Spaces

A range of virtual addresses, for which the platform provides a mapping to physical addresses. Programs are built to interact with virtual addresses, which are entirely independent from addresses in other virtual address spaces. Virtual address spaces can be much larger than, for example, physical RAM limitations.

Whether it’s through the easy or hard way, all programmers learn about pointers. When I first learned about pointers, I understood them as being used to encode integers, with the integers being addresses, which address bytes within memory, in linear order. Address 0 comes before 1, which comes before 2, and so on.

In other words, I was under the impression that physical memory, and its relationship to addresses, was structured like this:

This is a fine mental model to begin with. But it isn’t accurate.

When many independent programs execute on a single machine, it isn’t difficult to imagine one of them getting an address wrong. In fact, sometimes, it feels like “getting addresses wrong” is the only thing anybody talks about these days. If all of these programs shared a single memory space, this could easily lead to one program stomping over data that another program is using. It could also lead to, for example, a malicious program—let’s call it ryans_game.exe—reading information from chrome.exe, browsing a page from chase.com with all of your sensitive information on it. This is purely hypothetical!

Virtual address spaces are used to mediate between different programs accessing the same resource—physical memory. Addresses can be understood as integers, and as such, they are linearly ordered, and they do each refer to sequential bytes—but these bytes are sequential in virtual address space, not in physical memory.

Virtual address spaces are implemented with a mapping data structure known as a page table. Page tables can be used to translate a virtual address to a physical address. They can then be used directly by the CPU in order to do address translation. For instance, if a CPU core were to execute a mov (move) instruction, to load 8 bytes from address 0x1000 into a register, then before issuing a read from physical memory, the CPU would first treat 0x1000 as a virtual address, and translate it into a physical address, which might be completely different—like 0x111000.

“Page tables” are called as such, because they map from virtual to physical addresses at page-size granularity. A system’s page-size varies—on an x64 Windows PC, it’ll be 4 kilobytes. On an iPhone, it’ll be 16 kilobytes. Operating systems also expose larger page sizes under some circumstances.

This means the relationship between physical memory and an address—as used by a program, as a virtual address—looks more like this:

If a virtual address cannot be mapped to a physical address, then a “page fault” exception is issued by the CPU, and execution is interrupted. If this is done by a program’s code, then execution will be transferred to the operating system’s code, which can take measures to address the cause of the exception and resume, or do whatever else it deems appropriate.

This provides a great deal of flexibility to operating systems. An operating system can move memory allocated by one program to disk—what’s known as “paging out”, or “swapping out”—if it expects that memory to not be accessed in the near future. It can then use that physical memory for more frequently accessed addresses, in any of the active virtual address spaces. If a page fault occurs when code attempts to access memory which has been paged out, then the operating system can simply page that memory back in, and resume execution. Thus, even though hundreds—if not thousands—of programs can be executing at once, the operating system can make much more efficient use of physical memory, given its analysis of which addresses in which spaces are needed, and when. This is critical in building operating systems which can support the execution of many programs, where all programs are contending for the same physical hardware.

It also provides a great deal of flexibility to programs, as it can be used to implement virtual address spaces which are much larger than physical memory. Nowadays, nearly every consumer CPU—from phones, to game consoles, to PCs, to laptops—is a 64-bit processor. For PCs and laptops running on 64-bit CPUs, the CPU and operating system normally provide a 48-bit address space. On some server systems, it is larger, and on some mobile and console platforms, it is smaller.

Taking a 48-bit address space as an example—48 bits allow the representation of 2⁴⁸ different values (each bit multiplies the number of possible values by 2). Since each value refers to a different potential byte, that is enough address space to refer to 256 terabytes.

To understand this further, let’s dissect the “page table” data structure a bit more.

First, let’s assume a 48-bit address space, and a 4 kilobyte page size (the usual configuration on x64 Windows systems). As I said, page tables map from virtual to physical addresses at page-size granularity. Because of our 4 kilobyte page size, we can infer that the bottom 12 bits of any address are identical for both a virtual address and a physical address (2¹² = 4096 = 4 kilobytes).

This leaves 36 remaining bits, of each address, to map from virtual to physical addresses. These bits are used to index into several hierarchical levels, within the page table—it is actually a hierarchical data structure, despite its name, which sounds like it implies a flat table. To understand why, imagine, first, a naïve page table implementation, which simply stores a 64-bit physical address, for each value in this 36-bit space. This, unsurprisingly, would require an unrealistically large amount of storage. Instead, we can notice that the page table need only map virtual addresses which have actually been allocated. At the outset, none are allocated. When a virtual address space allocation is made, a hierarchical data structure allows the page table implementation to only allocate nodes in the hierarchy which are actually touched, by that one allocation.

Each node in the hierarchy can simply be a table of 64-bit addresses which point to children nodes (or, at the final level, it can store each page’s physical address). If each node is a 512-element table, and each element is a 64-bit address (8 bytes), then each node requires 4096 bytes, which is our page-size!

Because 2⁹ = 512, we can slice our 36-bits into 4 table indices—each 9 bits—and use that to traverse the page table. The first 9 bits indexes into the first level, the next into the second, the next into the third, the next into the fourth—the fourth provides the base address of the containing page of our address, and then the bottom 12 bits can be used as an offset from that base.

For each virtual address space, the operating system manages this page table structure. Before the operating system prepares the CPU to execute code for one program, it can supply this table, such that the CPU can appropriately issue memory reads and writes to physical addresses for the appropriate virtual address space. The end result is that each program can, in effect, live in its own universe of virtual addresses, as if it had access to the entire system’s memory space, and if that memory space far exceeded the limitations of a system’s random access memory (RAM) capacity.

Threads Of Execution

A bundle of state which is used to initialize the CPU to cohesively execute a sequence of instructions. Threads of execution are scheduled by the platform, such that many threads can execute on a small, fixed number of cores.

Beyond a page table, a CPU core requires other information to coherently execute code. For instance, it requires the “instruction pointer” (or “program counter”)—this is a register, which stores the virtual address of the next instruction which should execute, in a given instruction stream. After each instruction is executed, the value in this register is updated to reflect the base address of the next subsequent instruction. On x64, this is known as the rip register.

When using a debugger, you’ll often see golden arrows, pointing to lines of source code or disassembly. This directly visualizes the location of the instruction pointer.

There are several other registers, used for a variety of purposes, including general purpose slots for computations. The state of all such registers is called a “register state”, or “register file”. One register state is paired exclusively with one instruction stream, from one program—a register state should only change if a single instruction stream performs work which causes it.

But a CPU has only a fixed number of cores, be it 1, 2, 4, 8, 12, 16, 32, and so on—yet operating systems support a much larger number of programs executing simultaneously. Or, at least, it seems like they execute simultaneously.

The operating system implements this illusion—of hundreds if not thousands (if not more—unfortunately…) programs running simultaneously on a small, fixed number of cores—by scheduling work from these programs. One CPU core will perform work for one program, for some period of time—it will be interrupted, and the operating system can make the decision to schedule work from another program, for example.

A thread of execution is the name given to the execution state for one instruction stream. Each contains one register state, which includes the instruction pointer, and thus a stream of instructions—among whatever other state each operating system deems appropriate.

In other words, operating systems do not just schedule programs—they schedule threads. When an operating system schedules a thread, it incurs a “context switch”—this is the process of storing the CPU core state for whatever thread was executing to memory, and initializing that core to execute work for the thread which will execute.

Executable Images

A sequence of bytes encoding data in a platform-defined format, to encode executable machine instructions, as well as relevant headers and metadata. An independent code package’s non-live representation—a blueprint for execution.

On Windows, you’ll find executable images stored on the filesystem with a .exe, or a .dll extension. These files are stored in the Portable Executable (PE) format. The difference between .exe and .dll is that the former is used to signify that an executable image is a viable initial module for a process, whereas the latter is used to signify that an executable image is only to be loaded dynamically as an additional module for a process.

On Linux systems, there is a similar structure—executable images are stored on the filesystem (the extension convention varies—sometimes there is no extension for the equivalent of Windows’ .exe, sometimes there is a .elf extension, and for the equivalent of Windows’ .dll, the extension is generally .so). These files are stored in the Executable and Linkable Format (ELF).

When I say “these files are stored in” a particular format, what I mean is that the associated operating system’s loader expects files in that format. In order to produce code which can be loaded on a platform out-of-the-box, one must package that code in the format which is expected by that platform.

It’s not in this series’ scope to comprehensively dissect either the PE or the ELF formats. But to justify the definition and concepts I’ve provided, let’s investigate the PE format using a simple example.

First, consider the following code:

// sample.c

void WinMainCRTStartup(void)
{
  int x = 0;
}

This can be built with the following command, using the Visual Studio Build Tools:

cl /nologo /Zi sample.c /link /NODEFAULTLIB /INCREMENTAL:NO /SUBSYSTEM:WINDOWS

This command will produce an executable image, containing machine code. This machine code could be disassembled (for instance, using a debugger)—that would show something like this:

sub rsp, 0x18              ; - push 24 bytes onto the stack, for locals
mov dword ptr [rsp], 0x00  ; - set the 4 bytes we are using of the stack
                           ;     for `x` to 0
add rsp, 0x18              ; - pop the 24 bytes we pushed off the stack
ret                        ; - return to the caller of our main function

Even if we know nothing else about the PE format, we do know that these instructions need to be encoded somewhere in the file. We can identify how these are encoded using a disassembler tool as well, which should have an ability to visualize the machine code bytes which were parsed to form each instruction:

The above image shows the disassembled instructions in the RAD Debugger, as well as the bytes from which they were parsed. If you took a look at the disassembly yourself, and were confused by the add [rax], al instruction everywhere surrounding the actual code, the code bytes also clear that mystery up—that is simply the instruction one obtains when parsing two sequential zero bytes.

Given the above, we know that the generated machine code is encoded with 16 bytes. Each byte can be represented with two hexadecimal digits:

48 83 ec 18 c7 04 24 00 00 00 00 48 83 c4 18 c3

If we look at the generated EXE with a memory viewer, we can, indeed, find this sequence of bytes.

We know that this sequence of bytes is the primary “payload”—the actual program code. Everything else in the file is used to either instruct the loader how to correctly prepare a process for this code to execute, or to associate various metadata with the code.

For example, if you scan around the file, you’ll find the full path to the debug information file (PDB) for the executable image.

The executable image also must store data to which code refers. We can see this by inserting a recognizable pattern into a global variable:

// sample.c

static char important_data[] = {0x12, 0x34, 0x56, 0x78, 0x90};

void WinMainCRTStartup(void)
{
  int x = important_data[0];
}

We can also easily find the corresponding data in the PE file:

If you investigate formats like PE or ELF more closely, what you’ll find is that various categories of data—code, initialized global variables—are separated into sections. Each section has a name, which is also encoded in the file.

In PE, .text, for example, encodes all of the machine code (rather than, well, text…). .data stores data for initialized global variables. .rdata stores the same, but is separated to be allocated in read-only pages, such that code cannot modify that data.

.pdata and .xdata encode information about how, given a procedure, one may unwind a thread, to—for example—produce a call stack, which is simply recreating the information of which functions called which other functions in order to get a thread of execution to its current point in a procedure. But we’ll dig into that topic in a later post.

.edata and .idata encode information about exports and imports, respectively, which associate strings (“symbol names”) with locations in the file. An export is used by DLLs, for example, to export functions which can be dynamically loaded by name, by code in an executable or other DLL, and called. An import is used by either executables or DLLs to specify functions from other modules with which it must dynamically “link”.

When implementing a debugger, the precise details of formats like PE and ELF become relevant—but this should be a sufficient introduction for those unfamiliar with the basics.

Loaders & Modules

A loader — The part of an operating system responsible for parsing executable images—blueprints for execution—and instantiating them, so that the code encoded in the images may be actually executed.

A module — The loaded equivalent of an executable image. One process can load several modules, although a process is always initialized by the loading of one specific module (the initial executable image). Modules can be both dynamically loaded and unloaded.

More than a debugger, a loader must be highly aware of executable image format details, because it has the task of parsing those images and making preparations, such that the code contained in the executable image can be executed.

A loader executes when a program is initially launched, or when actively-executing code requests to dynamically load another image—for instance, via LoadLibrary (Windows) or dlopen (Linux).

To understand this, let’s build a toy executable image format, and write our own loader, which parses our format, rather than PE or ELF.

Consider the following code, from earlier:

// sample.c

void WinMainCRTStartup(void)
{
  int x = 0;
}

And its disassembly:

> c:/devel/sample/sample.c
> {
{48 83 ec 18}              sub rsp, 0x18
> int x = 0;
{c7 04 24 00 00 00 00}     mov dword ptr [rsp], 0x00
> }
{48 83 c4 18}              add rsp, 0x18
{c3}                       ret

Our toy format can have a simple header, at the beginning of the image, containing the following values, in order:

• An 8-byte signature, denoting that the file is in our format—must always be 54 4f 59 45 58 45 00 00—encoding the ASCII text TOYEXE, followed by two zero bytes.

• An 8-byte offset into the file, encoding where in the file all readable-and-writable global data is stored—the “data section”

• An 8-byte offset into the file, encoding where in the file all read-only global data is stored—the “read-only data section”

• An 8-byte offset into the file, encoding where in the file all executable data is stored— the “code section”

Each section size is determined by taking the next subsequent section offset (or the file size, in the case of the final section), and subtracting from it the section offset. If sections contain no data in any case, they will simply have the same offset as the next section.

Given this simple format, our full executable file for the simple example program can be encoded with the following bytes:

{54 4f 59 45 58 45 00 00} (magic)
{20 00 00 00 00 00 00 00} (read/write data offset)
{20 00 00 00 00 00 00 00} (read-only data offset)
{20 00 00 00 00 00 00 00} (executable data offset)
{48 83 ec 18 c7 04 24 00 00 00 00 48 83 c4 18 c3} (executable data)

In this case, our data sections are completely empty, because no global data is used by the code. Every section offset begins at offset 0x20 (or 32 bytes) into the file—or, directly after the header. The executable data section, being the final section, occupies the remainder of the file.

Our “loader” can define the format’s header with the following structure:

typedef struct ToyExe_Header ToyExe_Header;
struct ToyExe_Header
{
  U64 magic;       // must be {54 4f 59 45 58 45 00 00}
  U64 rw_data_off; // read/write
  U64 r_data_off;  // read
  U64 x_data_off;  // executable
};

It can begin by reading the file, and extracting the header:

// open file, map it into the process address space
HANDLE file = CreateFileA(arguments, GENERIC_READ, 0, 0, OPEN_EXISTING, 0, 0);
U64 file_size = 0;
if(file != INVALID_HANDLE_VALUE)
{
  DWORD file_size_hi = 0;
  DWORD file_size_lo = GetFileSize(file, &file_size_hi);
  file_size = (((U64)file_size_hi) << 32) | (U64)file_size_lo;
}
HANDLE file_map = CreateFileMappingA(file, 0, PAGE_EXECUTE_READ, 0, 0, 0);
void *file_base = MapViewOfFile(file_map, FILE_MAP_ALL_ACCESS, 0, 0, 0);

// extract the header
ToyExe_Header header_stub = {0};
ToyExe_Header *header = &header_stub;
if(file_base && file_size >= sizeof(*header))
{
  header = (ToyExe_Header *)file_base;
}  

It can then allocate memory, big enough for the image’s data, and copy the file’s contents into that address range.

// allocate memory for all executable data - ensure it is all
// writeable, executable, and readable
void *exe_data = VirtualAlloc(0, file_size, MEM_RESERVE|MEM_COMMIT, 
                              PAGE_EXECUTE_READWRITE);

// copy file's data into memory
CopyMemory(exe_data, file_base, file_size);

Given the header’s information encoding where in the executable data the code is stored, we can now call into that code directly:

// call the code
void *x_data = (U8 *)exe_data + header->x_data_off;
((void (*)())x_data)();

And it actually works! But there is, as you might expect, more minutiae to this in practice.

Per-Section Memory Protections

In this example, I’ve allocated all of the executable’s data with identical memory protections—all bytes in the executable’s data are legal to read, write, and execute. The point of having different sections at all is to organize data by how it will be accessed and used, so that—for instance—our “read-only data section” can actually be read-only (such that, if any code were to attempt writing to it, it would fail).

Because memory protections are assigned at page granularity, each individual section, after it’s loaded by our toy loader, must be at least one page size (so that we can assign appropriate protections to each section), and it must be aligned to page boundaries. But, were we actually designing a format, to require all sections be at least one page size (which is normally 4 kilobytes, if not larger), at least in the executable image itself (as it’s stored in the filesystem), can be fairly wasteful for smaller executables.

Instead of our loaded image being a flat copy from the image file:

We can adjust it to being an expansion for each section to page granularity, and a copy:

To do this, we can introduce a distinction between unloaded sections (that which stored in an executable image), and loaded sections (that which are loaded in memory, when a process executes). So far, our toy format has one notion of “offset”. We can break that down into two notions of offset, into two separate spaces—”unloaded space” and “loaded space”. These are generally called “file space” and “virtual space” (where “virtual” refers to a process’ “virtual address space”). Thus, instead of one type of offset, we can have file offsets, or virtual offsets. In code, instead of using off as our naming convention, we can explicitly encode which space we’re working within, by prefixing a name with either f or v. For example, “offsets” can now be referred to as either foff for file offsets, or voff for virtual offsets.

This distinction of unloaded and loaded images is the reason for the separation between the terms image and module. We call the image the “cold” equivalent of the data, and we call the module the “hot”—the loaded—equivalent of the data.

We can rewrite our header structure as follows, to encode both the locations of section data within the image, and to encode where the section data should be arranged within memory before execution:

typedef struct ToyExe_Header ToyExe_Header;
struct ToyExe_Header
{
  U64 magic;   // must be {54 4f 59 45 58 45 00 00}
  U64 padding; // (round up to 64 bytes)
  U64 rw_foff; // read/write (file)
  U64 r_foff;  // read (file)
  U64 x_foff;  // executable (file)
  U64 rw_voff; // read/write (virtual)
  U64 r_voff;  // read (virtual)
  U64 x_voff;  // executable (virtual)
};

Our test program can then be adjusted to the following bytes, assuming 4 kilobyte pages:

{54 4f 59 45 58 45 00 00} (magic)
{00 00 00 00 00 00 00 00} (padding)
{20 00 00 00 00 00 00 00} (read/write data file offset)
{20 00 00 00 00 00 00 00} (read-only data file offset)
{20 00 00 00 00 00 00 00} (executable data file offset)
{00 10 00 00 00 00 00 00} (read/write data virtual offset)
{00 10 00 00 00 00 00 00} (read-only data virtual offset)
{00 10 00 00 00 00 00 00} (executable data virtual offset)
{48 83 ec 18 c7 04 24 00 00 00 00 48 83 c4 18 c3} (executable data)

And our loader can be adjusted to perform the “expansionary copy”:

// unpack f/v dimensions of each section (and header)
U64 fdata_hdr_size = sizeof(*header);
U64 fdata_rw_size  = header->r_foff - header->rw_foff;
U64 fdata_r_size   = header->x_foff - header->r_foff;
U64 fdata_x_size   = file_size - header->x_foff;
U64 vdata_hdr_size = fdata_hdr_size;
U64 vdata_rw_size  = header->r_voff - header->rw_voff;
U64 vdata_r_size   = header->x_voff - header->r_voff;
U64 vdata_x_size   = fdata_x_size;

// round up virtual sizes to 4K boundaries
vdata_hdr_size+= 4095;
vdata_rw_size += 4095;
vdata_r_size  += 4095;
vdata_x_size  += 4095;
vdata_hdr_size-= vdata_hdr_size%4096;
vdata_rw_size -= vdata_rw_size%4096;
vdata_r_size  -= vdata_r_size%4096;
vdata_x_size  -= vdata_x_size%4096;

// calculate total needed virtual size, allocate
U64 vdata_size = (vdata_hdr_size + vdata_rw_size + vdata_r_size + vdata_x_size);
U8 *vdata = (U8 *)VirtualAlloc(0, vdata_size, MEM_RESERVE|MEM_COMMIT, 
                               PAGE_READWRITE);

// unpack parts of virtual data
U8 *vdata_hdr     = vdata + 0;
U8 *vdata_rw      = vdata + header->rw_voff;
U8 *vdata_r       = vdata + header->r_voff;
U8 *vdata_x       = vdata + header->x_voff;

// unpack parts of file data
U8 *fdata         = (U8 *)file_base;
U8 *fdata_hdr     = fdata + 0;
U8 *fdata_rw      = fdata + header->rw_foff;
U8 *fdata_r       = fdata + header->r_foff;
U8 *fdata_x       = fdata + header->x_foff;

// copy & protect
CopyMemory(vdata_hdr, fdata_hdr, fdata_hdr_size);
CopyMemory(vdata_rw, fdata_rw, fdata_rw_size);
CopyMemory(vdata_r, fdata_r, fdata_r_size);
CopyMemory(vdata_x, fdata_x, fdata_x_size);
DWORD old_protect = 0;
VirtualProtect(vdata_hdr, vdata_hdr_size, PAGE_READONLY, &old_protect);
VirtualProtect(vdata_rw, vdata_rw_size, PAGE_READWRITE, &old_protect);
VirtualProtect(vdata_r, vdata_r_size, PAGE_READONLY, &old_protect);
VirtualProtect(vdata_x, vdata_x_size, PAGE_EXECUTE, &old_protect);

And—since it’s easy to notice that this is getting rather repetitive for each section—we can table-drive this “expansionary copy”. Doing so will eliminate most of the per-section duplication:

// gather all information for all boundaries between all sections (& header)
struct
{
  U64 foff;
  U64 voff;
  DWORD protect_flags;
}
boundaries[] =
{
  {0,               0,               PAGE_READONLY},
  {header->rw_foff, header->rw_voff, PAGE_READWRITE},
  {header->r_foff,  header->r_voff,  PAGE_READONLY},
  {header->x_foff,  header->x_voff,  PAGE_EXECUTE},
  {file_size,       0,               PAGE_READONLY},
};
U64 region_count = (sizeof(boundaries)/sizeof(boundaries[0]) - 1);

// calculate vsize for all regions
U64 vdata_size = 0;
for(U64 idx = 0; idx < region_count; idx += 1)
{
  U64 vsize = (boundaries[idx+1].foff - boundaries[idx].foff);
  vsize += 4095;
  vsize -= vsize%4096;
  vdata_size += vsize;
}
boundaries[region_count].voff = vdata_size;

// allocate; iterate regions, do copy & protect
U8 *vdata = (U8 *)VirtualAlloc(0, vdata_size, MEM_RESERVE|MEM_COMMIT, 
                               PAGE_READWRITE);
U8 *fdata = (U8 *)file_data;
DWORD old_protect = 0;
for(U64 idx = 0; idx < region_count; idx += 1)
{
  CopyMemory(vdata + boundaries[idx].voff,
             fdata + boundaries[idx].foff,
             (boundaries[idx+1].foff - boundaries[idx].foff));
  VirtualProtect(vdata + boundaries[idx].voff,
                 (boundaries[idx+1].voff - boundaries[idx].voff),
                 boundaries[idx].protect_flags, &old_protect);
}

It’s not much shorter, but all work for all expanded and copied sections has been deduplicated. In order to adjust this for a larger number of different sections, only the boundary table must change.

Imports & Exports

Now that we have a basic structure for loading our image format, let’s consider how a loader is used. As I’ve stated, an executable image is loaded whenever a program is launched, or when an executing program requests to dynamically load another image (for instance, through LoadLibrary or dlopen).

We’ve already explored the first case—program launching—as that will consist of simply beginning execution of the program. With our toy loader, we can just immediately execute the code after we’ve loaded it:

// call the code
void *vdata_x = (U8 *)vdata + header->x_voff;
((void (*)())vdata_x)();

This makes the assumption that the first instruction stored is our entry point. If we ever wanted that to not be the case—as “real” executable image formats do—then we can simply store a virtual offset for the desired entry point within the image’s header.

But in the case of dynamic loading, our loader’s job is not to merely begin executing at a single point in some code. Our load instead must load the image, and prepare for dynamic lookups of potentially many named entry points. On Windows, the usage code for this looks something like:

HMODULE foo_library = LoadLibraryA("foo.dll");
void (*foo_function)(void) = GetProcAddress(foo_library, "foo_function");
foo_function();

To facilitate this path, our executable image format must associate a number of names—like foo_function—with specific virtual offsets in the executable data section. This concept is known as an executable image’s exports, and it can be straightforwardly encoded as a set of pairs of names and virtual offsets.

There’s a symmetric concept known as imports, which function as a fast path for the manual lookup of functions from a loaded executable image like the above code. On Windows, the usage code for that looks something like:

__declspec(dllimport) void foo_function(void);
#pragma comment(lib, "foo.lib")
foo_function();

Both explicitly loaded (via GetProcAddress on Windows, or dlsym on Linux), and implicitly loaded (via __declspec(dllimport) on Windows, which is more automatic on Linux toolchains) functions are called through a double indirection. To perform an actual function call, first the CPU must follow the address of the pointer in which the loaded address is stored, then it can use whatever the value of that pointer as the address of the function to call.

In the above example with explicit loading, the address of the loaded function is stored in the explicit foo_function function pointer variable. In the above example with implicit loading, the address of the loaded function is implicitly stored in hidden state, as an implementation detail.

When an executable image has imports, in order for the loading of that image to succeed, the associated imports must be dynamically linked. This will be done automatically, as opposed to the program code manually calling—for instance—LoadLibrary or GetProcAddress.

Address Stability & Relocations

Machine code contained in an executable image can be hardcoded to refer to specific addresses. But as you’ll notice, in our toy loader, we don’t control which address at which our module for an image is placed in memory. We call VirtualAlloc to allocate memory for our module data, and whatever address it returns, we use that. Of course, we can request that VirtualAlloc place our allocation at a specific address, but that is not necessarily guaranteed to succeed.

This means that if we, for instance, had an image with instructions which referred to a global variable’s absolute address, they would only be valid given that the image is loaded at a particular address.

In principle, a loader could guarantee a fixed virtual address for a program’s initially loaded executable image. They don’t. But in any case, that cannot generally be true, because images can be loaded or unloaded dynamically, and they are not built to be aware of which other images are loaded simultaneously. Thus, they must be dynamically arranged—each image’s code should be able to operate correctly, irrespective of where its loaded module equivalent is placed in memory.

In many cases, especially nowadays, addresses are encoded as relative to some offset into code, in which case they’re always valid, irrespective of which runtime address at which the module is loaded. But, nevertheless, there still exist mechanisms for code to be hardwired to refer to specific addresses. In such cases, an executable image also contains relocations, which encode locations within the executable image which must be reencoded after the base address of the loaded image is determined at runtime.

It is the loader’s job to iterate these relocations, and patch in the appropriate addresses given the only-then-available knowledge of where the image’s loaded data is actually stored.

And Finally, A Process

An instance of a live, running program. Instantiated by the platform’s loader using the initial executable image to determine how it’s initialized, and what code is initially loaded. The granularity at which operating systems assign virtual address spaces. The container of several modules, and threads of execution.

We’ve covered everything we need to sketch out a definition of a process—a running program.

Each process is the owner of some number of threads, and some number of modules. It is the owner of a single virtual address space.

Threads and virtual address spaces are, in a sense, orthogonal concepts—threads are used to virtualize CPU cores, virtual address spaces are used to virtualize physical storage—the process is the concept which binds them together.

When a program is launched, a process is created, an executable image is loaded to produce an initial module, and an initial thread is spawned.

An operating system’s scheduler then considers that process’ main thread as a viable candidate for scheduling. When it’s scheduled, the program executes.

When a debugger is used to analyze another program, it does so at process granularity. It is registered by the operating system as being attached to a process. When a debugger is attached to a process, the operating system enables additional codepaths, which report information about that process’ execution to the debugger’s process. If that information includes addresses, they’re reported as virtual addresses, within the address space of the process to which the debugger is attached.

But that’s enough for now! We’ll dig into exactly what kind of information an operating system reports to a debugger process, how it can do so, and how the debugger can interact with the debugged process, next time.

If you enjoyed this post, please consider subscribing. Thanks for reading.

-Ryan

Subscribe now

Debuggers exist at the intersection of many parts of the computing ecosystem—they must contend with intricate details of kernels, compilers, linkers, programming languages, and instruction set architectures.

My familiarity with debuggers has improved my programming abilities, the utility of debuggers in my day-to-day programming, and my general knowledge of computing. Back in January, the RAD Debugger—the project I work on full-time—was open sourced to the public, to mark the start of its open alpha phase. I’ve been working on the debugger, or the technology on which it depends, for almost four years full-time now. The project has taught me an enormous number of lessons, through exposure to an enormous number of problems. There is still a lot of work to do, and so I expect it will continue to do so, for many years to come.

But perhaps most importantly, debuggers are an intricate piece of the puzzle of the design of a development platform—a future I become more interested in every day, given the undeniable decay infecting modern computing devices and their software ecosystems.

To emphasize their importance, I’d like to reflect on the name “debugger”. It is not a name I would’ve chosen, because it can give the impression that a debugger is an auxiliary, only-relevant-when-things-break tool. Of course, a debugger is used to debug—which is why it was named as such—but it is also enormously useful to analyze working code’s behavior, and to verify code’s correctness, with respect to the expectations of the code.

A good debugger provides clear and insightful visualizations into what code is doing. As such, they are also enormously useful educational tools—for beginners and experts alike—because they make what is normally opaque, visible. They provide these features by dynamically interacting with running programs—as such, they can also dynamically modify code. At the limit, this approximates (or employs) JIT-compilation and hot-reloading, making traditional compiled toolchains have much more runtime flexibility for developers.

For these reasons, “debugger” is much too special-purpose of a name for the full set of capabilities that debuggers actually provide—they offer glimpses into the lower level inner-workings of a computer. If one designed a computing system from scratch, they might not ideally be independent from the operating system itself. Instead, perhaps the same capabilities could simply be provided through first-class visualization and dynamic execution adjustment features that the operating system naturally exposes. But that is a topic for another day.

I hope this sheds light on the imbecility of Internet debates about the utility of debuggers—for example, where one might find comments like, “I don’t need debuggers, because I can just use printf”, or “I don’t need debuggers if I can statically guarantee correctness”. It’s akin to suggesting that someone does not benefit from vision, because they can feel their way around with a mobility cane, or read text through Braille. Even though mobility canes and Braille are obviously good inventions for people who can’t have vision, that doesn’t somehow imply that vision isn’t an obvious benefit, or that it isn’t obviously preferable. Similarly, even though logging and static verification are obviously good inventions for programs or circumstances which cannot be easily debugged at runtime, or when those things are simply preferable in context, that doesn’t somehow imply that actively visualizing the runtime execution of programs through a debugger isn’t an obvious net benefit, or that it isn’t obviously preferable in many cases. To suggest otherwise in either case is absurd. The more useful debuggers become, the shorter the iteration loop of the programmer, the more efficient software production becomes, and the more trivially that programmers can obtain true from-first-principles reasoning about their code.

Given their importance for both the present and future, and their utility to myself (and thus perhaps readers), I’m writing a series explaining and documenting debugger architecture.

In this series of posts, I’ll cover the following topics:

• The Anatomy Of A Running Program — On the concepts involved in a running program. What happens, exactly, when you double click an executable file, or launch it from the command line, and it begins to execute?

• Debugger-Kernel Interaction — On how kernels collect and expose information about program execution to debuggers, like “debug events”, encoding changes like thread creation & destruction, dynamic module loading & unloading, low level exceptions being hit by threads, and more; or like the reading & writing of memory & thread registers, or like the suspension and resumption of threads.

• CPU Features & Debuggers — On the features that CPUs commonly expose for debuggers, like interruption instructions, debug registers, single-stepping mode, and more.

• Instruction-Level Stepping & Breakpoints — How a debugger can use kernel and CPU features to implement instruction-level stepping and breakpoints.

• Debug Info & Toolchains — On the traditional compilation and linking pipeline, how “debug info” is produced, what it contains, and how it helps debuggers implement higher level features, which can correlate a program’s state with source code or language constructs.

• Evaluation — On evaluating expressions using an expression language and “location info” and “type info”—two parts of “debug info”.

• Breakpoints — On how “breakpoints” are implemented, from address breakpoints, symbol breakpoints, source code location breakpoints, to conditional breakpoints and processor (or data) breakpoints.

• Stepping — On the various “stepping” features in debuggers, from the barebones single-instruction stepping, to disassembly stepping, to source line stepping, all with variants like “step into”, “step over”, and “step out”, all while correctly handling multithreaded programs.

• Unwinding — On “unwinding”, which is how a debugger determines a thread’s current “call stack”, and is able to correctly evaluate values from all scopes in a call stack.

• Graphical Debugger Multithreaded Architecture — On the structure of a graphical debugger, which employs the aforementioned features and concepts, and exposes them through a real-time interactive interface.

• The Watch Window, & General-Purpose Data Visualization — On the traditional “watch window” graphical debugger interface, and how it may be extended to support general-purpose data visualization.

• …and anything else I stumble across while writing that I think would be appropriate to cover!

In discussing these topics, I’ll try to abstract over platform and architectural details when possible, but I’ll base my writing on my experience from working on the RAD Debugger, which has begun its journey as a Windows, user-mode, x64 debugger (although it’s not finishing its journey as merely that). I’ll also use the RAD Debugger to demonstrate certain concepts and features concretely.

When I am explicitly relying on that context, I’ll do my best to state so, but I’ll also do my best to extrapolate to more generalized information when appropriate, as many of the concepts have similar if not identical analogs on other platforms, and so I feel the knowledge is quite generalizable.

I hope you’re excited to come along for the ride, and demystify debuggers for yourself!

If you enjoyed this post, please consider subscribing. Thanks for reading.

-Ryan

Subscribe now

There’s a pervasive idea in the software industry that’s often posed to defend abstractions and software layers which are easily (and often) used to generate tremendous amounts of bad code, in favor of programmer efficiency in generating that bad code.

When I say “bad code”, I don’t refer to what the programmer sees in his text editor. I am talking about the actual code—that is, what the user’s CPU executes.

The argument goes: given these abstractions or layers—maybe they’re language features, or programming paradigms—programmers can be more efficient in generating bad code which accomplishes the same meaningful work as the alternative case, where the programmer does not use these abstractions or layers, and perhaps produces good code, but was more inefficient in doing so. The claim—the hypothesis, or as it is often used, the assertion—is that these paradigms, abstractions, languages features, and so on, are more efficient from a business perspective, at the expense of efficiency from a software quality perspective.

I recently saw this argument manifest in practice when I (finally) watched gingerBill’s video breakdown of the written discussion between Casey Muratori and Robert “Uncle Bob” Martin, on the technical merits of Clean Code. It’s important to understand that “Clean Code” doesn’t refer to “clean code”. Capital-C “Clean” describes code, in the same way that “Democratic People’s Republic” describes North Korea. It is a label. It has as much information content as a person’s name. Bill correctly identifies this and says that it’d really more appropriately be called “Mr. Martin Style”.

In the video (which I recommend watching), Bill mostly doesn’t focus on the technical side of the discussion. And with good reason—as Bill demonstrates, Robert Martin didn’t focus on the technical side of the discussion either. Mr. Martin instead opted to debate rhetorically—Bill correctly likens this to being akin to a form of political maneuvering. Throughout the debate, no technical questions are satisfyingly answered by Mr. Martin—if that is what you’re looking for, you’ll be disappointed. It isn’t named “Clean” (rather than Mr. Martin Style) by accident—this convenient word choice is often used by “Clean Code” proponents to mask unfalsifiable assertions, in order to justify certain “Clean Code” guidelines, by subtly shifting between “clean” and “Clean” throughout a discussion.

Casey and Bill both did an excellent job, but in both the written discussion and the video breakdown, I kept having a recurring thought that I didn’t see directly expressed by anyone, and that’s why I’m writing this post.

You could spend several careers researching programmer productivity. The interface of programming is, of course, tightly intertwined with programmer psychology, the programmer’s ability to reason, and the programmer’s ability to easily express computational effects. Not all interfaces are equally efficient for programmers to use. Not all interfaces can have equally efficient implementations. Barely investigating this subject leads to the conclusion that some interfaces are simultaneously better for the programmer, and better for the CPU, than others. Programmer efficiency and CPU efficiency are not always mutually exclusive—one does not necessarily need to always pick one.

Unfortunately, serious investigations of these topics are seemingly never carried out by “Clean Code” proponents. Claims of improved programmer efficiency, at the expense of CPU efficiency, are mere assertions.

But in this post, I want to zoom out from those investigations. I want to give “Clean Coders” the benefit of the doubt. Let’s say that we are operating in a space where we must trade CPU productivity for programmer productivity. Let’s say that we even have tools which we know trade CPU productivity for programmer productivity. Never mind the fact that these premises are never proven to hold—let’s say we find a case where they do hold.

I’d like to explain why, in that context, it still doesn’t make sense to overwhelmingly prefer the comfort and generative productivity of the programmer writing the code. Even when I’ve given “Clean Coders” everything they could possibly ask for, everything they attempt to assert, there is still a strong argument in favor of software performance, at the expense of initial programmer performance. Part of this argument has to do with the overall quality of the computing system, but it is not strictly altruistic—there is also a strong business case for software performance, even with heightened upfront programming cost. As such, this argument does not require that you have business inclinations, nor altruistic inclinations—you could love, or hate, the concept of a free market with private property. In all such cases, this argument still applies.

The fundamental mistake I see as responsible for entirely destabilizing the argument in favor of programmer code-writing efficiency (at the expense of software efficiency) is in an incorrect model of the software ecosystem.

Programmers Run Their Own Code

The implicit model being used to justify the tradeoff assumes a single software producer, which produces the code, and a single customer, which executes the code:

But this model ignores the everyday reality of programming. The reality is that the producer must also execute the code, as they develop, debug, and test it. Only after they’ve done that process innumerable times, they will submit the software to the user to be executed.

Already in this model, bad software execution time has already caused a slowdown in programmer productivity. Some amount of the programmer’s efficiency in reading or writing the code has already been counterbalanced by the software’s inefficiency, because the programmer is blocked on executing his own software. The slow software execution time has thus had an impact on the programmer’s iteration cycle time.

To continue justifying the choice to prefer programming methodologies which promote fast generation of bad code, one must prove that this degradation of programmer iteration cycle time does not outweigh the improvement in programmer efficiency in generating the code. Given that code can often be written once, but tested, debugged, and tweaked hundreds if not thousands of times, this fact is important to demonstrate. But it is never demonstrated.

Customers Are Programmers

But this problem gets worse. Who is the software producer? What are they making? Who is the software consumer? What are they using the software for? It seems like “Clean Code” arguments are often made assuming a very narrow definition of these two entities—for example, as used in the original written discussion, some payroll software, used by a business to pay employees. But what is the common reality in the software world?

Let’s take an example from the real world—from my own life, in fact. I work on a debugger project full-time. As part of my work on that project, I need to use a file explorer. The Microsoft Windows built-in file explorer is an incredibly poor quality experience for a large number of reasons—probably related to “Clean Code”, in fact—which is why I’m excited for my friend Vjekoslav’s file explorer project, File Pilot. Vjekoslav is a programmer too, and so as part of his file explorer project work, he needs to use a debugger.

Instead of some arbitrary payroll software producer, and some arbitrary business, let’s say the entity on the left is me writing the debugger, and the entity on the right is Vjekoslav, writing his file explorer. This is a slight extension of the initial oversimplified model—it demonstrates one simple case of cyclical dependencies within the software ecosystem. This is an overwhelmingly common relationship—it regularly manifests within a single company or organization, but it extends beyond those boundaries as well.

In this context, if my software is unnecessarily slow, not only does it slow myself down before I release it, it also slows Vjekoslav down in his development. This degradation in his programming performance manifests in fewer iteration cycles for his software, which I also depend on. Because I began by making my software worse than it could’ve been, that effect has propagated outward into the computing ecosystem, and because the computing ecosystem is a cyclical graph of dependencies, that effect propagates back to me.

Slowness Propagates, Then Compounds

But this problem is worse, still, than I’ve described. A more detailed graph of the software ecosystem is more complex than the earlier examples. The software ecosystem is a vast cyclic network of independent entities producing software, and relying on other software.

Poor performance, introduced at points in this network, propagate outwards, because “depending on” software means executing that software. Execution time of an entity’s dependency is, one way or another, factored into the production of the dependent itself. This may be literally execution time in the dependent software, if it relies on a poorly constructed library—or it may be in programming resources, if the programmer relies on a poorly constructed tool. Whether it blocks the programmer, or the software, it has a cost.

The longer the dependency chain becomes, the more likely these effects will be multiplied or otherwise duplicated—and the more likely that these effects will back-propagate to the dependent software. Not to mention that, the longer this dependency chain becomes, the less likely that anyone will understand what the problem truly is, or how to fix it.

This reality has explanatory power in describing the state of modern software. Despite vast improvements in computing hardware capabilities, some software often performs dramatically worse than its contemporary equivalent running several decades ago, on hardware that was sometimes a thousand times slower, with a thousand times fewer resources. This is because poor performance in this network does not stop at one edge in the network—it is multiplied several times over.

During the initial discussion, Mr. Martin introduced several time domains. He described some problems as being within the “nanoseconds domain”, where wasting nanoseconds matters. He described the “microseconds domain”, where wasting microseconds matters. He described the “milliseconds domain”, where wasting milliseconds matters.

The problem is, however, that there is only one time domain in reality. And wasted nanoseconds add up to wasted microseconds. And those add up to wasted milliseconds. And those add up to wasted seconds. This can happen before network effects in the software ecosystem are considered, simply within one piece of software.

After network effects are considered, it is perhaps no surprise that programs now accomplish in thirty seconds what they used to accomplish in less than a second on hardware which was thousands of times less capable.

Spending software efficiency to obtain programmer efficiency may seem like an appealing idea. But I hope this post helped clarify why it isn’t that simple. The acceptance of software production methodologies like the one Mr. Martin himself promotes are in part responsible for a vast degradation of software quality, and ironically enough, decreases in programmer efficiency.

I opened this post with a quote from Mr. Martin himself: “I want the act of programming to be easier, more pleasant, and more productive for the vast majority of programmers”.

Unfortunately, the act of programming is harder, less pleasant, and less productive than ever, because of systemic excuse-making for mediocrity. Sacrificing software efficiency for programmer efficiency seems to simply result in losing both.

If you enjoyed this post, please consider subscribing. Thanks for reading.

-Ryan

Subscribe now

This is a very flexible high-level interface design. It’s a useful way to structure the user interface for programs which require a large number of integrated, but varied, user interface designs, such as the program pictured below, for which I used this basic design:

Read more

By inserting itself into the build, the codebase is not dependent on a compiler or language designer providing the right features—it simply has the space of compile-time execution available for its own computation.

This model has other many notable benefits, which compiler-executed metaprograms lack—the metaprogram can be debugged like a normal executable with normal debug information, and it can run as a native program at full speed; it does not need to execute within a bizarre pure-functional type language, or within a virtual execution environment, it is just regular code, which analyzes and produces text.

Read more

The small sample program—along with many other projects and examples, all built using arenas as the core memory management primitive—is included in the Digital Grove codebase, which is available to paid subscribers. You can subscribe below:

Subscribe now

I hope it helps to better communicate some of the ideas I’ve written about. If you enjoy the talk, please consider subscribing. Thanks for watching.

-Ryan

In my UI series, I wrote about my preference for an “immediate mode” interface for building UI. This preference came from years of attempting to build UI from scratch in various ways. When I’d use a retained mode interface, it would always feel like I was pushing sand around.

When I discovered the immediate mode approach to UI—similar in spirit to the immediate mode approach to rendering—it felt like a breath of fresh air. My UIs became not only easier to build—which is in itself not enough justification for the preference, although still important in improving iteration time—but also more robust, more easily made dynamic, and more naturally reactive to state changes in my programs.

There is a fair amount of rhetoric that suggests immediate mode interfaces are less capable than retained mode interfaces. The most common arguments are: (a) there are caching opportunities that immediate mode interfaces cannot take advantage of; (b) immediate mode interfaces must update all state for all changes, thus causing worse performance and higher battery usage; (c) immediate mode interfaces prohibit the use of user interfaces as a data structure, thus prohibiting integration with operating system accessibility features, keyboard navigation, autolayout, and so on.

Statements (a), (b), and (c) are all false. The embedded links provide compelling counterarguments to each. It is a bit tiresome that these arguments are still continuously made—asserted—considering the lack of investigation and thought behind them.

But despite that, there is still a point at which immediate mode interfaces lose their utility, and where retained mode interfaces become preferable, if not required.

As I’ve written about before, one place where this rings true is in controlling higher level, user-controlled “interface instantiation entities”—think windows, tabs, panels, and so on. In other words, things that the user might explicitly create with a “+” button, and destroy with an “x” button.

But the same is true, for example, in the implementation of an immediate mode interface with persistent “key-based” caching behavior. The implementation of the cache cannot be immediate mode—the cache is implemented with “retained mode pieces”, they are just localized to the implementation of the immediate mode interface.

Another example would be state for entities in a game level. In a usual game scenario, entities are dynamically created and destroyed depending on various gameplay conditions. Usage code of an immediate mode interface, however, is static, as code is immutable:

// every frame:
Entity("player", player_sprite, player_pos, ...);
Entity("goblin", goblin_sprite, goblin_pos, ...);
Entity("chest", chest_sprite, chest_pos, ...);

In the above example, let’s say that the gameplay involves the player defeating the goblin, in order to obtain valuables from the chest.

There are countless possible combinations of state in this scenario, accounting for all the various entity positions, entity health levels, and so on. But one important possibility is: is the goblin alive, or is the goblin dead?

Imagine that you need to encode this possibility into usage of the “immediate mode entity interface”, used above as Entity. You’d need something like the following:

// every frame:
Entity("player", player_sprite, player_pos, ...);
if(is_goblin_alive)
{
  Entity("goblin", goblin_sprite, goblin_pos, ...);
}
Entity("chest", chest_sprite, chest_pos, ...);

And what is is_goblin_alive? It’s usage-code-side state. And usage-code-side state requires a usage-code-side state machine. This state machine requires retained-mode-like mutations.

In other words, it’s impossible to escape state machines—or “retained mode interfaces”—they are the exactly correct choice in many places.

But “many” is not “all”—and the reason why immediate mode interfaces become preferable in some places is because the introduction of yet another state machine is actually a burden on usage code, rather than providing some necessary functionality.

Thinking about this led me to a useful analogy—is the system I’m writing upstream? Or is it downstream?

Downstream of what? Downstream of state machines.

Upstream systems are shallow layers in a call stack. They need control over state, and they want to describe how many computational effects flow from that state.

Downstream systems are deeper layers in a call stack. They are called into to produce computational effects from user-provided state. Their purpose is to organize the particulars of how some system functionally derives from another.

When a downstream system introduces new state machines for usage code, when those state machines merely mirror an upstream state machine, this introduces additional cruft, additional busywork for usage code, possibility for bugs, worse iteration time, and a substantially worse programming experience. This is what happens when a “retained mode system” is inappropriately introduced.

One important reason why is because there are far more—exponentially more—downstream system entry points than upstream, because every entry point can itself call into N other entry points. Thus, introducing usage-code-controlled state machines unnecessarily in downstream systems explodes the number of possibilities the usage code author must actually be concerned about.

This explains why, over time, it has become obvious that, for rendering, immediate mode interfaces are dramatically better than retained mode interfaces. This was not always the case, but this model makes it clear why—on-screen artifacts produced by rendering are exactly that: artifacts. They’re a function of some invisible state—the entire purpose of rendering is to take some invisible state in the machine, and turn it into something visible.

UI is only one step further—instead of merely visual artifacts as “output” of a system, it is also concerned with user inputs as “input” to a system. But it still, nevertheless, remains largely downstream of that system.

But immediate mode interfaces—to specify “downstream effects”—is a technique far from localized to rendering and UI.

And when truly downstream systems are correctly organized as such—providing an immediate mode interface to usage code—it becomes trivial to dynamically produce a multitude of possible downstream effects, with few upstream state changes. Every upstream state change becomes much more powerful and meaningful—it costs little code, little computation, and little work to produce a new world of effects. There is no need to inform every downstream system of the state change.

These reasons are why I’ve found the analogy useful, and why I wanted to share it. When writing a system, ask yourself—are you downstream, or are you upstream? The answer to that question can help inform you about the appropriate design for that system.

If you enjoyed this post, please consider subscribing. Thanks for reading.

-Ryan

Subscribe now

On day one, to even begin programming anything, programmers must understand how to write text files, and use an interpreter, or compiler, or assembler, to turn them into executing programs. Soon after, they must roughly understand what memory is, and how to use addresses and pointers (even if they’re working in various high level languages in which those concepts masquerade under other names). This naturally follows into a basic understanding of memory allocation. An important next step is learning to write code to operate on batches of data, rather than single elements of data, for both performance and simplicity. They can then understand how to more simply manage memory, by being organized about these batches, and performing the allocations themselves as a batch, upfront. They can become more familiar with the underlying mechanisms of the CPU, how it obtains memory, performs computations, and writes memory.

At each of these milestones, the programmer’s understanding of the underlying hardware, and how to gracefully use it, broadens. And because modern hardware offers several independently-executing processor units, one of the next most crucial milestones in this journey is writing code to execute in parallel, in multiple simultaneously-executing instruction streams, rather than just a single instruction stream. This is critical for both software performance and utility. When writing user-mode code on a multitasking operating system, like myself and most developers, this is done with multi-threading.

Many of these milestones introduce new constraints which apply pressure on code to be shaped in a particular way. If a programmer solves for only some of these constraints, it’s like building a piece of furniture and tightening one particular screw fully, before even placing all other screws. Code, much like many other things, is partly a function of its environment—that is, the constraints to which it’s regularly exposed.

Multi-threading is one such milestone. Single-threaded code can be built with a different set of constraints than multi-threaded code, and as such, it is often extremely difficult to later add the “screw” of multithreading to a mostly-fully-built piece of “furniture”. In other words, if code needs to be multi-threaded, then it should be built under those constraints as early as possible; more and more of it will need to be rewritten the later those constraints are introduced.

Sometimes, it makes more sense to willingly accept the cost of rewriting, especially if it’s difficult to predict what the correct multi-threaded design for a problem will be, and a single-threaded version must be first written to explore the problem space. I think there is often too much reluctance toward rewriting systems, and too many resources put into “reusability” (before the exact circumstances of that reuse are fully understood), but I digress.

For me, multi-threading was initially an extremely intimidating milestone to tackle, because it introduces so many of these additional constraints. It seems to add an entirely new dimension to programming, and many techniques one learns to work adequately in single-threaded space suddenly become invalid in multi-threaded space.

But as I’ve learned over the past couple of years, this doesn’t mean that, at the end of the day, it always needs to be difficult. It can instead be quite easy, after internalizing these additional constraints, and designing systems accordingly. Designing such systems can obviously be tricky and dependent on one’s own ingenuity, but multi-threading being intimidating was more a function of its opacity rather than its intrinsic difficulty. In other words, hard problems are still hard, but when I was first thinking about multithreading, there were many problems that seemed like they should be easy that were also hard.

Anytime someone writes a shader, for example, they’re doing multi-threaded programming. Shadertoy proves that this can be simple for something like tens (if not hundreds) of thousands of programmers, and it mostly happens invisibly, because the underlying primitives are well-designed. I learned that I can also design underlying primitives, for a given problem, to make multithreaded programming simple.

I’ve written before about some lessons I’ve learned which have helped me internalize these additional constraints, and design multi-threaded systems. Recently, I’ve been doing much more multi-threaded programming, and I want to share more of those lessons in this post.

It’s Not How You Synchronize—It’s How You Don’t

Generally, when someone is being taught multithreaded programming, they are taught about synchronization mechanisms—for instance, atomic CPU operations, and the higher-level abstractions that they are often used to implement, like mutexes, condition variables, semaphores, and so on. And while understanding synchronization mechanisms is important, the true task of writing multithreaded systems is not figuring out how to synchronize, it’s how to not synchronize. One of the major benefits of multi-threaded code is, obviously, that it allows multiple tasks to happen simultaneously. The more synchronization a multithreaded system requires, the more this benefit dissolves. Of course, some amount of synchronization must exist in order for two systems to work together, and at those synchronization points, getting the tricky details of synchronization mechanisms is critical—but ideally, the vast majority of multi-threaded code is executing without synchronization at all.

One intrinsic characteristic of multi-threaded code is that simultaneous reads can happen without any synchronization. Writes, on the other hand, can require synchronization (with both other writes and reads)—for one touched region in memory, it is really only ever well-defined to have a single write at a time, and thus there must be some synchronized order of writes, and reads cannot occur while a write occurs. This means either reads or non-conflicting writes lend themselves better to multi-threaded code than potentially-conflicting writes.

Thus, designing multi-threaded systems which execute cleanly in parallel requires careful attention to where reads occur, where writes—mutations—occur, and what is mutated. Writes should not conflict with other threads also performing writes as often as possible, lest the system requires additional synchronization. In other words, the mutations performed by each thread should be bucketed whenever possible.

A simple example of bucketed mutations would be when many threads allocate and mutate a local variable on the stack. This is trivially conflict-free and thus requires no synchronization, because the term “the stack” actually refers to a per-thread stack, and thus each thread mutates a completely different region in memory:

void ThreadEntryPoint(...)
{
  U64 x = 0;
  for(U64 idx = 0; idx < 1000; idx += 1)
  {
    x += idx; // will *never* conflict with other threads
  }
}

void EntryPoint(...)
{
  LaunchThread(ThreadEntryPoint, ...);
  LaunchThread(ThreadEntryPoint, ...);
}

And a simple example of non-bucketed mutations would be when many threads write into the same fixed address in memory, e.g. through a static variable:

void ThreadEntryPoint(...)
{
  static U64 x = 0;
  for(U64 idx = 0; idx < 1000; idx += 1)
  {
    x += idx; // will *always* conflict with other threads
  }
}

void EntryPoint(...)
{
  LaunchThread(ThreadEntryPoint, ...);
  LaunchThread(ThreadEntryPoint, ...);
}

In the above example, the conflicting write—the non-bucketed mutation—is not properly synchronized, and so the value of x is not reliably well-defined, and thus of much use.

Bucketing Allocations & Arenas

A common example of mutation is mutating some data to allocate memory. Because allocations are a subset of mutations, they have the same characteristics in multi-threaded systems that other mutations do. So bucketing mutations wherever possible implies the rule to bucket allocations wherever possible.

Bucketing allocations for multi-threaded purposes often leads naturally to many opportunities to bucket allocations by lifetime. As such, the arena allocator is an excellent tool for bucketing allocations.

Many people have asked me about synchronization mechanisms being baked into arena allocators, such that the lowest level arena-based allocation operation—ArenaPush—is thread-safe. This is an understandable question, because other allocators—e.g. malloc—are designed to be callable from several threads at once.

But making the basic arena operations thread-safe is putting the cart before the horse, and my reasoning for preferring read-only access and bucketed mutations in multi-threaded systems hopefully clears up why—it is assuming that the correct architecture requires synchronized access to the same arena. But the much more preferable alternative, when it is indeed possible for a given problem, is to use each arena as a per-thread bucketed allocation (and thus bucketed mutation) mechanism, such that only one thread trivially accesses an arena at a time with no synchronization.

An arena can be, of course, used along with synchronization mechanisms, such that an arena is not per-thread but rather per-data-structure, or per-hash-table-stripe, and so on, but baking in that synchronization at the lowest level is a misunderstanding of the ideal multi-threaded system, where threads execute almost entirely without synchronization (and thus almost entirely with read-only access to shared regions, or non-conflicting reads and writes). Using synchronization mechanisms with an arena is a compromise on that ideal. This is perfectly acceptable in many circumstances, since it may be for a different ideal (e.g. storing a heavy resource once, saving both memory and compute time, thus requiring synchronized access to a shared cache), but in any case, the arena implementation is more flexible, and functions well for both cases, when synchronization is user-defined, often resulting in no synchronization whatsoever.

Implicit Mutations & Work Independence

In single-threaded space, it is often convenient or useful to combine several codepaths and produce one effective codepath. But sometimes, this is done such that one low-level codepath in an effective codepath contains a mutation, and another low-level codepath in the same effective codepath contains only a read. Take the following example, from a previous post:

// retrieves value associated with `key` - if it does not
// exist inside `table`, then inserts it with initial value
// of `default_val`
Val ValFromKey(Table *table, Key *key, Val default_val);

Key key = ...;
Val val = ValFromKey(table, &key, default_val);

In that previous post, I wrote about how this kind of immediate-mode API can be extremely useful in collapsing the number of low-level codepaths required in a specific problem, particularly the number of low-level codepaths responsible for maintaining complex state machines.

But in multi-threaded systems, this kind of API requires extra analysis. The effective codepath must be treated as having the mutational properties of all of its potential low-level codepaths. Thus, the effective codepath which calls ValFromKey can only be understood as mutational of its table argument. If that table is a shared cache, perhaps shared across threads, then it has potentially conflicting mutations, thus requiring synchronization.

As I discussed earlier, this may be a perfectly suitable design—for example, if I replaced Val with TextureHandle, and ValFromKey with TextureFromKey, and this API was used to do quick lookups into a shared texture cache given a key, and I expect that mostly to consist of several in-flight reads after textures are loaded (and thus not requiring synchronization with writes), then synchronization is a perfectly reasonable trade.

But the details can change. Suppose that this effective codepath is used as a helper mechanism in two other overarching codepaths. In the first case, the call to ValFromKey is mutational in 99% of cases. In the second task, the call to ValFromKey is 100% non-mutational, and thus only doing a read-only lookup.

This manifested recently on a project I work on, where a ValFromKey mechanism was being used to build a large deduplicated hash table of strings, and the same mechanism was later being used to look up nodes in that hash table and gather information from them. The system I was working on was originally written as single-threaded code, and I was doing a pass over the system to improve its performance, particularly by moving independent streams of work to execute simultaneously, with no synchronization, on multiple threads.

Of course, in the single-threaded context, this code functions perfectly correctly—but by blurring the line between mutational writes and read-only access, it gave both a 99% mutational low-level codepath and a 100% read-only codepath the same name. Of course, if I verified that indeed the latter case was 100% read-only, I could simply call it anyways as a read-only effective codepath, but this is awfully close to “cheating”—it’d only take a small number of completely innocent changes to invisibly break the system with new race conditions, and so I found it much more reasonable to explicitly separate the “mutate explicitly” and “read-only” codepaths, and use each accordingly.

After doing so, because all allocations in the system are explicit to callers with Arena parameters, I instead have an API like the following:

void TableInsert(Arena *arena, Table *table, Key *key, Val *val);
Val *TableLookup(Table *table, Key *key);

With this style of API, the caller is in control of the allocation bucket via the Arena parameter. As such, the fact that TableLookup is read-only is explicit (as there is no such parameter), and thus guaranteed to function correctly in parallel with other codepaths also executing TableLookup.

In the aforementioned problem, that allowed the second codepath to be massively improved in performance—the previously single-threaded code doing 100% read-only lookups into a table could now be reorganized to execute completely in parallel.

Making Conflicting Writes Non-Conflicting, With Join Operations

But what about the case of TableInsert? It’s easy to understand the analysis of looking at the codepath responsible for building the table, and concluding that: each call to TableInsert mutates arena and table, which conflicts with other calls to TableInsert, and therefore all TableInserts must be synchronized.

And again, in some cases this may be completely reasonable. And in those cases, as I’ve found, it isn’t necessarily the end of the world. I’ve found that it’s often easy to implement a much cleverer synchronization mechanisms than—for instance—just throwing a lock around table and arena. For instance, if table is a hash table, instead of taking an Arena, TableInsert can take a different mechanism—I’ll call it a Guard—such that the insertion mechanism can map a (Guard, Hash) pair to an Arena and a read-write lock, where the Guard can have several Arenas and locks, subdividing the hash table. Thus, in order to actually require a writing lock, multiple threads need to not only be writing to the same table, but to the same “stripe” in the same table.

This already provides an improvement over the single lock and arena, but beyond this, it’s often well within reason to do much better than that, with carefully-designed atomic locking operations rather than the heavier-handed locking abstractions. But a full investigation of such techniques deserves its own post. Clever locking mechanisms, from what I can tell, seem like they’re useful in getting another 20%, 30%, or 40% out of code which already has synchronization built in, but they’re still much slower than code which requires no synchronization whatsoever.

But a simple tweak to the problem allows all calls to TableInsert to be bucketed, and thus non-conflicting. Instead of having a single table, I can also simply have many tables, and then have a separate “join” step, which allows me to combine the many tables into a single joined table. In many cases, a multi-threaded “build” step, with each thread operating without synchronization (and on separate arenas), followed by a “join” step, may be much more efficient than a multi-threaded “build” step which requires synchronized access to a single data structure.

Assume Table is a simple linked-list-chaining hash table:

struct Node
{
  Node *next;
  Key key;
  Val val;
};

struct Slot
{
  Node *first;
  Node *last;
};

struct Table
{
  U64 slots_count;
  Slot *slots;
};

A “join” operation for two Tables with the same slots_count can then be easily written as a linked-list concatenation for each slot.

Table *dst = ...;
Table *src = ...;
for(U64 idx = 0; idx < slots_count; idx += 1)
{
  if(dst->slots[idx].last && src->slots[idx].first)
  {
    dst->slots[idx].last->next = src->slots[idx].first;
    dst->slots[idx].last = src->slots[idx].last;
  }
  else if(src->slots[idx].first)
  {
    MemoryCopyStruct(&dst->slots[idx], &src->slots[idx]);
  }
}

If type-system-enforcement of the same slots_count is desired, then the API can be slightly tweaked to the following:

struct Table
{
  // `slots_count` is omitted - chosen/passed by user
  Slot *slots;
};

void TableInsert(Arena *arena, Table *table, U64 slots_count, Key *key, Val *val);
Val *TableLookup(Table *table, U64 slots_count, Key *key);
void TableJoin(Table *dst, Table *src, U64 slots_count);

A “join” operation can also be extended with a number of other features, like sorting each bucket to ensure deterministic results. It can also be easily parallelized, as each slot’s “join” operation is entirely independent from the joining work for all other slots. As such, the mutations for the “join” operation are bucketed.

Closing Thoughts

Similar to my feeling after learning simpler memory management techniques, I’ve learned that multi-threaded programming becomes significantly easier when I adopt a careful, grounded, and organized approach, and when I’m closely familiar with a given problem’s details, and meticulously untangle operations by their lower level properties, rather than relying purely on stories told at higher levels of abstraction. In this case, those details include reads, conflicting writes, and non-conflicting writes.

I hope this post provided similar insight to you.

If you enjoyed this post, please consider subscribing. Thanks for reading.

-Ryan

Subscribe now

This concretizes many of the ideas I’ve previously written about, and walks through the codebase’s basic building blocks for hand-rolling custom data structures which are better suited for one’s problem than—for instance—a generic data structure object in a language standard library.

Read more

Read more

Programmers are taught early to spend a great deal of time considering “errors”, and how to “handle” them. But one of the most important programming lessons I’ve learned over the past several years is to dismiss the idea that errors are special. At the bottom, the computer is a computer. It’s a data transformation machine. An error case is simply a case. Data encoding an error is simply another form of data. Irrespective of how many language features and type variants one decides to layer on top of the computer, nothing changes this fact.

In the usual educational programming journey today, this underlying reality is ignored if not quickly dismissed, and the programmer is taught to consider errors as higher level, abstract concepts, which are distinct from non-errors. But regardless of the time spent learning to “handle errors” in this way, actually handling errors never seems to become completely painless—in the best (for the programmer’s comfort—but perhaps worse for the software) case, the programmer is taught to remain blissfully unaware of the holes in their “error handling” strategies, and the consequences of that are, well, all around the modern software world, in overly-rigid or legalistic designs, unexplained failures, invisible rules which dictate successful or unsuccessful software usage, and so on.

One of the important design problems in any computing system is in robustness to a variety of conditions—including error conditions. Computers, at the bottom, are precise physical machines. If one hopes for them to interact seamlessly with other computers, or with humans, then they must fluidly adapt to error conditions, and provide pathways to gracefully recover and continue operating, such that a human could learn, adapt, and try again.

At some levels of abstraction, the concept of an “error” does arise—but as I’ve discussed before, it’s much more productive (both for the programmer and the computer) to consider the higher level manifestation of an idea as emerging from some underlying computational machinery, rather than architecting computational machinery according to some abstract idea. Things which are concretely identical ought to be organized as such. Systems which respect concrete reality will perform better, and result in fewer headaches, than those which don’t.

Once error cases are understood to be no different from normal cases, they may be understood with normal analysis. In past posts, I’ve written about codepaths—one possible trace of execution on a computer—and how code can be architected such that it produces a large combinatoric explosion of codepaths, or—preferably—such that it collapses all desired effects into as few codepaths as possible. Fewer codepaths means those codepaths receive more programming time, they’re exercised more regularly, and tuning those codepaths (for performance, for instance) will have a much larger impact.

In other words, because every new codepath represents a new possibility of code execution, it also represents a new possibility of code failure, and thus an opportunity for testing and further development work.

Error cases being simply cases is another way of saying they correspond to a subset of all possible codepaths. If error cases can gracefully flow through the same effective codepaths through which other cases flow, then handling those errors becomes nearly free. As such, the handling of those errors can come at a remarkably low cost, instead of being an annoying extra constraint which must be “handled” by the “real” code.

It’s all real code, because they’re all real cases.

In this post, I’ve gathered a set of principles and techniques which have helped me use this reality to my advantage, and helped me write much more robust and failure-resistant code, which more gracefully adapts to changing conditions.

Guarantee Valid Reads (Nil Pointers)

One way code may fail is reading or writing to a virtual address to which it doesn’t have access, causing an access violation. This might be by reading or writing to a null pointer. Because of this, code is often written in an extremely paranoid fashion, littered with ifs or asserts, in an attempt to catch invalid pointers (and ideally deduce how they arose), hopefully before software runs on a user’s machine:

Foo *foo = malloc(sizeof(Foo));
assert(foo != 0);
InitializeFoo(foo);
Bar *bar = foo->bar;
if(bar)
{
  // use `bar`
}

In simple cases this is not a major concern, but the complexity compounds for each pointer which, when read or written to, would cause an access violation. Each if, assert, or early return is representative of a multiplying by 2 of the number of possible codepaths. This leads to situations like those I’ve described in past writing, where the programmer must explicitly add code per codepath. When the set of codepaths grows exponentially with each new possible failure point, this results in slow, long, noisy, and difficult-to-predict code.

Now, some of these possible codepaths correspond to genuine failure cases. If code attempts to allocate a buffer to which it then writes, but the allocation fails, there is not really much that the rest of the code can validly do. I’ll speak more on that case later. But first, I’d like to focus on the case in which a pointer is only being read from.

The primary reason reading from pointers (prepared by other codepaths) may cause an access violation is because code preparing the pointers often makes the decision to (a) return a pointer to some valid and accessible spot, or (b) return a null pointer. In the case of (a), the pointer can be read from without an access violation. In the case of (b), it cannot.

But a simple solution is to prefer option (c) to (b)—return a pointer to a “nil struct”, rather than a null pointer, if (a) fails. This nil struct is allocated up-front and may be stored in read-only memory. Any pointers contained in the nil struct point to other nil structs of the associated type. If there are pointers to the same type, they point at the same nil struct in which they’re contained (in other words, the pointers are self-referential).

struct Node
{
 Node *first;
 Node *last;
 Node *next;
 Node *prev;
 Node *parent;
 Payload v;
};

read_only Node nil_node = {&nil_node, &nil_node, &nil_node, &nil_node, &nil_node};

Note: read_only, a macro I define in the Digital Grove codebase, can be easily implemented as expanding to compiler-specific allocation attributes—e.g. on MSVC, __declspec(allocate(".roglob")), where .roglob is a section earlier defined with #pragma section(".roglob", read)

If this pattern is adopted, then irrespective of failure or success of the codepath preparing a pointer for later use, it is guaranteed in all cases—given completion of the codepath—that the resultant pointer is readable.

This collapses all cases in which later code required two codepaths—one for a valid pointer, one for a null pointer—down into only requiring a single codepath. This simplifies the multiplicative effect that each pointer brought, down to a multiplication of one.

In other words, this (an example from a previous post):

Node *SearchTreeForInterestingChain(Node *root)
{
  // assuming `ChildFromValue` derefs its parameter.
  // obviously these `if`s can be hidden inside the
  // API too, but that doesn't change the fact that
  // they happen, nor remove the burden from *someone*
  // to check...

  Node *result = 0;
  if(root)
  {
    Node *n1 = ChildFromValue(root, 1);
    if(n1)
    {
      Node *n2 = ChildFromValue(n1, 2);
      if(n2)
      {
        Node *n3 = ChildFromValue(n2, 3);
        if(n3)
        {
          result = ChildFromValue(n3, 4);
        }
      }
    }
  }
  return result;
}

…turns into…

Node *SearchTreeForInterestingChain(Node *root)
{
  Node *n1 = ChildFromValue(root, 1);
  Node *n2 = ChildFromValue(n1, 2);
  Node *n3 = ChildFromValue(n2, 3);
  Node *n4 = ChildFromValue(n3, 4);

  // we will necessarily get here - and we're also
  // guaranteeing for all of *our* callers that they
  // can dereference this result, even if it's 'invalid'
  return n4;
}

In my experience this dramatically simplifies a lot of code, which never needed the “validity” check, but could gracefully work with the “empty” values of a nil struct.

This technique is further helped if the nil struct contains useful default values, and ensuring this is especially simple (and performant) when zero is a useful default value in all cases.

Make Zero Values Valid (Zero-Is-Initialization)

Many modern programming styles are obsessed with default values and initialization, as a property of type information. C++ added “constructors” to what otherwise would be plain-old-data structs, which are arbitrary functions that implicit run when the language can statically recognize that a type is being instantiated (e.g. via stack allocation or the new operator). The entire concept of RAII (Resource Acquisition Is Initialization)—one of the goals of C++’s constructors—is that the presence of some type instantiation offers the guarantee that the instance is initialized.

But the best—simplest, fastest, and most maintainable—code is that which never needed to be written, or to be executed (as its intended effects are already gracefully accomplished through other means). In the case of initialization, this is trivially attainable if all initialization can be simplified to zero initialization.

If zero initialization is sufficient for memory initialization, several benefits follow. First, freshly committed memory returned by modern operating system allocation APIs (e.g. via VirtualAlloc) is always already zeroed entirely, and so in that case, no additional work must happen. Second, the code for zero-initializing memory (if unable to rely on default zero-allocation of freshly allocated pages—e.g. if reallocating an already-previously-allocated chunk) is trivial, being a single memset. It’s completely generic and independent of what the underlying type is, which members it has, its size, and so on. Ultimately, zero initialization is trivial to write, trivial to execute, and trivial to maintain.

Within the context of error condition robustness, these benefits go further. When this pattern is adopted codebase-wide such that most data is zero-initialized, and also that most codepaths accept zero values, any data which fails to be constructed (due to what might be considered an “error”) will remain zero-initialized, and thus will gracefully work with the codepaths that later consume it. This applies to nil struct pointers—which, with the exception of pointers to other nil structs, are completely zero-initialized—but also other cases, like a struct allocated on an arena or the stack, returned from some API.

Zero initialization is subtle because it—despite its title—describes only a small part of “initialization” code—after all, almost all initialization code disappears. It instead mainly describes a general property of all non-initialization code—the fact that any codepath which reads from some input, could have that input made zero, and it’d behave as expected, with as-sensible-behavior-as-possible (for whatever the context is).

A trivial example of this is the String8 type, and related string processing code, in the Digital Grove codebase:

struct String8
{
  U8 *str;
  U64 size;
};

This is the “string view”, or “slice” type, used for all codepaths which read from strings. If an instance of String8 is zero-initialized, all code into which that instance is fed will interpret the instance as encoding an empty string—the code will gracefully allow str to be 0, as all operations are delimited by size (also, in this context, obviously zero).

Another, slightly more complex example is found in the Metadesk text format parser in the Digital Grove codebase. This layer contains a lexing API (which takes care of all tokenization work before the primary tree parsing pass):

struct MD_TokenArray
{
 MD_Token *v;
 U64 count;
};

struct MD_MsgList
{
 MD_Msg *first;
 MD_Msg *last;
 U64 count;
 MD_MsgKind worst_message_kind;
};

struct MD_TokenizeResult
{
 MD_TokenArray tokens;
 MD_MsgList msgs;
};

MD_TokenizeResult MD_TokenizeFromText(Arena *arena, String8 text);

The types which are returned from MD_TokenizeFromText—MD_TokenArray and MD_MsgList—are both initialized to zero. Both are processed by code which looks like the following:

MD_TokenizeResult tokenize = MD_TokenizeFromText(arena, text);
for(MD_Msg *msg = tokenize.msgs.first; msg != 0; msg = msg->next)
{
  // print out `msg` info
}
for(U64 token_idx = 0; token_idx < tokenize.tokens.count; token_idx += 1)
{
  // use `tokenize.tokens.v[token_idx]`
}

Similar to String8, this code gracefully works with zero initialization, as both for-loops above are delimited by either a null first pointer in the MD_MsgList, or by a zero count in the MD_TokenArray. So a zero initialized String8, MD_TokenArray, MD_MsgList instances are all naturally consumed as “empty”.

One of the major exceptions to zero initialization as a rule is that it’s sometimes worthwhile to compromise it for the purpose of providing nil struct pointers. But whether or not this is worthwhile depends on the case. In the above case—the MD_MsgList pointers—I don’t use nil struct pointers, because the list is processed as a batch, and as such is naturally terminated by null pointers. If I wanted the ability for code to—for example—gracefully read the first node and read the first value—or a nil struct if no such value existed—then nil struct pointers might be worthwhile. But for simple list types like this, my feeling is that the tradeoff weighs more strongly in the “don’t add more global nil structs, keep loops terminating at 0” direction, since it’d be unlikely that the list isn’t processed entirely at once.

One way in which both zero initialization and nil struct pointers can work together without conflict is when pointers are not stored in any stateful, mutable data structures, but only constructed on-the-fly (in an immediate-mode fashion) from—for instance—indices or handles stored in mutable data structures. In that case, zero indices or handles can be treated as mapping to a nil struct pointer, such that when user code does eventually resolve those indices or handles to pointers, it’ll resolve zeroes to nil struct pointers.

If You’re Going To Fail, Fail Early

I’ll now return to the “allocate buffer, write into the buffer” case. I already covered what might happen when a pointer is being read from—an API implementation can guarantee for its users that pointer reads (with some caveats—e.g. delimited by a size) are always valid. But nil structs are allocated in pages marked as read-only, and for good reason—if they weren’t, code which received a nil struct pointer and wrote to it could compromise other code which expected reads from nil structs to contain useful and empty defaults.

So, receiving a nil struct pointer and writing to it is a bug. What approach should one take, then, when using pointers for the purpose of writing?

To dig into this problem, consider the following code:

U64 *buffer = ...;
buffer[0] = 123; // ???

On the second line, is accessing buffer[0] “unsafe”?

The answer, of course, depends on what the “...” is replaced with. Contrary to popular belief, using a normal pointer isn’t a game of Russian Roulette for a program—it all depends on what guarantees have been made at this point in the code. Also contrary to popular belief, special language features do not magically make this problem go away, because depending on said guarantees, there may be a genuinely possible failure point, or there may not be.

But consider that “...” is replaced by malloc(64*1024*1024*1024):

U64 *buffer = malloc(64*1024*1024*1024);
buffer[0] = 123; // ???

There is no way around the fact that this represents a genuinely possible failure case—the software was produced with the expectation that the user’s machine would allow a 64 gigabyte buffer to be malloc’d. The software is running on a real machine, with real, fixed resources, and is using an implementation of malloc which was built with various constraints. Such an allocation may not succeed, depending on many factors.

Let’s also suppose that this 64 gigabyte buffer is completely necessary for the problem. This means that the programmer has no choice with respect to whether or not he allocates the buffer—he knows he needs it. But he does have a choice with when the buffer is allocated, and what to do if the allocation fails.

Furthermore, the user cares when the allocation happens, if the allocation fails. The user also cares about not losing progress, or state. For the user, if the allocation is going to fail, the earlier the better. This is for a fundamental design reason—human computer interaction is cyclic. The user supplies input to a compute system, the compute system produces output, which is then interpreted by a user, which the user anticipates before feeding more input into the compute system.

If the relevant output of some system is that “the operation cannot possibly succeed”, forcing the user to jump through several hoops, prepare state, and then telling them the operation cannot succeed is wasteful of the user’s time—and a hell of a lot more frustrating—than simply telling them up-front, immediately after the user triggers the operation.

Communicating such information to the user as early as possible is paramount—the software must not arbitrarily prolong the length of the “compute system cycle”.

A good rule-of-thumb I’ve found is that these potential failure points should occur in as shallow frames in the program’s call stack as possible, and I ought to exert pressure to eliminate many of these potential failure points—having them around is not free.

Take the following call stack:

If I force these failure points to occur, say, within U_EntryPoint, rather than UI_SignalFromBox, I force more pre-allocation. Before the software gets into the weeds of its operations, it guarantees it can successfully perform those operations, and that it has the required resources. This way, I guarantee more sooner, and learn more about failure points earlier. This allows me to inform the user much earlier, which keeps the compute system cycle short.

Arena allocators are a beautiful example of abiding by this rule-of-thumb without sacrificing the potentiality and upsides of dynamism. Arenas have the ability for dynamic growth—and each dynamic growth represents a possible failure point—but the shape of their allocation interface is catered to lightning fast allocations off a pre-allocated block of memory. So during development, or in shipping builds where dynamism is preferred over static requirements (or required), arenas can implement common case allocations with lightning fast allocations off a pre-allocated block of memory, but fall back to dynamic growth paths. But when tightening up code such that it’s nearly bulletproof, the same arena can be adjusted to work completely off of a fixed up-front reservation size in all expected cases—and the decision about the static requirements of this fixed up-front reservation can be made later, when there’s enough information to responsibly make it.

In any case, after the guarantee has been made that this sort of buffer allocation has succeeded, accessing the buffer (assuming the access is in-bounds) is guaranteed to be valid. All code subsequent the guarantee may take advantage of this fact—ultimately, this means that there are no pointers to check, no asserts to write, no paranoia—the pointer is valid, guaranteed. All codepaths which access the buffer need not be bifurcated for the possibility of failure—they’re already by definition within the success path, so no further combinatoric explosion of codepaths occurs.

Prefer Fewer Types—Or, Prefer AND over OR

Another facet of modern programming techniques and tooling is the addiction to the generation of new types. Many programmers have come under the spell of deeply desiring the structure of the program itself to be encoded, somehow, within a language’s type system, with some vague expectation that the code—after all the types are established—will practically write itself. This addiction is also facilitated by a programmer’s fixation on perfectly compressing all instances of all types—all data must be encoded in types in which all parts of that type are likely used, and wasted space is not acceptable.

I think this perspective misses a crucial detail of the relationship between types and code. The fact of the matter is, the larger the number of types, the larger the number of required codepaths. It’s not obvious how exactly these numbers are related, but the general correlation is clear—types define the texture which two codepaths must agree on in order to “click together”. Introducing more types necessarily requires more codepaths in order to “click into” those textures.

One way in which the addiction to generating types surfaces in the context of “error handling” is the presumption that the resultant type of some codepath is either a successful value produced by the codepath, or it’s an error. Ultimately, the instinct in this style of programming is to build a sum type which may encode valid results or invalid results. And as I’ve previously covered, a source of combinatoric codepath generation is in the overuse of sum types—in short, because they introduce questions about an instance of a type’s data format, rather than answers.

A helpful lesson for me was in reframing error information returned by a codepath as error information in addition to whatever the “non-error result” is. This small change eliminates needless bifurcation of the code receiving the result—it can simply be one codepath which processes both valid results (or gracefully no-ops, if the valid results are zero-initialized), and any error information.

The Metadesk lexer API from the Digital Grove codebase is a simple example:

MD_TokenizeResult tokenize = MD_TokenizeFromText(arena, text);
for(MD_Msg *msg = tokenize.msgs.first; msg != 0; msg = msg->next)
{
  // print out `msg` info
}
for(U64 token_idx = 0; token_idx < tokenize.tokens.count; token_idx += 1)
{
  // use `tokenize.tokens.v[token_idx]`
}

The parser API follows the same pattern:

struct MD_ParseResult
{
 MD_Node *root;
 MD_MsgList msgs;
};

MD_ParseResult MD_ParseFromTextTokens(Arena *arena, String8 filename, String8 text, MD_TokenArray tokens);

This makes especially good sense because it allows the returned MD_MsgList to refer to MD_Nodes and MD_Tokens—both types, together, are more powerful than either in isolation.

In my experience, this API style allows usage code to, more-or-less, ignore the distinct idea of “errors” altogether, until it actually cares to use error information (e.g. to display error information to the user). Most usage code can simply be a single effective codepath—or in other words, it can have a completely flat control flow structure, which simply performs the required data transformation steps, in order.

Writing this, I can’t help but feel that it’s a fairly trivial lesson—but for those feeling similarly, it’s good to remind oneself just how overcomplicated programmers like to make their problems, and just how much they like to present the façade of productivity by playing around with type systems. I don’t care to digress into modern language feature design around errors—it’s trivial to find online—but in my view, the fact that there is such a widescale (and often passionate) conversation about the “need” for “error handling language features” is indicative enough of the embarrassing state of software development.

Error Information Side-Channels

errno is a globally-available, thread-local integer in the C standard runtime library, which is mutated by APIs implicitly when an error condition occurs.

Aside from a number of silly implementation details of errno, there are reasonable aspects of its design. Notably, it follows other principles outlined in this post—it doesn’t needlessly harp on the validity of results, resulting in needlessly-bifurcated usage code codepaths. It follows the principle that error information ought to be available in addition to, rather than instead of, “valid data”. Instead, it’s used to implicitly collect error information, and it allows usage code to perform whatever operations it needed, and only check when needed.

One serious problem with errno is that it’s a single thread-local slot for error information, so it’s only enough to inform usage code of the most recent error. Another is that there’s no way of knowing which code set errno last, and when.

These are fairly serious limitations. The lack of capacity and the lack of error source location information prohibits errno from being meaningfully used in more shallow frames in a call stack, which would otherwise be an extremely powerful usage pattern.

Consider, for example, if I wanted to—at the end of each simulation loop, in a game project—process all implicitly-gathered error information, and maybe render them in a debug user interface (maybe as a timeline, or flame graph). Every point in this timeline would represent a place at which error information was gathered—it could include call stack with function names, source code location, and so on. It could be integrated with a debugger such that breakpoints could be easily set when one particular error (in a particular call stack) occurs.

This is all easily doable with more powerful “side-channels” of error information, which don’t have the critical shortcomings of errno.

And contrary to yet another popular belief, programming is not delimited by decisions made while designing the C runtime library. All languages—including C—can take on entirely new forms when someone is willing to go to work and design an alternative “standard” library and environment (which is part of my goal through the Digital Grove codebase). Memory management, string processing, error handling, run-time type information, compile-time execution, metaprogramming, and a great many other things can become dramatically simpler than they’re often portrayed by those who spend more time arguing about languages on HackerNews or Reddit than they do programming.

Instead of each thread being equipped with a single integer, it can be equipped with an entire arena and message log:

per_thread Arena *log_arena;
per_thread MsgList log_msgs;

Each message stored in the MsgList can be equipped not only with a single integer, but with a callstack, a string, a color, and whatever other information is deemed useful.

This kind of design avoids the extreme lossiness of e.g. errno, but keeps code humming along smoothly. Within this context, “error handling” is simply the ability for code to gracefully continue operating in the presence of errors, with the error information accumulated in a log to be later inspected.

Logs of this kind allow not only for powerful introspection of a trace of codepath execution, but using them as a primary mechanism for error data gathering keeps each codepath honest about operating soundly and gracefully in the presence of errors. Like many of the lessons in this post, this keeps the number of codepaths small, and thus the value of programmer work high.

Closing Thoughts

The tendency for programmers to build towers of abstractions over bad fundamentals has several fundamental issues. Better designs—which do not compromise on the fundamentals—are obfuscated. Programmers detach from fundamental reality. They teach others to detach from fundamental reality, such that future improvements are much more difficult to design and conceptualize. Programmers settle for subpar compromises on the performance and simplicity of an abstraction (which, in a pragmatic context, may be completely reasonable—but poor abstractions substituted for good ones, used for pragmatic reasons, are obviously not the endpoint of abstraction design in any area). Arenas are an example of one such better design—they do not compromise on the low level details, and they simplify higher level work. They’re a clearly superior alternative to gigantic complex memory management systems, and one that could only be identified when considering the problem near the fundamentals.

“Error handling” is much the same way. By dispensing with abstract notions of “errors”, and treating the computer as a computer, and data as data, and code as code, the fundamental reality of errors can be grappled with, and grappling with that reality bottom-up can lead to much simpler problems than when theorizing about the same problem top-down. Errors, by-and-large, disappear.

I’ve found that, with this mindset, my software became much more robust, my code became much simpler, and I rely on much simpler tools. I didn’t need another language feature; I didn’t need another library; I just needed to take control of my problems.

I hope this post helped communicate to you these same lessons, which I’ve benefitted so much from.

If you enjoyed this post, please consider subscribing. Thanks for reading.

-Ryan

Subscribe now