And for a moment he held out his hands as if to steady himself or as if to bless the ground there or perhaps as if to slow the world that was rushing away and seemed to care nothing for the old or the young or rich or poor or dark or pale or he or she.

— All the Pretty Horses

TIL 2026-03-26 - Magical Realism and Parallel Bash Commands

2026-03-26

Part of a series of “Today I Learned”s

Contents

Magical Realism

I’ve been reading Gabriel Garcia Marquez’s Love in the Time of Cholera. TIL this falls into a broader literary genre called Magical Realism. Real narratives - especially those coming from an Oral Tradition - tend to play around with the laws of the universe, compressing time and exaggerating coincidences while still maintaining the core truths and plausibility of the tale. It’s an uneasy read for exactly this reason, with some sense of the supernatural or ’ surely this can’t be happening,’ though the characters seem to take it face-on. In that sense, it feels like Kafka’s The Trial, where only the reader feels unsettled while everyone else thinks this is perfectly normal.

I don’t think it’s as straightforward as the ‘Unreliable Narrator’ trope. It feels more like a communal mythology, twisted over many retellings. Choosing what the core truth is makes the reader self-conscious of their own world (metafiction).

Running multiple programs in parallel in bash

program1
program2

Runs these programs sequentially. Using an & at the end of the command tells it to run in the background. 

program1 &
program2 &

We can collect the process IDs of all programs running in the background in some bash array pids = () and wait for them to finish. 

program1 &
pids+=($!)

program2 &
pids+=($!)

Here $! is a special bash variable which means “the process ID of the most recently started background process”

These programs are now all running in the background. We need to wait so that the script doesn’t exit or move on to the next step while programs are still running. To wait for them to finish, use the wait <ID> command: 

for pid in "${pids[@]}"; do
  wait "$pid"
done

wait "$pid" is actually overkill in this example. We can just use a basic wait without bothering to collect process IDs. wait on its own waits for all currently running background jobs started by this shell. 

program1 &
program2 &
wait
echo "All done"

Worker pools

We don’t want to run 1000 tasks at once. Instead, we should choose to work with a worker pool pattern - here, a fixed number of tasks is run in parallel. E.G., instead of 1000 tasks run at once by 1000 workers, we have a set of 5 running jobs handled by 5 workers. Whenever a worker finishes a task, they pick up the next.

wait -n is a useful flag for this - it tells the script to wait for any background process to finish.

Here’s an example implementation of a worker pool. First, set the maximum number of jobs

max_jobs=4

Then loop over all of our tasks. 

for task in {1..10}; do  
    echo "Starting task $task"  
    sleep 2 &  # pause execution for 2 seconds
    ...

We run each of the tasks, sending them to the background. The loop continues, adding more tasks to the background until we hit this condition:

for task in {1..10}; do  
    echo "Starting task $task"  
    sleep 2 &  # pause execution for 2 seconds
    if (( $(jobs -r | wc -l) >= max_jobs )); then  
        wait -n  
    fi
done

wait

Decoding this: jobs -r lists all running background jobs that are started by the current shell. wc -l gives the line count of the standard input. So,$(jobs -r | wc -l) gives a count of the running background jobs. If we have more jobs than workers, we wait for a worker to get freed up. Finally, we need to remember to wait at the end of the loop, for the last set of max_jobs background processes running.

Back to posts