Encountering Some Python Trickery

Despite the code appearing to be correct, I was dismayed to find my tests were failing, but I eventually started to laugh when I found out the root cause of the failure. The truth is, after adding this patch to ‘pebble’, Python began to think that the worker process was still alive, despite it already being deceased.

Let’s take a look at the primitive code as an example:

WHAT? How is this possible?

Pay attention here to line #166. On this line you can see that if there’s no returncode the process continues working and is alive, but if that’s not the case, Popen.poll returns returncode if it was captured after the process finished. It then returns nothing if the process is still running or hasn’t been launched since PID can’t be found (look at lines #28–31).

Popen.poll will also return empty if the process was already finished and Popen.poll didn’t capture the status. This is because os.waitpid had already been called and grabbed the PID! This is what happened in my code example above. Now let’s debug my code line by line to find a solution:

When the process finishes, Python signal calls handler:

Next it waits for PID to finish and then erases it.

Next I call:

Which calls:

And this calls:

This entire situation is why I sometimes think of Python as sly and looking mischievous. With it being unaware that os.waitpid makes multiple calls with the same PID which then leads to broken application behaviour. This is all because the next os.waitpid call doesn’t know that the requested process is already finished.

Once the process has finished, the operating system keeps the PID and status, waiting for its reaping (holding the process as zombie). This data is then erased and the zombie disappears.

After this, multiple calls of os.waitpid will return the same result for the same PID. Of course, this logic change can be dangerous for code compatibility so use it at your own risk, but I think this logic is more correct. Let me show you a small example of what I mean:

and launch:

After this patch, applied zombie processes were successfully defeated in pebble.ProcessPool 😎

Many kudos for text review to David Lorbiecke.

Related posts:

I once had a student who followed a very predictable routine during writing feedback. We would sit down and begin reading his writing together, as he nervously anticipated a response. I would point…

One of my favorite passages from the New Testament is as it appears in Matthew 22:21– Jesus when asked about paying taxes, says — “Render to Caesar the things that are Caesar’s; and to God the things…

In the business world, there is an obsession with scaling. Investors repeatedly ask “How are you going to bring your idea to scale?” Environmentalists often ask a related question by seeking to…