You’ve Got Us Under Your Skin

This little ditty, to the tune of I’ve Got You Under My Skin, was created for the 2013 Courthouse Follies. It is disrespectfully dedicated to the NSA.

They’ve got us under their skin
They’ve got us deep in the heart of them
So deep in their heart, we’re really a part of them
They’ve got us under their skin

We’re always listening in
Their emails and phone calls and texts always go down so well
To our data centers where they’re lovingly stored so well
[addressing audience from here on]
You’ve got us under your skin

We’ll sacrifice anything, come what might
For the sake of stoking up fear
We violate every constitutional right
That we still claim to hold dear

Don’t you know little fools, you can never win?
Forget legality
Wake up to reality
If you really do value privacy
You should stop before you log in
‘Cause you’ve got us under your skin

Mysterious failure of Debian maxlifetime session cleanup, solved

Abstract a/k/a spoiler alert:  
Invalid directives in a php.ini file cause the silent failure of the cron job/shell script that Debian systems use to garbage-collect expired PHP session data files.

One of the things I do at my job is maintain a couple of LAMP servers for our office’s use. We recently upgraded them from a long-outdated Ubuntu server edition to Debian 7 and PHP 5.4.4. Among the duties of these boxes is running some archaic and crudely written PHP code, so I expected some compatibility nightmares. I was able to resolve most of the errors, warnings, and weird behaviors one way or another, but the session garbage collection would not work, and I spent too long trying to figure out why. I don’t generally blog about technical stuff, but this might save someone some pain in the future — possibly even myself, when I forget the lesson learned here.

If you’re familiar with Debian and progeny (like Ubuntu), then you know Debian likes to store PHP session data files in /var/lib/php5 and put permissions on it such that if you let PHP try to do garbage collection, it fails with an error like this:

session_start() [function.session-start]: ps_files_cleanup_dir: opendir(/var/lib/php5) failed: Permission denied (13)

And if your php.ini settings for session.gc_probability and session.gc_divisor are the defaults, this error will only occur 1% of the time, making it fun to debug. But you did your Google diligence and RTFM, and learned that out of the box, Debian sets up a cron job that once per 30 minutes removes session data files whose modification times are older than session.gc_maxlifetime/60 minutes old. The Debian way seems to be to disable PHP garbage collection by setting session.gc_probability to 0 and letting cron do its thing, as
we can see in /etc/cron.d/php5:

09,39 * * * * root [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -ignore_readdir_race -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete

Fine. The Debian dudes know what they are doing, so I went with it. Before long, however, I started wondering why none of my session files were getting deleted ever. I am not a black belt in system administration, but I really scoured the logs and tried to figure it out, and Google was no help. So I took a closer look at that cron entry. What’s it doing? In essence, it is using the output of /usr/lib/php5/maxlifetime to determine how many minutes old a session data file has to be to be eligible for deletion. OK, then let’s have a look at /usr/lib/php5/maxlifetime. Not being fluent in shell scripting, I had to squint and scratch my head a bit to follow this.

#!/bin/sh -e
if which php5 >/dev/null 2>&1; then
for sapi in apache2 apache2filter cgi fpm; do
if [ -e /etc/php5/${sapi}/php.ini ]; then
cur=$(php5 -c /etc/php5/${sapi}/php.ini -d “error_reporting=’~E_ALL'” \
-r ‘print ini_get(“session.gc_maxlifetime”);’)
[ -z “$cur” ] && cur=0
[ “$cur” -gt “$max” ] && max=$cur
for ini in /etc/php5/*/php.ini /etc/php5/conf.d/*.ini; do
cur=$(sed -n -e ‘s/^[[:space:]]*session.gc_maxlifetime[[:space:]]*=[[:space:]]*\([0-9]\+\).*$/\1/p’ $ini 2>/dev/null || true);
[ -z “$cur” ] && cur=0
[ “$cur” -gt “$max” ] && max=$cur
echo $(($max/60))
exit 0

If you are new to shell scripting and would like to understand this, pay attention. Otherwise, skip ahead. We start by setting max=1440 — that means it will be our default if we don’t overwrite it in the code that follows. That first if condition tests whether there is a php5 binary installed someplace on this system. If so, then we iterate through the standard places where php.ini files are found, and use that php5 to load that configuration file, print out its session.gc_maxlifetime setting, and assign it to the variable $cur. Then we check to make sure we actually got a non-empty $cur before comparing it to $max. By the way, it appears that

[ -z "$cur" ] && cur=0

is a popular shell scripting idiom that is faster to type than the more verbose if construct. The stuff to the right of && only executes if the test on the left of it is true. In this case, the -z tests whether $cur is a zero-length string. This is kind of counter-intuitive to the uninitiated, because one might think the && really means and rather than if. It does mean and, but only if the expression to its left evaluates to true.

If $cur is greater than $max, we assign it to $max, and the result of this iterative processing is that the highest $cur we ever see will become our $max, as long as it’s greater than the initial 1440.

OK, and what’s with the next big else block? If we don’t find a php5 executable, we use sed — the Stream EDitor — to try to parse out the session.gc_maxlifetime value from the .ini file, and otherwise do the same thing as the preceding if.

Well, you may wonder, if they think sed is guaranteed to work, why not just do it that way in the first place? Frankly, I’m not sure. It turns out, as we’ll see, the use of php5 eventually alerted me to a misconfiguration that would not have been picked up had we used sed

So, what is the output of this damn thing, which the cron job is relying on? I ran it from the command line and found that it printed exactly nothing. So I made a copy, which I loaded into a text editor, and started applying my primitive debugging techniques, i.e., using echo statements just to figure out what was getting run and to see the values of things like $max and $cur. Pretty soon I determined that the script was choking on

cur=$(php5 -c /etc/php5/${sapi}/php.ini -d "error_reporting='~E_ALL'"
-r 'print ini_get("session.gc_maxlifetime");')

Then I started looking at the exit status. When I ran /usr/lib/php5/maxlifetime; echo $? from the command line, behold, our exit status is 127. Another consultation with Mr. Google reveals that 127 means “command not found.” What the hell is the command that isn’t being found? I was baffled.

Going back to the very top of this shell script, I wondered what the -e switch means at the shebang line. Back to Google to learn that it means “if not interactive, exit immediately if any untested command fails.” OK, then let’s remove the -e and see what happens:

10: [: Illegal number:
Fatal error: Directive 'allow_call_time_pass_reference' is no longer available in PHP in Unknown on line 0

Maybe I am a little slow, but I had to think on this for a while. Who said anything about allow_call_time_pass_reference? Look again at the first thing that php5 command is doing: loading the configuration file indicated by the -c switch. It is puking because there is an invalid configuration directive in php.ini! The above output is telling us that php5 is complaining about that, and the shell script in turn is complaining because it expects a number in the comparison at line 10.

Why is allow_call_time_pass_reference even there in the first place? Because it’s my habit to recycle php.ini across PHP upgrades so I don’t have to hand-edit all my settings again. I fixed that setting and tried again, then got another complaint about register_long_arrays, another configuration setting that is no longer valid in PHP 5.4. Both of these really are things I can do without, so good riddance. The lesson is that if you are going to keep an old php.ini, you should test it after an upgrade with a php5 command that does what this does: load the config, set error reporting to the max, and see what happens.

But getting back to the cron job:

09,39 * * * * root [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -ignore_readdir_race -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete

Now it’s clear that -cmin was getting an empty string, which will make find choke. But cron doesn’t email me the error output because 2>/dev/null sends it into the void.

The thing that made this adventure so much fun is that the cron job was failing because the shell script was failing because the php5 command therein was failing because of an obsolete php.ini setting that had no relevance to sessions — several levels of indirection — and error suppression was making it all the harder to track down.

A final weird note: I never did figure out why I was getting exit status 127. I tried to recreate the scenario while writing this entry, and couldn’t get it to happen again; instead, I got exit status 1. Something was different. That’s one mystery I can live without solving.

2010, another good year

It’s interesting — and sometimes, gratifying — to look back at the previous year after the completion of another cycle through the Gregorian calendar. And I enjoy the narcissistic satisfaction of celebrating my achievements. So here’s a look at noteworthy events in 2010, from my self-centered point of view:

  • I got married to my wife on May…. something (no worries, I have until May 2011 to remind myself of the exact date). That was a courageous act for both of us. Yet I can state in all honesty that I have no regrets — and I do believe she can say the same. If you think that should go without saying, think again. The hard truth is that all intimate relationships have their difficulties, and why shouldn’t they? People are imperfect. How could their relationships with one another be otherwise? In a recent conversation with my beloved wife, we concluded that a good marriage is in a sense comparable to raising children: it makes your life harder, but better. (Whereas a shitty marriage simply makes your life harder.)

    Don’t feel bad if you weren’t invited to the ceremony — almost nobody was. We had the obligatory two witnesses, an officiant, one witness’ spouse as photographer, and our kids. We stood barefooted in our back yard and the whole thing was over within three minutes.

  • I let go of my formal Zen practice for the time being, withdrawing from the zendo at which I had been a student for three plus years. The reasons are somewhat complex. Suffice it to say was time to move on. I might have liked to join some other zendo, be part of a sangha, work with a teacher. But I had to accept the fact that I while I may have a pretty good shot at doing most of what I want to do sequentially, I can’t have everything I want all at the same time because there simply isn’t enough time. Raising kids, having a substantial commute, being married, and maintaining a committed running practice — that’s about all I can handle. I sit as much as I can when I can, generally at least 15 minutes every day save a very few days per year. I sit longer when I can.
  • Speaking of running, I ran no marathon in 2010, but nailed personal records in shorter distances, such as
    • a 10K race in March that I jumped into spontaneously — i.e., without training for it explicitly — and surprised myself with a 6:55/mile pace, 5th in my 50-54 age group. I didn’t think I could run it at a sub- 7:00/mile pace; my previous best had been 7:18. This meant it was time to raise the bar.
    • the Brooklyn Half Marathon in May, where I was aiming for a 7:25 pace and ran at a 7:22 average pace. That was good for 17th of 169 in my age group, which might not sound stellar except that this popular race draws a pretty strong field. Not the least satisfying aspect was that it felt like I was pressing to maintain a quality pace the whole way and was actually concentrating on what I was doing for almost the entire 1:36:35. I probably could have done better if I had trained with a coach and run more strategically — but rare is the race in which you cannot say you might have done better if this or that.
    • a 5K race in August on Martha’s Vineyard with over 1500 runners, at which your servant finished 5th in his age group at a 6:39/mile pace. Not two weeks before I had run another 5K in hot and humid conditions and had gone out too fast, finishing at a 6:52 pace — which happened to be good enough for 2nd in my age group, but I went out too fast and struggled later. This one was pleasing because I kept up a reasonably even pace the whole way and did better. My wife also ran it just for the pleasure of running.
    • a five mile New York Road Runners race in November, an event in Central Park with over 2000 runners. My 6:46 pace was good for 4th place in my age group. It’s exciting to come this close to actually being given some sort of award.

    This bragging may be unseemly. All the ass-kicking runners I know around town are genuinely modest. But I am still getting over my astonishment at the sudden and unexpected gift of being able to run this well, because I never knew what it was to attach a timing chip to a shoe, pin a number to my shirt, and run seriously until three years ago. I am a kid who is thrilled with his cool new toy.

  • We acquired two more cats.
  • Our daughter Mylie got two guinea pigs whom she named Calvin and Hobbes.
  • Speaking of kids, all four of ours got a year older, got bigger, moved up a grade in school, kept on inhaling and exhaling, kept us on our toes — and for this we are boundlessly grateful.

multitasking, monotasking, nulltasking

People often speak of multitasking to refer to things like, talking on the phone and taking a dump and reading a magazine all at the same time. I am not so sure anyone has introduced a term to refer to a sort of failed multitasking that happens all the time. You’re talking on the phone to somebody, having a reasonably linear conversation, when suddenly the other party says “damn, what does this dick want from me now?” And you’re like, huh? “Oh nothing I’m just looking at my email.” That isn’t multitasking, it’s monotasking. The person whom you were talking to was talking to you, then he wasn’t, then he was.
It gets better. I’m in the kitchen preparing dinner and turn away from the counter, walk two and a half steps to the refrigerator, open it, and — the screen goes blank. No, my friends, I don’t merely forget why I opened the refrigerator. I forget it all, including my name and any notion of time or place or purpose in this universe. Everything comes to a complete halt, a perfect stillness. Yes, I suppose certain involuntary bio-activities like breathing and circulation are still carrying on. But for all practical intents and purposes, this is nulltasking.

IED is an MMA

MMA: Mendacious Military Acronym

We are given to understand that IED stands for Improvised Explosive Device, the things that Iraqi insurgents use to blow away US troops. There’s just one problem: there is nothing improvisatory about them. I heard a radio interview with a high level U.S. military guy who sounded like he knew what he was talking about. He described how the people who create these “improvised” devices have adapted them to changing defenses. For example, the devices used to be hard-wired. So the U.S. started training guys to spot the wires. So the IED developers switched to wireless. The U.S. started armouring the vehicles more heavily (a stroke of genius there, don’t you think?), so the IED dudes started packing bigger payloads. To my mind, this is anything but improvised — it is carefully planned, deliberate, systematic, methodical.

Here’s what the improvised in IED really stands for. It means whoever built it had the audacity to build it himself instead of being a good citizen and buying it from a respectable, world-class arms dealer such as France, China, Russia, Israel, or the USA.

Open Letter to Yankees Fans

That’s about all a proper pro-Red Sox Yankees-hating Mets fan can say at a time like this.
But I’ll continue anyway. Whose your daddy now? Huh? Woohoooo! You wanna talk superstition? Who’s superstitious now? Did you see all those Yankee fans holding up pictures of the Babe, trying to invoke The Curse? Pathetic! Ah hahahahahahaha haaaaaaaa!
Sure, you can route for the Sox to lose the World Series, but even if they do, that’s pretty cold comfort, isn’t it? Tee hee heee. Because the real series is over and you blew it, you fucking blew a 3-0 series lead and made major league baseball history. Imagine that. Go Red Sox, I love you guys!
Gee, I wonder how poor George is feeling today. Are you gonna fire Joe Torre now, George? Will that help?
Bwaaa ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha….