Problems with the OS X 10.5.5 update and X11

After upgrading to OS X 10.5.5 I ran into an issue with X11 refusing to start. I first noticed this issue as a problem with ssh hanging while trying to establish a connection. When I dug into this problem I noticed that ssh was hanging immediately after some X11 related operations were performed. I have ssh configured to set up X11 forwarding unless otherwise specified on the ssh command line. By explicitly disabling X11 forwarding (using ssh -x) the connection would connect normally. Unfortunately this means that the very cool auto-start of X11 that Apple implemented with OS X 10.5 is lost.

With X11 being the suspected cause of the ssh problem I tried launching X11.app manually by double clicking the X11 application in /Applications/Utilities/X11.app. I noticed that X11 did not bounce in the Dock. In fact X11 did not appear in the Dock at all. That’s odd…

I noticed the following lines repeating in the system logs:
Oct 17 10:25:34 macBook com.apple.launchd[113] (org.x.startx): Throttling respawn: Will start in 10 seconds
Oct 17 10:26:04 macBook com.apple.launchd[113] (org.x.startx): Throttling respawn: Will start in 10 seconds

At least these log entries confirm that X11 is unhappy.

The machine on which I first noticed this problem may have had the XQuartz updates to X11 installed at some point. To eliminate this third-party code as a contributor to the problem I attempted to reproduce the problem a few configurations:

Since the problem can be reproduced on machines that have never had XQuartz installed I can rule out XQuartz as contributing to the problem.

Based on a tip found on the Apple support forums I investigated whether any of my shell start-up scripts had any impact on the problem. It appears that Apple has made a change to the way that X11 starts in 10.5.5. These changes make X11 start-up brittle in the face of the common shell start-up idiom of using the exec command to replace the current shell process with another process.

I have used my old .profile start-up script for bash has been in use on various machines since OS X 10.2 without problems. This old script was:

if [ -e /sw/bin/tcsh ]; then
    # We prefer to use the fink installed version of tcsh..
    exec /sw/bin/tcsh
elif [ -e /bin/tcsh ]; then
    # ... but we will use the OS X installed version if it exists
    echo "warning: using Apple version of tcsh from /bin/tcsh"
    exec /bin/tcsh
else
    echo "error: could not find a version of tcsh to exec!"
    echo "       dropping to default bash setup (no fink environment)"
fi

Nothing exciting there other than the exec. exec is commonly used in this manner to dynamically change the user’s shell for typically one of two reasons:

• to change to a user shell that uses a different syntax (such as csh or ksh) without risking breaking system shell scripts and installer scripts, or
• to lock the user into a program that limits what the user can do on the system. (For example, locking the user into a menu-driven terminal program rather than providing full shell access.)

Back when I worked on Solaris and Ultrix (showing my age) machines, the guidance was that shells used to start system services should be executed with options that prevent execution of user-supplied shell configuration files. This was advised to ensure that commands and environment variables applied by the user-supplied configuration files would not interfere with the system service. Said in another way, you want to ensure that your services scripts are executing in a known environment rather than an environment that the user has futzed with. bash provides the options –noprofile and –norc to provide this level of control over the script execution environment.

Working on the theory that the exec commands are tickling this X11 start-up problem I have worked around the problem by switching to the following .profile start-up script:

if [ -t 0 ]; then
  export INTERACTIVE=true
else
  export INTERACTIVE=false
fi
  
if [ "${INTERACTIVE}" == "true" ]; then
  if [ -e /sw/bin/tcsh ]; then
    # We prefer to use the fink installed version of tcsh..
    exec /sw/bin/tcsh
  elif [ -e /bin/tcsh ]; then
    # ... but we will use the OS X installed version if it exists
    echo "warning: using Apple version of tcsh from /bin/tcsh"
    exec /bin/tcsh
  else
    echo "error: could not find a version of tcsh to exec!"
    echo "       dropping to default bash setup (no fink environment)"
  fi
fi

This work-around avoids tickling the problem with the Apple start-up scripts for X11 by determining if the shell invoking my .profile is an interactive shell (i.e. something that a user will type commands into). If the shell is not interactive then set an environment variable and exit. Otherwise, the .profile file executes the commands in my old start-up script.

Note the use of test -t 0 to determine if we are an interactive shell. Until recently I used tty -s to determine if the script was in an interactive shell. Apparently POSIX has deprecated the -s option so test -t 0 may be a safer choice.