Deploying Catalyst applications with lighttpd and FastCGI

Why lighttpd?
lighttpd limitations
FastCGI modes
Deployment of a 'stage' and 'prod' FastCGI setup
init scripts
Zero-downtime upgrade
Load balancing
What if MyApp dies?
Serving static files
Troubleshooting
See also

This is a tutorial on deploying a zero-downtime configuration of Catalyst with lighttpd, with support for load balancing. It has been tested on an Ubuntu box with Catalyst 5.7014 and lighttpd 1.4.19. It achieves the following objectives:

uses one instance of lighttpd to power two virtual hosts, one used for production and one for staging
restarting the Catalyst app can be done without restarting the whole web server
enables very easy load balancing
zero-downtime upgrade
logs all application errors to separate error logs
start/stop/restart scripts for the Catalyst app

Why lighttpd?

See http://redmine.lighttpd.net/wiki/lighttpd/PoweredByLighttpd for some very traffic-intensive sites that are powered by lighttpd (youtube, isohunt, mininova, imageshack.us etc.). See also lighttpd's benchmarks vs. Apache and other servers.

lighttpd limitations

lighttpd does not separate error logs per virtual host, but jumbles them all into one server.errorlog file (see ticket 665 and please vote for log separation). nginx can log errors separately for each virtual host.

FastCGI modes

First off, make sure to read through the lighttpd documentation on mod_fastcgi and enable mod_fastcgi. For example, on Ubuntu you can run:

lighty-enable-mod fastcgi

Otherwise, you can just add the module in a server.modules block, as we'll see in an example below. Note: check the fastcgi.conf file in your config directory (usually /etc/lighttpd/conf-enabled/10-fastcgi.conf) and remove any PHP fastcgi server that may be defined there.

There are two ways that lighttpd can communicate with your Catalyst application:

it can launch it directly (the bin-path option) (this is called "static FastCGI")
the Catalyst app is launched standalone and talks to lighttpd via UNIX-domain sockets or TCP/IP connections ("external FastCGI")

Having lighttpd launch the Catalyst application directly, forces you to incur some downtime between upgrades. Assuming you have a 'stage' and a 'production' version of your application running in parallel, and you do your work on 'stage', at some point you will want to move the 'stage' code over the 'production' code. For anything but template or layout changes, this will require restarting your application. During that startup time, your web application will effectively be "down". If you want zero-downtime upgrades scalability, you need to go the "standalone" way.

In the standalone deployment mode, the Catalyst application is started separately (using the myapp_fastcgi.pl script) and communicated with lighttpd through sockets or TCP/IP. Since sockets are only local to one machine, TCP/IP is the way to go if you want to be able to run your application on the same machine as the web server, or on different machines for load balancing.

We have therefore narrowed down our FastCGI mode choices to one: "standalone" or "external" Catalyst application communicating with lighttpd via TCP/IP.

Deployment of a 'stage' and 'prod' FastCGI setup

We're going to use lighttpd's very simple and effective conditional virtual host mechanism to setup two virtual hosts:

stage.myapp.com - for development work
myapp.com (and www.myapp.com, but see http://no-www.org/) - for production

We'll also run two myapp_fastcgi.pl servers, listening at two arbitrarily chosen ports (55900 for the production server and 55901 for the stage server).

Let's start: create /etc/lighttpd/conf-available/20_prod.conf:

server.modules += (
    "mod_fastcgi",
)
$HTTP["host"] =~ "^(www\.)?myapp.com" {
    fastcgi.server = (
        "" => (  # the extension is empty because we want to match on any extension
            "myserver1" => (
                "host" => "127.0.0.1",
                "port" => 55900,
                "check-local" => "disable"
            )
        )
    )
}

For the stage virtual host, create /etc/lighttpd/conf-available/20_stage.conf:

$HTTP["host"] =~ "^stage.myapp.com" {
    fastcgi.server = (
        "" => (  # the extension is empty because we want to match on any extension
            "myserver1" => (
                "host" => "127.0.0.1",
                "port" => 55901,
                "check-local" => "disable"
            )
        )
    )
}

Now, start the two severs. In the production myapp directory, run:

script/myapp_fastcgi.pl --listen 127.0.0.1:55900 --nproc 5 --keeperr 2>>log/error.log &

Please note you may need to install:

cpan FCGI FCGI::ProcManager

If any other modules are needed it will be shown in log/error.log

The final ampersand leaves the application running in the background and -keeperr sends error messages to STDERR. Catalyst applications don't abort on SIGHUP, so myapp_fastcgi.pl ... & will continue to run even after you log out. If you find that that is not the case, prefix the command with nohup. Note that myapp_fastcgi.pl has a -daemon option which should do the same thing, but redirecting its STDERR output to a file will not work (only the initial startup screen gets dumped, and no request debug info).

Similary, In the stage myapp directory, run:

script/myapp_fastcgi.pl --listen 127.0.0.1:55901 --keeperr 2>>log/error.log &

To have lighttpd reload the configuration files, run

/etc/init.d/lighttpd force-reload

init scripts

The setup above will work really well, but we want to create some start, stop and reload scripts. Placing them in /etc/init.d will also ensure that our application gets started automatically in case of a system reboot. Below is a simple but solid init script. For more bells and whistles and maybe more fault tolerance, see fcgi-init.

Here is an init script for the production server, /etc/init.d/myapp_prod.sh:

#!/bin/bash

# APP_NAME is the Catalyst-normalized name of your app (lowercased,
# s/::/_/g)
# i.e. the portion before "_create.pl" in scripts/*_create.pl
# myapp_user must exist and have a valid shell (i.e. /bin/bash)
APP_NAME=myapp
APP_PATH=/path/to/$APP_NAME
APP_USER=myapp_user
FCGI_TCP_CONNECTION=127.0.0.1:55900
PID_PATH=/var/run/$APP_NAME.prod.pid
LOG_FILE="$APP_PATH"/log/myapp-err.log
NPROC=5

case $1 in
start)
    if [ -r "$PID_PATH" ] && kill -0 $(cat "$PID_PATH") >/dev/null 2>&1
    then
        echo " PROD $APP_NAME already running"
        exit 0
    fi

    echo -n "Starting PROD $APP_NAME (${APP_NAME}_fastcgi.pl)..."

    touch "$PID_PATH"
    chown "$APP_USER" "$PID_PATH"

    cd "$APP_PATH"
    su -c "\"script/${APP_NAME}_fastcgi.pl\"\\
       --listen $FCGI_TCP_CONNECTION\\
       --pidfile \"$PID_PATH\"\\
       --nproc $NPROC\\
       --keeperr\\
       --daemon" "$APP_USER" 2>>"$LOG_FILE"

    # Wait for the app to start  
    TIMEOUT=10; while [ ! -r "$PID_PATH" ] && ! kill -0 $(cat "$PID_PATH")
    do
        echo -n '.'; sleep 1; TIMEOUT=$((TIMEOUT - 1))
        if [ $TIMEOUT = 0 ]; then
            echo " ERROR: TIMED OUT"; exit 0
        fi
    done
    echo " started."
    ;;

stop)
    echo -n "Stopping PROD $APP_NAME: "
    if [ -r "$PID_PATH" ] && kill -0 $(cat "$PID_PATH") >/dev/null 2>&1
    then
        PID=`cat $PID_PATH`
        echo -n "killing $PID... "; kill $PID
        echo -n "OK. Waiting for the FastCGI server to release the
        port..."
        TIMEOUT=60
        while netstat -tnl | grep -q $FCGI_TCP_CONNECTION; do
            echo -n "."; sleep 1; TIMEOUT=$((TIMEOUT - 1))
            if [ $TIMEOUT = 0 ]; then
                echo " ERROR: TIMED OUT"; exit 0
            fi
        done
        echo " OK."
    else
        echo "PROD $APP_NAME not running."
    fi
    ;;

restart|force-reload)
    $0 stop
    echo -n "A necessary sleep... "; sleep 2; echo "done."
    $0 start
    ;;

*)
    echo "Usage: $0 { stop | start | restart }"
    exit 1
    ;;
esac

The init script for the stage server differs only in the TCP port (55901), the number of processes to keep to serve requests (not specified, hence 1 by default, as opposed to 5 for the production server), and a clear indication that we are running the STAGE application:

#!/bin/bash

# APP_NAME is the Catalyst-normalized name of your app (lowercased, s/::/_/g)
# i.e. the portion before "_create.pl" in scripts/*_create.pl

APP_NAME=mds
APP_PATH=/path/to/$APP_NAME.stage
FCGI_TCP_CONNECTION=127.0.0.1:55901
PID_PATH=/tmp/$APP_NAME.stage.pid

case $1 in
     start)
        echo -n "Starting STAGE $APP_NAME (${APP_NAME}_fastcgi.pl)..."
        cd $APP_PATH
        script/${APP_NAME}_fastcgi.pl --listen $FCGI_TCP_CONNECTION --pidfile $PID_PATH --keeperr 2>>$APP_PATH/log/error.log &

        if [ -r $PID_PATH ]; then
            echo " PROD $APP_NAME already running"
            exit 0
        fi

        # Wait for the app to start
        TIMEOUT=10; while [ ! -r $PID_PATH ]; do
            echo -n '.'; sleep 1; TIMEOUT=$((TIMEOUT - 1))
            if [ $TIMEOUT = 0 ]; then
                echo " ERROR: TIMED OUT"; exit 0
            fi
        done
        echo " started."
        ;;

    stop)
        echo -n "Stopping STAGE $APP_NAME: "
        if [ -r $PID_PATH ]; then
            PID=`cat $PID_PATH`
            echo -n "killing $PID... "; kill $PID
            echo -n "OK. Waiting for the FastCGI server to release the port..."
            TIMEOUT=60
            while netstat -tnl | grep -q $FCGI_TCP_CONNECTION; do
                echo -n "."; sleep 1; TIMEOUT=$((TIMEOUT - 1))
                if [ $TIMEOUT = 0 ]; then
                    echo " ERROR: TIMED OUT"; exit 0
                fi
            done
            echo " OK."

        else
            echo "STAGE $APP_NAME not running."
        fi
        ;;

    restart|force-reload)
        $0 stop
        echo -n "A necessary sleep... "; sleep 2; echo "done."
        $0 start
        ;;

    *)
        echo "Usage: $0 { stop | start | restart }"
         exit 1
        ;;
esac

Zero-downtime upgrade

See SO_REUSEADDR. TBD.

Load balancing

You've probably noticed the "myserver1" line in the .conf files. Adding another server into the load balancing scheme is as easy as adding another item in array of servers that handle the "" extension:

$HTTP["host"] =~ "^(www\.)?myapp.com" {
    fastcgi.server = (
        "" => (  # the extension is empty because we want to match on any extension
            "myserver1" => (
                "host" => "127.0.0.1",
                "port" => 55900,
                "check-local" => "disable"
            ),
            "myserver2" => (
                "host" => "heavy-duty-server.myapp.com",
                "port" => 55900,
                "check-local" => "disable"
            )
        )
    )
}

What if MyApp dies?

Catalyst applications generally don't die, but return a nicely-formatted error page. However, if you want to make sure the myapp_fastcgi.pl script keeps getting spawned, look into FastCGI Deployment with bells on.

Serving static files

To serve static files, an easy way would be to tell lighttpd that all .pl files are to be handled by the FastCGI handler. However, that would not let you use RESTful URLs (which don't have a file extension tacked on), and you wouldn't want .pl tacked at the end of every URL anyway. There is a more sophisticated way to serve static files while letting Catalyst handle the rest - you need to handle requests in two separate categories:

static files (e.g. favicon.ico or directories (e.g. /js/) - served by lighttpd
everything else - served by Catalyst

The way to distinguish between these two cases is very simple: we will examine the URL, that is, the $HTTP["url"] variable in the lighttpd config file:

server.modules += (
    "mod_alias", 
)
$HTTP["host"] =~ "^stage.myapp.com" {
    # Serve static content via lighttpd directly
    alias.url = (
        "/favicon.ico" => "/path/to/myapp/root/favicon.ico",
        "/js/"      => "/path/to/myapp/root/js/",
        "/css/"     => "/path/to/myapp/root/css/",
        "/images/"  => "/path/to/myapp/root/static/images/",
    )
    $HTTP["url"] !~ "^/(favicon.ico$|js/|css/|images/)" {
        fastcgi.server = (
            "" => (  # anything not matching the URL above is handled by Catalyst
                "load_balancer1" => (
                    "host" => "127.0.0.1",

                    "port" => 55901,
                    "check-local" => "disable"
                )
            )
        )
    }
}

Note that we also set a different document-root for the static files using mod_alias in the alias.url block.

If you want to serve static files only after Catalyst has performed some operation (e.g. after authentication), look into Catalyst::Plugin::XSendFile and see http://blog.lighttpd.net/articles/2006/07/02/x-sendfile.

Troubleshooting

As usual, check the server's error log for any messages: /var/log/lighttpd/error.log. Speaking of the lighttpd error log - lighty's messages tend to disrupt messages from Catalyst and the log looks messy:

[info] Request took 0.005979s (167.252/s)
.----------------------------------------------------------------+-----------.
| Action                                                         | Time      |
+---------------------
2008-06-06 06:07:18: (mod_fastcgi.c.2592) FastCGI-stderr: -------------------------------------------+-----------+
| /mycontroller/my_action                                        | 0.000746s |
| /end                                                           | 0.000517s |
'----------------------------------------------------------------+-----------'

To work around this, we have launched myapp_fastcgi.pl with the --keeperr parameter, which sends error messages to STDOUT, not to the webserver.

lighttpd fastcgi