Saturday, June 8, 2013

Django 1.4 on Google App Engine with Cloud SQL

Here's the basic skeleton for getting a Django 1.4 project up and running on App Engine (v1.8.1). There's a great tutorial that got me most of the way there, but there were some oddities or inconsistencies that took me a while to iron out.

The biggest issue is basic Django project structure and references within the app.yaml and settings.py files. I don't want to cover duplicate ground, so follow my approach here, but if there are any missing holes (such as the setup on the App Engine and Cloud SQL side) refer to the tutorial linked above.

Here's how I worked it all out:


Django project structure
Create a directory for your new project: mkdir mygaeproject

Consider this a kind of super parent directory--not to be confused with the project dir that will hold the actual Python code. Because I'm the whole dev team, UI, and marketing (at least initially), I'll end up with more than just code in here. I might have a number of subdirectories that will store resources like vector files that aren't published to the site, DB schema diagrams, etc. So I like to create a separate subdirectory for my code.

So move into your project directory cd mygaeproject and mkdir src

cd src and create your Django project using the default structure:

django-admin.py startproject mygaeproject


You'll now have:
mygaeproject
 +-src
    +-mygaeproject
       +-manage.py
       +-mygaeproject
          +-__init__.py
          +-settings.py
          +-urls.py
          +-wsgi.py

It's silly and a bit confusing to have three mygaeproject dirs, but such is life.

I'll also create a static/ dir which will receive all the static files during a python manage.py collectstatic call.

mygaeproject
 +-src
    +-mygaeproject
       +-manage.py
       +-mygaeproject
          +-__init__.py
          +-settings.py
          +-urls.py
          +-wsgi.py
       +-static


app.yaml
In mygaeproject/src/mygaeproject create an app.yaml file. I found that the DJANGO_SETTINGS_MODULE param was necessary. Other Django + App Engine tutorials removed any project name references ("mygaeproject") in app.yaml and settings.py, but things make a lot more sense with the project-relative references and work just fine if you follow the setup in this post.
application: app-engine-application-name
version: 1
runtime: python27
api_version: 1
threadsafe: true

libraries:
- name: django
  version: "1.4"

builtins:
- django_wsgi: on

env_variables:
  DJANGO_SETTINGS_MODULE: 'mygaeproject.settings'

handlers:
- url: /static
  static_dir: static
  expiration: '0'
- url: /static/admin
  static_dir: static/admin
  expiration: '0'

settings.py
Edit the auto-generated settings.py file. The beginning of the file should include:
# Django settings for mygaeproject project.

# Set up relative references with "os"

import os
BASE_DIR = os.path.abspath(os.path.dirname(__file__)) + os.sep

# Make sure the project is on the PYTHONPATH
import sys
if BASE_DIR not in sys.path:
    sys.path.append(BASE_DIR)


if (os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine') or os.getenv('SETTINGS_MODE') == 'prod'):
    is_appengine = True
else:
    is_appengine = False


Then replace the database connection settings to support Cloud SQL on production and a local MySQL instance in development. I had to add the SOUTH_DATABASE_ADAPTERS line to get App Engine to cooperate with South.

if is_appengine:
    # Running on production App Engine, so use a Google Cloud SQL database.
    SOUTH_DATABASE_ADAPTERS = {'default':'south.db.mysql'}
    DATABASES = {
        'default': {
            'ENGINE': 'google.appengine.ext.django.backends.rdbms',
            'INSTANCE': 'cloudsql-project-id:instance-name',
            'NAME': 'db-schema-name',
        }
    }
else:
    # Running in development, so use a local MySQL database
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'USER': 'local-db-username',
            'PASSWORD': 'local-db-password',
            'HOST': '',          # Set to empty string for localhost.
            'NAME': '
db-schema-name',
        }
    }



Now set the various directory references:
STATIC_ROOT = BASE_DIR + '..' + os.sep + 'static'

# URL prefix for static files.
# Example: "http://media.lawrence.com/static/"

STATIC_URL = '/static/'

# Additional locations of static files
STATICFILES_DIRS = (
     # Put strings here, like "/home/html/static" or "C:/www/django/static".
     
     # Always use forward slashes, even on Windows.
     # Don't forget to use absolute paths, not relative paths.
     BASE_DIR + '_static',
)



I put my templates in mygaeproject/src/mygaeproject/_templates. Despite the comments in the auto-generated file, these dirs do not require an absolute path. Notice that my directory references are relative to the myproject/src/myproject directory. This makes the most sense to me because I will be including a number of other Django modules alongside mygaeproject in the mygaeproject/src directory, such as django-registration-email.

TEMPLATE_DIRS = (
    # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
    # Always use forward slashes, even on Windows.
 
    'mygaeproject/_templates',
    'registration_email/templates',
)



Notice the project name references are kept as-is:
ROOT_URLCONF = 'mygaeproject.urls'

# Python dotted path to the WSGI application used by Django's runserver.WSGI_APPLICATION = 'mygaeproject.wsgi.application'


The INSTALLED_APPS param looks as you'd expect. Notice that I'm also including the South, Stripe, Django-registration, and django-registration-email modules for this particular project (you'll have your own list of additional modules here).
INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    # Uncomment the next line to enable the admin:
    'django.contrib.admin',
    # Uncomment the next line to enable admin documentation:
    # 'django.contrib.admindocs',

    'south',
    'stripe',
    'registration',
    'registration_email',
    'mygaeproject',
)



And finally some minor config to support django-registration-email and email sending in general. Getting emails out of a local App Engine dev server is a bit tough. More info in the next post.
# Support for django-registration and django-registration-email
if is_appengine:
    EMAIL_BACKEND = 'djangoappengine.mail.AsyncEmailBackend'
else:
    EMAIL_BACKEND = 'djangoappengine.mail.EmailBackend'


ACCOUNT_ACTIVATION_DAYS = 7
SERVER_EMAIL = 'your-system-email-add@blah.com'
DEFAULT_FROM_EMAIL = 'default-from-email-addr@blah.com'



wsgi.py
I confess that I don't really understand what the app.yaml django_wsgi builtin does, but all I can say is that with it set I did not have to alter the auto-generated wsgi.py in any way. Perhaps the django_wsgi builtin just ignores the wsgi.py file if it's enabled.
import os

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mygaeproject.settings")

# This application object is used by any WSGI server configured to use this
# file. This includes Django's development server, if the WSGI_APPLICATION
# setting points here.
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()



Where to put additional modules
One annoying thing is that you can't just use pip or easy_install to add additional modules (e.g. south). Whatever resources you're going to need have to be included when you push your project to App Engine. So you're going to have to download copies of the source and put them somewhere that makes sense.

Django modules are built to be pluggable and therefore independent of your particular project. So it makes sense then that they should live in directories that are parallel to your project code. Note: Most modules have a root directory that has various licensing or doc files. All you need is the source subdir.

For email support you'll also need the mail.py file from the djangoappengine project for the EmailBackend and AsyncEmailBackend.

Here's how my modules are arranged:
mygaeproject
   +-src
      +-mygaeproject
         +-app.yaml
         +-manage.py
         +-djangoappengine 
            +-mail.py
         +-mygaeproject 
            +-__init__.py
            +-settings.py
            +-urls.py
            +-wsgi.py
         +-registration 
         +-registration-email
         +-south
         +-stripe
         +-static


Great, however, you'll find that some of your modules rely on other modules. Normally a tool like pip will take care of those dependencies for you. But in the App Engine world, you'll need to grab those dependencies, too. I ended up with a few that I'd never heard of.

mygaeproject
   +-src
      +-mygaeproject
         +-app.yaml
         +-manage.py
         +-django-extensions
         +-djangoappengine
         +-mygaeproject
            +-__init__.py
            +-settings.py
            +-urls.py
            +-wsgi.py
         +-registration
         +-registration-email
         +-six
         +-south
         +-stripe
         +-static 



Misc typical Django steps
Prep your static files by going to the src/mygaeproject directory and run: python manage.py collectstatic

You'll also need to python manage.py syncdb


Launch dev server and test
Go to the src/ subdir and run: dev_appserver.py mygaeproject

You should see the dev server start up:
INFO     2013-06-08 18:21:04,246 sdk_update_checker.py:244] Checking for updates to the SDK.
INFO     2013-06-08 18:21:04,750 sdk_update_checker.py:272] The SDK is up to date.
INFO     2013-06-08 18:21:04,838 api_server.py:153] Starting API server at: http://localhost:52884
INFO     2013-06-08 18:21:04,846 dispatcher.py:164] Starting server "default" running at: http://localhost:8080
INFO     2013-06-08 18:21:04,861 admin_server.py:117] Starting admin server at: http://localhost:8000



Push to App Engine
Follow the tutorial linked at the top for info on how to directly access your Cloud SQL instance. You'll need to create your initial db schema and run syncdb against the Cloud SQL instance.

Then pushing the code is easy. From the src/ dir, run: appcfg.py update mygaeproject

When the push completes you'll be able to access your site at:

http://app-engine-application-name.appspot.com