Saturday, June 8, 2013

Outbound email from Django on the App Engine dev server

Once everything is configured as described in my previous post, your code will run. But you won't be able to send outbound email from the App Engine dev server.

If you just want to see the email output but don't need it actually sent, you can simply enable the --show_mail_body flag when you launch the dev server and it'll just dump the contents out to the console.

app_devserver.py --show_mail_body mygaeproject

Wish I'd known that a few hours ago. Grumble.


But since we're here and I did eventually figure it out, here's how you can enable SMTP relaying from the local dev server out through a Gmail account:

Gmail access
Set up an "application-specific password" for your Gmail account that will only be used by your local dev server. Info is here and is pretty straightforward. Keep the 16-character password that is generated for you handy.


Avoid port 25
The default email port is blocked by a lot of ISPs, mine included. Instead use the alternate port 587.


Enable TLS
I have no clue what that is, but you won't be able to SMTP through Gmail without it. Patch the App Engine code to support it as described here. Note: each time you download a new version of the App Engine API you will have to re-patch this bit of code.


Launch the dev server with SMTP relaying
Use the SMTP option flags when you start the local server:

dev_appserver.py --smtp_host=smtp.gmail.com --smtp_port=587 --smtp_user=your_addr@gmail.com --smtp_password=16_char_app_password mygaeproject


Make a launcher script
That's a bunch to type so save yourself the trouble and go to your mygaeproject/src dir and create a simple script: nano local_server__mygaeproject.sh

Paste the above dev_appserver.py call in and save the file.

Make the script executable: chmod 700 local_server__mygaeproject.sh

Then execute it with: ./local_server__mygaeproject.sh

Django 1.4 on Google App Engine with Cloud SQL

Here's the basic skeleton for getting a Django 1.4 project up and running on App Engine (v1.8.1). There's a great tutorial that got me most of the way there, but there were some oddities or inconsistencies that took me a while to iron out.

The biggest issue is basic Django project structure and references within the app.yaml and settings.py files. I don't want to cover duplicate ground, so follow my approach here, but if there are any missing holes (such as the setup on the App Engine and Cloud SQL side) refer to the tutorial linked above.

Here's how I worked it all out:


Django project structure
Create a directory for your new project: mkdir mygaeproject

Consider this a kind of super parent directory--not to be confused with the project dir that will hold the actual Python code. Because I'm the whole dev team, UI, and marketing (at least initially), I'll end up with more than just code in here. I might have a number of subdirectories that will store resources like vector files that aren't published to the site, DB schema diagrams, etc. So I like to create a separate subdirectory for my code.

So move into your project directory cd mygaeproject and mkdir src

cd src and create your Django project using the default structure:

django-admin.py startproject mygaeproject


You'll now have:
mygaeproject
 +-src
    +-mygaeproject
       +-manage.py
       +-mygaeproject
          +-__init__.py
          +-settings.py
          +-urls.py
          +-wsgi.py

It's silly and a bit confusing to have three mygaeproject dirs, but such is life.

I'll also create a static/ dir which will receive all the static files during a python manage.py collectstatic call.

mygaeproject
 +-src
    +-mygaeproject
       +-manage.py
       +-mygaeproject
          +-__init__.py
          +-settings.py
          +-urls.py
          +-wsgi.py
       +-static


app.yaml
In mygaeproject/src/mygaeproject create an app.yaml file. I found that the DJANGO_SETTINGS_MODULE param was necessary. Other Django + App Engine tutorials removed any project name references ("mygaeproject") in app.yaml and settings.py, but things make a lot more sense with the project-relative references and work just fine if you follow the setup in this post.
application: app-engine-application-name
version: 1
runtime: python27
api_version: 1
threadsafe: true

libraries:
- name: django
  version: "1.4"

builtins:
- django_wsgi: on

env_variables:
  DJANGO_SETTINGS_MODULE: 'mygaeproject.settings'

handlers:
- url: /static
  static_dir: static
  expiration: '0'
- url: /static/admin
  static_dir: static/admin
  expiration: '0'

settings.py
Edit the auto-generated settings.py file. The beginning of the file should include:
# Django settings for mygaeproject project.

# Set up relative references with "os"

import os
BASE_DIR = os.path.abspath(os.path.dirname(__file__)) + os.sep

# Make sure the project is on the PYTHONPATH
import sys
if BASE_DIR not in sys.path:
    sys.path.append(BASE_DIR)


if (os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine') or os.getenv('SETTINGS_MODE') == 'prod'):
    is_appengine = True
else:
    is_appengine = False


Then replace the database connection settings to support Cloud SQL on production and a local MySQL instance in development. I had to add the SOUTH_DATABASE_ADAPTERS line to get App Engine to cooperate with South.

if is_appengine:
    # Running on production App Engine, so use a Google Cloud SQL database.
    SOUTH_DATABASE_ADAPTERS = {'default':'south.db.mysql'}
    DATABASES = {
        'default': {
            'ENGINE': 'google.appengine.ext.django.backends.rdbms',
            'INSTANCE': 'cloudsql-project-id:instance-name',
            'NAME': 'db-schema-name',
        }
    }
else:
    # Running in development, so use a local MySQL database
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'USER': 'local-db-username',
            'PASSWORD': 'local-db-password',
            'HOST': '',          # Set to empty string for localhost.
            'NAME': '
db-schema-name',
        }
    }



Now set the various directory references:
STATIC_ROOT = BASE_DIR + '..' + os.sep + 'static'

# URL prefix for static files.
# Example: "http://media.lawrence.com/static/"

STATIC_URL = '/static/'

# Additional locations of static files
STATICFILES_DIRS = (
     # Put strings here, like "/home/html/static" or "C:/www/django/static".
     
     # Always use forward slashes, even on Windows.
     # Don't forget to use absolute paths, not relative paths.
     BASE_DIR + '_static',
)



I put my templates in mygaeproject/src/mygaeproject/_templates. Despite the comments in the auto-generated file, these dirs do not require an absolute path. Notice that my directory references are relative to the myproject/src/myproject directory. This makes the most sense to me because I will be including a number of other Django modules alongside mygaeproject in the mygaeproject/src directory, such as django-registration-email.

TEMPLATE_DIRS = (
    # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
    # Always use forward slashes, even on Windows.
 
    'mygaeproject/_templates',
    'registration_email/templates',
)



Notice the project name references are kept as-is:
ROOT_URLCONF = 'mygaeproject.urls'

# Python dotted path to the WSGI application used by Django's runserver.WSGI_APPLICATION = 'mygaeproject.wsgi.application'


The INSTALLED_APPS param looks as you'd expect. Notice that I'm also including the South, Stripe, Django-registration, and django-registration-email modules for this particular project (you'll have your own list of additional modules here).
INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    # Uncomment the next line to enable the admin:
    'django.contrib.admin',
    # Uncomment the next line to enable admin documentation:
    # 'django.contrib.admindocs',

    'south',
    'stripe',
    'registration',
    'registration_email',
    'mygaeproject',
)



And finally some minor config to support django-registration-email and email sending in general. Getting emails out of a local App Engine dev server is a bit tough. More info in the next post.
# Support for django-registration and django-registration-email
if is_appengine:
    EMAIL_BACKEND = 'djangoappengine.mail.AsyncEmailBackend'
else:
    EMAIL_BACKEND = 'djangoappengine.mail.EmailBackend'


ACCOUNT_ACTIVATION_DAYS = 7
SERVER_EMAIL = 'your-system-email-add@blah.com'
DEFAULT_FROM_EMAIL = 'default-from-email-addr@blah.com'



wsgi.py
I confess that I don't really understand what the app.yaml django_wsgi builtin does, but all I can say is that with it set I did not have to alter the auto-generated wsgi.py in any way. Perhaps the django_wsgi builtin just ignores the wsgi.py file if it's enabled.
import os

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mygaeproject.settings")

# This application object is used by any WSGI server configured to use this
# file. This includes Django's development server, if the WSGI_APPLICATION
# setting points here.
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()



Where to put additional modules
One annoying thing is that you can't just use pip or easy_install to add additional modules (e.g. south). Whatever resources you're going to need have to be included when you push your project to App Engine. So you're going to have to download copies of the source and put them somewhere that makes sense.

Django modules are built to be pluggable and therefore independent of your particular project. So it makes sense then that they should live in directories that are parallel to your project code. Note: Most modules have a root directory that has various licensing or doc files. All you need is the source subdir.

For email support you'll also need the mail.py file from the djangoappengine project for the EmailBackend and AsyncEmailBackend.

Here's how my modules are arranged:
mygaeproject
   +-src
      +-mygaeproject
         +-app.yaml
         +-manage.py
         +-djangoappengine 
            +-mail.py
         +-mygaeproject 
            +-__init__.py
            +-settings.py
            +-urls.py
            +-wsgi.py
         +-registration 
         +-registration-email
         +-south
         +-stripe
         +-static


Great, however, you'll find that some of your modules rely on other modules. Normally a tool like pip will take care of those dependencies for you. But in the App Engine world, you'll need to grab those dependencies, too. I ended up with a few that I'd never heard of.

mygaeproject
   +-src
      +-mygaeproject
         +-app.yaml
         +-manage.py
         +-django-extensions
         +-djangoappengine
         +-mygaeproject
            +-__init__.py
            +-settings.py
            +-urls.py
            +-wsgi.py
         +-registration
         +-registration-email
         +-six
         +-south
         +-stripe
         +-static 



Misc typical Django steps
Prep your static files by going to the src/mygaeproject directory and run: python manage.py collectstatic

You'll also need to python manage.py syncdb


Launch dev server and test
Go to the src/ subdir and run: dev_appserver.py mygaeproject

You should see the dev server start up:
INFO     2013-06-08 18:21:04,246 sdk_update_checker.py:244] Checking for updates to the SDK.
INFO     2013-06-08 18:21:04,750 sdk_update_checker.py:272] The SDK is up to date.
INFO     2013-06-08 18:21:04,838 api_server.py:153] Starting API server at: http://localhost:52884
INFO     2013-06-08 18:21:04,846 dispatcher.py:164] Starting server "default" running at: http://localhost:8080
INFO     2013-06-08 18:21:04,861 admin_server.py:117] Starting admin server at: http://localhost:8000



Push to App Engine
Follow the tutorial linked at the top for info on how to directly access your Cloud SQL instance. You'll need to create your initial db schema and run syncdb against the Cloud SQL instance.

Then pushing the code is easy. From the src/ dir, run: appcfg.py update mygaeproject

When the push completes you'll be able to access your site at:

http://app-engine-application-name.appspot.com


Wednesday, July 25, 2012

Registering users with django_facebook

Hoping to save you some sanity with this one.

Thierry Schellenback's django_facebook module promises to make life easy for Django developers to instantly hook into facebook for their registration and login needs. This is kind of true. Unfortunately the documentation is way too sparse and it's missing a critical explanation: how to redirect the user after registration.

The default code redirects to Thierry's demo page after registration. It took me forever to figure out how to override this. It turned out to take only one extra parameter in the registration POST:


His examples use the "next" hidden param and it works as expected during the login flow. But registration requires the "register_next" hidden param. That's it.

Hope this saves you some time.

Friday, June 8, 2012

eHarmony's response: Really weak

Just like LinkedIn, eHarmony found themselves victim to hackers this week. But unfortunately (and again, just like LinkedIn), the hack revealed that eHarmony had failed to take even basic precautions to secure users' passwords.

Read all about the security gaffe in my blog post here.

And here comes eHarmony's not-so-mea culpa. It's even less inspiring than LinkedIn's official comment on the matter.

eHarmony says that "a small fraction of our user base has been affected" which is probably half-true since it seems that only 1.5 million of their passwords were leaked (presumably eHarmony has a much larger user base than this). Though they are probably also deluding themselves (or their users); the hackers only released the passwords that they needed help cracking. And, indeed, help did arrive.
Less than two and a half hours later, someone with the username zyx4cba posted a list that included almost 1.2 million of them, or more than 76 percent of the overall list. [...] As of Tuesday, following the contributions of several other users, just 98,013 uncracked hashes remained. (ArsTechnica)

eHarmony, liked LinkedIn, has to scramble to save face here. They can't just come out and say, "we were incompetent, sorry." But their actual comment, "Please be assured that eHarmony uses robust security measures, including password hashing and data encryption", is just a blatant falsehood. Password hashing on its own is not "robust security"; password hashing, salting, and iterating is robust. Had actual "robust security" been in place, the hacker community would not have been able to crack those 1.5 million passwords as quickly as it did.

But it gets worse: "We also protect our networks with state-of-the-art firewalls, load balancers, SSL and other sophisticated security approaches." Oh, yes, let's throw every techie-sounding term out there to impress our users! The one that really irks is "load balancers". Come on, eHarmony, you have got to be kidding with that one. Your load balancers are part of your "sophisticated security" measures? Load balancers do what it sounds like they do - they evenly distribute website traffic so no one server is overburdened. And what role do load balancers play in securing user passwords? None. None whatsoever. Lame. So, so lame, eHarmony.


Not "sophisticated," cut the BS
For all their claims of "sophisticated security" they failed to make use of the most basic password security best practices out there. And these aren't new, cutting edge techniques; the kind of encryption best practices we're talking about are almost ancient by tech standards. But somehow eHarmony and LinkedIn's developers missed that memo.

It's clear that their PR department is intent on painting them as the helpless victim here. The magically powerful hackers broke through their "robust" and "sophisticated" security and had their way with poor eHarmony. But the reality is that while, yes, anyone can get hacked, this is why you take all reasonable measures to properly encrypt your user passwords. They did not take all reasonable measures. They followed only one of the three best practices for securing passwords and now have a black eye because of it.

The hackers broke in, shame on them, but they should have found nothing more than a collection of millions of additional locks. They didn't. Shame on eHarmony. Now own up to it.

LinkedIn's response: Weak

After LinkedIn's password hacking fiasco this week, they released a blog post describing the incident and the steps they're taking to recover from it.

It's not very impressive.

The Real Lesson of the LinkedIn Password Hack, pt1

Technology is confusing but encryption and the mysterious world of hacking exist on a whole other level. It’s Matrix-like tech voodoo. 

Hackers are a 21st-century boogeyman. They possess incomprehensible powers, ninja-like access to any digital domain they choose. They can outsmart your cleverest developer. If a hacker wants your company’s data, you are powerless to stop it. Right?

Probably, yes (sorry, this post isn’t about reassurances). But that’s not the lesson of the LinkedIn debacle.

LinkedIn was hacked. It happens. But the encoded passwords that the hackers posted revealed something much more disconcerting: LinkedIn’s password encryption was awful. Borderline criminally negligent, in fact.

The Real Lesson of the LinkedIn Password Hack, pt2


In part 1 I explained the logic behind the first password best practice that LinkedIn was just barely smart enough to use. Now we explore the simple - yet shockingly effective - ways to further enhance password security.


Password Best Practice #2: Use a random salt to produce unique hashes.
A “salt” (no clue why it’s called this) is simply extra random text that is added to your password. Instead of encoding “mypassword” the site will encode “mypassword/2dsdjkl23r”. Notice the different result:

91dfd9ddb4198affc5c194cd8ce6d338fde470e2 = “mypassword”
eb8b3a04fb5bb65ecc02f40e83fa4d8065b26af6 = “mypassword/2dsdjkl23r

And every user gets her own random salt. So all of the fools who used the weak “mypassword” will end up with different encodings:

6097a22f84896b07dcd240f18b2a79ff84bec499 = “mypassword/8sadfljk23j2d08
85673bdb67447ca772199344de3fbb9ddb360368 = “mypassword/jkl34rjsdf89kl

So even if one “mypassword” account is compromised, the other “mypassword” fools are still safe, for now. Better.

But even this isn’t perfect.