Tag Archives: python

Django Serving Static Files without a Web Server

Posted on by
2

When deploying Django site it is always recommended to serve static files (e.g. js, css, img, static, media) using a normal web server instead of relying on Django built development server or Gunicorn.

But sometimes you want to test something quick and you don’t want to bother with installing or configuring a web server, the solution is to use the built-in django.views.static.serve view to serve the static files from Django it self:


$ vi project/urls.py
:
# Serve Static Files 
from django.conf import settings
if settings.DEBUG:
    urlpatterns += url(r'^(?P<path>(js|css|img|static|media)/.*)$',
 'django.views.static.serve', {
   'document_root':  '/path/to/static/files/'
  }),

Notice it is configured to work on DEBUG mode only since it isn’t efficient nor tested for production use.

Managing Python Environments with “virtualenvwrapper”

Posted on by
1

What is virtualenvwrapper

virtualenvwrapper is a tool that (as the name suggest) wrap “virtualenv” program, virtualenv solve the following problem:

The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python2.7/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.

“virtualenvwrapper” will is simple wrapper that allows to create virtual environments and switch between them easily!

Installation

First we will make sure “pip” is installed (system wide), then install “virtualenvwrapper” using “pip”:

$ sudo aptitude install python-pip
$ sudo pip install virtualenvwrapper

If you don’t have root access try: pip install –install-option=”–user” virtualenvwrapper

Then you need to add “virtualenvwrapper” to your startup script, “.bashrc” if you are using bash (most OS use it by default):

$ vi ~/.bashrc
# add the following at the end
export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh

Usage:

We will make new virtualenv named “myenv1″ and install Django 1.4 on it:

rayed@ubuntu:$ mkvirtualenv myenv1
(myenv1)rayed@ubuntu:$ 
(myenv1)rayed@ubuntu:$ pip install django==1.4
(myenv1)rayed@ubuntu:$ pip freeze
Django==1.4

“pip freeze” list all installed Python packages

Now we will make a new environment “myenv2″ and install latest Django on it:

(myenv1)rayed@ubuntu:$ mkvirtualenv myenv2
(myenv2)rayed@ubuntu:$ 
(myenv2)rayed@ubuntu:$ pip install django
(myenv2)rayed@ubuntu:$ pip freeze
Django==1.5

Now we have 2 isolated environment each with its own version of Python packages.

Now you can use “workon” command to switch between them:

(myenv2)rayed@ubuntu:$ workon myenv1
(myenv1)rayed@ubuntu:$ pip freeze
Django==1.4
(myenv1)rayed@ubuntu:$ workon myenv2
(myenv2)rayed@ubuntu:$ pip freeze
Django==1.5

To list all created virtualenvironment use the command:

rayed@ubuntu:$ workon
myenv1
myenv2

A neat little trick is to associate a dir with a virtualenv, using the commands “setvirtualenvproject”:

(myenv1)rayed@ubuntu:~$ cd my_django_site
(myenv1)rayed@ubuntu:~/my_django_site$ setvirtualenvproject

So next time you switch to “myenv1″ it will automatically switch to “my_django_site” directory.

When you don’t need a virtualenv any more just delete it:

(myenv1)rayed@ubuntu:~$ deactivate
rayed@ubuntu:~$ rmvirtualenv myenv1

Open Questions?

By default “virtualenvwrapper” doesn’t include system installed site packages and start with fresh set of packages (almost), which is fine for most package, but some packages have many large complex dependencies which make installing them using pip very lengthy and hard process, example for these packages MySQLdb, PIL, psycopg2, this why I prefer to install them using system package manager (aptitude in Ubuntu).

You can pass “–system-site-packages” to mkvirtualenv command:

mkvirtualenv --system-site-packages  mytest1

Or make the default behaviour: 

export VIRTUALENVWRAPPER_VIRTUALENV_ARGS='--system-site-packages'

But in this case we will import ALL Python system packages!

A dirty work around is to link package manually:

(myenv1)rayed@s1:~$ cdsitepackages
(myenv1)rayed@s1:~/.virtualenvs/myenv1/lib/python2.7/site-packages$ ln -s /usr/lib/python2.7/dist-packages/_mysql* .
(myenv1)rayed@s1:~/.virtualenvs/myenv1/lib/python2.7/site-packages$ ln -s /usr/lib/python2.7/dist-packages/MySQLdb 
(myenv1)rayed@s1:~/.virtualenvs/myenv1/lib/python2.7/site-packages$ ln -s /usr/lib/python2.7/dist-packages/PIL 
(myenv1)rayed@s1:~/.virtualenvs/myenv1/lib/python2.7/site-packages$ cdproject
(myenv1)rayed@s1:~$

To automate this process with new virtualenv add the commands to “postmkvirtualenv” hook:

rayed@s1:~$ cat >> ~/.python_envs/postmkvirtualenv

# Link MySQLdb and PIL from system packages
OLDPWD=`pwd`
cdsitepackages
ln -s /usr/lib/python2.7/dist-packages/_mysql* .
ln -s /usr/lib/python2.7/dist-packages/MySQLdb  .
ln -s /usr/lib/python2.7/dist-packages/PIL  .
cd $OLDPWD

Thank You

I learned about virtualenvwrapper from Two Scoops of Django book which is great e-book about Django development, and also from Yasser Alraddadi, Thank you guys

Minimal Django Project: Part 2 … Flat Pages

Posted on by
3

In this step we will setup the Flatpage App, Flatpage allows you to create simple static pages from the Admin interface:

mycms$ mkdir templates
mycms$ vi mycms/settings.py
:
MIDDLEWARE_CLASSES = (
:
    # Add Flatpage Middleware
    'django.contrib.flatpages.middleware.FlatpageFallbackMiddleware',
)
:
TEMPLATE_DIRS = (
    :
    '/Users/rayed/python/mycms/templates',
)
:
INSTALLED_APPS = (
:
    # Add Flat Pages App
    'django.contrib.flatpages',
:
)
:


mycms$ vi mycms/urls.py
:
# Your other patterns here
# SHOULD BE THE LAST RULE
urlpatterns += patterns('django.contrib.flatpages.views',
    (r'^(?P<url>.*)$', 'flatpage'),
)

mycms$ ./manage.py syncdb
mycms$ mkdir -p templates/flatpages
mycms$ vi templates/flatpages/default.html
<!DOCTYPE html>
<html>
<head>
<title>{{ flatpage.title }}</title>
</head>
<body>
<h1>{{ flatpage.title }}</h1>
{{ flatpage.content }}
</body>
</html>

No from the admin interface add a new page from “Flat pages” app, and you can view directly from the normal site.

Minimal Django Project: Part 1 … First Run

Posted on by
Reply

A Django site.

The following is the shortest way to create DB backed Django project, it uses SQLite as a backend, which make the setup very easy and fast.

I usually use for testing new Django modules or apps.

$ django-admin.py startproject mycms
$ cd mycms
mycms$ chmod +x manage.py 
mycms$ vi mycms/settings.py 
:
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': './mycms.db',
:
    }
}
:
TIME_ZONE = 'Asia/Riyadh'
:
INSTALLED_APPS = (
:
    'django.contrib.admin',
:
)
:
mycms$ vi mycms/urls.py 
# Uncomment the next two lines to enable the admin:
from django.contrib import admin
admin.autodiscover()

urlpatterns = patterns('',
:
    # Uncomment the next line to enable the admin:
    url(r'^admin/', include(admin.site.urls)),
)
mycms$ ./manage.py syncdb
:
provide username, email, password for admin
:
mycms$ ./manage.py runserver


To access it:
Frontpage @ localhost
Admin @ localhost

Mac OSX errors
You might get “ValueError: unknown locale: UTF-8″ error, check this post on how to fix it.
http://rayed.com/wordpress/?p=1134

My Python and Django bookmarks

Posted on by
Reply

Python

This a list of Django and Python websites I found useful, I hope you find it useful as well, and may be add to it in the comments:

Deploying Django

Posted on by
1

Django is a web framework written in Python language, my favorite computer programming language.

Python a language is really great language, easy to read and understand, and very easy to learn, but for me I always found setting up Python for web development as a challenge, socially if compare it to PHP, which almost works out of the box, actually most people think PHP works only in the web server and can’t work as stand alone application, search for PHP CLI if you are interested.

I will try here to document an easy Django deployment for my future reference and hopefully it will help others:

Continue reading →

Faster file syncing with Redis

Posted on by
Reply

The problem
In alriyadh.com most of the site administration take place inside the premises of Alriyadh Newspaper offices, and as you can image the bandwidth dedicated to the website team isn’t that big. This why we designed our system to have two parts, one inside Alriyadh Newspaper internal data center where the local access is very fast, and another part accessible to the public hosted in MeduNet, and we would have a database replication for the website data, and file system replication for the web site images and media.

Current Solution
MySQL replication is straight forward and very easy to setup, and since we are mostly replicating textual data, the replication is relatively fast and reliable, given that you have a reliable Internet connectivity.

For images replication we used “rsync” that run from a cron job, rsync is very efficient with bandwidth consumption, but it had to a scan a large number of directories to see which files aren’t synced, and this usually very slow process, and this why it would take couple of minutes for some images to appear.

Redis to the rescue
Redis is key-value store, it is similar to memcached in many ways, especially being super fast and very suitable for caching, it also have tons of extra features that makes it really interesting.

The feature that caught my eyes is “Publish/Subscribe”, basically a client would “publish” a message to a given channel, and all other client who “subscribed” to the same channel would receive the message, if no one “subscribed” no harm is done and the publishing succeeded, if multiple client “subscribed” they will all get a copy of the message.

The plan was to have the CMS (written in PHP) publish a message with a name of any file changed, and fast_sync daemon (written in Python) would collect these files and copy them individually using “rsync” to the remote server, instead of syncing a whole directory.

Conclusion
faster file system syncing of course … you forgot the title already!
Seriously, the file syncing now takes few seconds, instead of couple of minutes or more, we could also do image thumbnail generation before syncing inside fast_sync daemon, and if everything fails we still have cron+rsync

Code
From CMS in PHP, this should be integrated into the CMS file manipulation code:

< ?php
$r = new Redis();
$r->connect('127.0.0.1', 6379, 0.5);
$r->publish("new_files", "/path/to/www/htdocs/img/test.jpg");
?>

“fast_sync” in “Python”, this should run all the time, may be start it using “nohup fast_sync.py &”

#!/usr/local/bin/python -u

import redis
import time
import thread
import os


files = []
lock = thread.allocate_lock()


def send_files():
    global files, lock

    while 1:
        lock.acquire()
        my_files = list(files)
        del files[:]
        lock.release()
        if len(my_files) > 0:
            command="rsync -azv %s www:/path/to/www" % (" ".join(my_files))
            os.system(command)

        time.sleep(2)


def main():
    global files, lock, base

    # Start file sending thread
    thread.start_new_thread(send_files, ())

    # Start redis queue listener
    files = []
    r = redis.Redis(host='localhost', port=6379, db=0)
    r.subscribe("new_files")
    for msg in r.listen():
        # add file to sending queue
        if msg["type"] != "message": continue
        file = msg["data"]
        print "LISTENER: Adding ", file
        lock.acquire()
        files.append(file)
        lock.release()
        time.sleep(.1)

if __name__ == "__main__":
    try: main()
    except KeyboardInterrupt: print "Stopped by user"