Django Image and File Field Caveats

Everytime I work with Image or File fields in Django I forget some tiny detail that waste 10-20 minutes until I remember what was I missing, I always say I will remeber it next time but I never do! so I made a list of common errors I keep doing while working with Image/File fields:

For complete working project: https://github.com/rayed/dj-imagefield-example

Setting MEDIA_URL and MEDIA_ROOT

Make sure you set proper values for MEDIA_URL and MEDIA_ROOT in your settings.py, e.g. I use the following structure:

my_new_site/
    apps/
       apps/
          settings.py
       gallery/
       blog/
    www/
       media/
       static/
    requirements.txt


I would put the following settings (for both MEDIA and STATIC files):

STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, '..','www','static')
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, '..','www','media')

Make sure uploaded files are accessable in development server

Serving uploaded files (media) is the job of the web server and not Django, but to ease the development process I usually make Django serve it in debug mode.

This is done by adding the following at the end of your main urls.py:

from django.conf import settings
from django.conf.urls.static import static
if settings.DEBUG:
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

HTML forms changes

Make sure you have enctype=”multipart/form-data” in your form tag, i.e.:

<form method="post" enctype="multipart/form-data">


Instead of:

<form method="post">

Don’t forget request.FILES

I always forget to include “request.FILES” in my ModelForms, and I always get “This field is required” error message!

def author_create(request, template_name='author/form.html'):
    form = ImageForm(request.POST or None, request.FILES or None)
    if form.is_valid():
        form.save()
        return redirect('author:home')
    return render(request, template_name, {'form':form})    

ImageField requires Imaging library

If you want to use Django built in ImageField you have to install Pillow Imaging library! FileField doesn’t needed it though.

View the image

Just a reminder that you can access the image URL in your template like so:

<img src="{{ author.image.url }}"  />

Git directory outside working directory

I have an old PHP website that I wanted to have its code/content versioned with Git, normally Git setup the repo directory in the “.git” directory inside the working directory, but I faced a problem, if the working directory is accessible from the web server it means “.git” is also accessible too.

Luckily git have an option to have the repo directory located somewhere else using the GIT_DIR environment variable, so here what I did:


$ vi .profile
GIT_DIR=/home/rayed/my_website_git
GIT_WORK_TREE=/var/www/my_website
export GIT_DIR
export GIT_WORK_TREE

You notice that the web site is located in “/var/www/my_website” but the repo is located under totally different directory “/home/rayed/my_website_git”, so the web server can’t access it by mistake.

Install psycopg2 (PostgreSQL adapter for Python) on OSX

I was playing with Django with Postgres backend, and I had little difficulty installing “psycopg2” the Python DB adapter for Postgres on my Mac OSX.

I’ve installed Postgres using Postgres.app for OSX which is straight forward and standard Mac app.

But when I tried installing “psycopg2″ using “pip” (the python package manager) I got an error:

$ pip install psycopg2
:
Error: pg_config executable not found.
:

I just searched for “pg_config” in my system:

$ find / -name pg_config  2>/dev/null
/Applications/Postgres.app/Contents/Versions/9.3/bin/pg_config

Then added to my PATH env and pip worked:

$ export PATH=$PATH:/Applications/Postgres.app/Contents/Versions/9.3/bin/
$ pip install psycopg2

Limit SSH to Copy a Single File Only

I want to allow host-2 to copy a file securely from host-1, so the easiest way is to use “scp” command which use “ssh” as a transport to copy the file.

If you want to do it manually it is straight forward “scp” invocation:

host-2$ scp host-1:data.csv .

But if you want to automate it you have to use “ssh” keys, but this means leaving a private ssh key on host-2 that can access host-1 without any restriction, i.e.

host-2$ ssh host-1  # FULL ACCESS NO PASSWORD NEEDED!!

A better way is to generate a new ssh-key on host-2, like:

host-2$ ssh-keygen
:
host-2$ ls ~/.ssh/id_rsa*
id_rsa
id_rsa.pub
host-2$ cat id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHBoO5JciwnRKWzbmZiZ68J7Vouim+ZUNvmsXYeCFa6TDGTmG9Wh1KhAAgQDqTuwL9BcgbOM2qiwOlLMREtH6LYLbbp9RIBIGNb0a8UL3Fka++vziHkTgaqPJ2Uq0Qd8J0oZCqseBQqSMlebO4BxOYuRMqEFn7ETR5N+SM/hq5PeuS5SVGnleJOqaO8Cq5AcoIdlYeRXjDIFw9x7DugHKP4uBTr2o+lft7seyHjYOmrWiX0+GFiDsdTzqIMC+Px3pqY8Hcd4DC2lmYDJCDG7Js3zzvzp8Xs6sBEwqZpECh8TmXZxl5/OHt8XtVCJs0lfqiHhQWFIlsYqPg+4AsjiUP

Then add the the key to host-1 authorized_keys file with one small change:

host-1$ vi ~/.ssh/authorized_keys
:
command="scp -f data.csv" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHBoO5JciwnRKWzbmZiZ68J7Vouim+ZUNvmsXYeCFa6TDGTmG9Wh1KhAAgQDqTuwL9BcgbOM2qiwOlLMREtH6LYLbbp9RIBIGNb0a8UL3Fka++vziHkTgaqPJ2Uq0Qd8J0oZCqseBQqSMlebO4BxOYuRMqEFn7ETR5N+SM/hq5PeuS5SVGnleJOqaO8Cq5AcoIdlYeRXjDIFw9x7DugHKP4uBTr2o+lft7seyHjYOmrWiX0+GFiDsdTzqIMC+Px3pqY8Hcd4DC2lmYDJCDG7Js3zzvzp8Xs6sBEwqZpECh8TmXZxl5/OHt8XtVCJs0lfqiHhQWFIlsYqPg+4AsjiUP

Notice the command part, which limit the given key to a given command.

NOTE: the public key is the same one generated from previous step on host-2

Now if you try to access the machine it will fail.

host-2$ ssh host-1
Connection to host-1 closed.

Even if you try to copy another file it will download the file you specify in the authorized_keys:

host-2$ scp host-1:data.xml .
data.csv    100%

Notice that it downloaded the data.csv and not data.xml!

“sar” command cheat sheet

“sar” is a Unix command that collect, report, or save system activity information, it is different from other system status command like “top” or “vmstat” that only show real time status only, “sar” in the other hand collect these data so you can find the system state at any time.

OPTIONS

# Live values: interval count
sar 1 3

# historical values
sar

Previous Days

# Day 11 of current month Ubuntu
sar -f /var/log/sysstat/sa11

# Day 11 of current Month CentOS
sar -f /var/log/sa/sa11

Time Range

# show from 10:00 am to 11:00 am
sar -s 10:00:00 -e 11:00:00

Data Options

sar      # CPU 
sar -r   # RAM
sar -b   # Disk

Mixing options

sar   -b   -s 10:00:00 -e 11:00:00   -f /var/log/sa/sa11  
-b   # disk 
-s   # from 10:00 to 11:00
-f   # day 11

Installation

CentOS

$ sudo yum install sysstat

Ubuntu

$ sudo apt-get install sysstat
$ sudo vi /etc/default/sysstat
ENABLED=”true”

More info:
http://www.linuxjournal.com/content/sysadmins-toolbox-sar

Solving Python virtualenv “DistributionNotFound: distribute”

After upgrading my Ubuntu machine from 12.04 to 14.04 I had this error on virtualenv wrapper:

stevedore.extension distribute
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 75, in _load_plugins
    invoke_kwds,
  File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 87, in _load_one_plugin
    plugin = ep.load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2087, in load
    if require: self.require(env, installer)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2100, in require
    working_set.resolve(self.dist.requires(self.extras),env,installer)))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 628, in resolve
    raise DistributionNotFound(req)
DistributionNotFound: distribute

After some investigation I found out the cause of the error, it seems that I’ve installed virtualenv-wrapper using pip and not Ubuntu apt-get, so when I installed it using apt-get it conflicted with the pip installation.

Solution

  • Remove virtualenvwrapper Ubuntu package: sudo aptitude remove virtualenvwrapper
  • Remove virtualenvwrapper pip package: sudo pip uninstall virtualenvwrapper virtualenv-clone virtualenv stevedore
  • Reinstall virtualenvwrapper Ubuntu package: sudo aptitude install virtualenvwrapper