Category Archives: Uncategorized

Accelerating Postgres connections with PgBouncer

PgBouncer is a lightweight connection pooler for PostgreSQL, connection pooling makes Postgres connection much faster, which is important in Web applications.

Here I will explain the steps I used to configure it under Ubuntu 14.04 LTS.

Step 1: We would configure users allowed to connect to PgBouncer:

$ sudo vi /etc/pgbouncer/userlist.txt
"rayed"  "pgbouncer_password_for_rayed"

Step 2: We configure databases PgBouncer will pool for, and how PgBouncer will authenticate user:

$ sudo vi /etc/pgbouncer/pgbouncer.ini
:
[databases]
rayed = host=localhost user=rayed password=postgres_password_for_rayed
:
auth_type = md5
;auth_file = /8.0/main/global/pg_auth
auth_file = /etc/pgbouncer/userlist.txt


The default value for “auth_type” is “trust” which means a system user “rayed” will be allowed to connect to Postgres user “rayed”, I change to “md5″ to force a password checking from the file “/etc/pgbouncer/userlist.txt”.

Step 3: We will allow PgBouncer to start:

$ sudo vi /etc/default/pgbouncer 
:
START=1
:


The default value is “0” which means don’t start PgBouncer ever, it is a way to make sure you know what you are doing :)

Step 4: Starting pgBouncer:

$ sudo service pgbouncer start

Step 5: Test the connection, by default “psql” connect using port “5432”, and pgBouncer use “6432”, so to test a pgBouncer connection we would use the following command:

$ psql -p 6432 


If you get “Auth failed” error make, make sure the password you entered is the one you typed in step 1, if the command doesn’t ask for password try it with “-W” option, e.g. “psql -p 6432 -W”.

Translation in Django

Settings

In your project “settings.py” setup the following values:

LOCALE_PATHS = (
    os.path.join(BASE_DIR, 'locale'),
    )
LANGUAGE_CODE = 'ar'

Here we defined the location of our translation files, by default Django will look for it under application directories under “locale” directory, but here we define it for the whole project.

The LANGUAGE_CODE line define a fixed translation to Arabia “ar”.

Source Code

For Python source code Django uses “ugettext” function aliased as “_” (underscore) to translate text strings:

from django.utils.translation import ugettext as _
from django.http import HttpResponse
from django.shortcuts import render

def page1(request):
    output = _("Welcome to my site.")
    return HttpResponse(output)

def page2(request, template_name='index.html'):
    return render(request, template_name)

Template

Inside Django templates, Django uses trans template tag with the text to translate, don’t forget to load the tag using “{% load i18n %}”

{% load i18n %}

<h1>{% trans "Hello World" %}</h1>

Commands

After preparing the code we the following steps:

  • Collect translation string using “django-admin.py makemessages” command.
  • Edit the translation file “django.po”
  • Compile the translation to “django.mo”.

$ cd project_home

# Make the "locale" directory to store translation data
$ mkdir locale

# scan the project for translation strings
$ django-admin.py makemessages -l ar

# Edit the translation file and add your translation
$ vi locale/ar/LC_MESSAGES/django.po
:
msgid "Hello World"
msgstr "مرحبا يا عالم"
:

# compile django.po to django.mo
$ django-admin.py compilemessages

That’s it, you should be able to see your applications translated!

Graph you Data and Email It

I have a new website and I want to know the number of new signup every day, so I wrote a small script new_users.sh that will print the number of new signups today.

$ new_users.sh
280

I ran this script daily using a cron job and add it to user.dat file:

0 0 * * *   /home/rayed/bin/new_users.sh >> /home/rayed/var/user.dat

After few days the file will look like this:

#Users
50
104
202
298
290
289
291
310
311
280

I could send this file daily and read, but it wouldn’t give a good picture of how new user signup is changing by time, so the next logical step is to convert it to a graph for easier understanding.

I used gnuplot to convert the textual data to a graph, and automatically email it to me.

So I wrote the following small script, email_graph.sh:

#!/bin/sh

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

from_email="rayed@example.com"
to_email="rayed@example.com"

plot="
set terminal png \
	medium \
	size 800,400;
plot  'user.dat' with lines;
"

img_base64=`echo $plot | gnuplot | base64`

sendmail $to_email <<EOF
From: <$from_email>
To: <$to_email>
Subject: Plot and Inline image from CLI
Mime-Version: 1.0
Content-Type: multipart/related; boundary="boundary-example"; type="text/html"

--boundary-example
Content-Type: text/html; charset="US-ASCII"

This email sent from Linux CLI:
<br>
<IMG SRC="cid:plot_image_1" ALT="Plot">

--boundary-example
Content-ID: <plot_image_1>
Content-Type: IMAGE/PNG
Content-Transfer-Encoding: BASE64

$img_base64

--boundary-example--
EOF

When you run it you will receive the following graph on your email:

plot

Of course you can edit the email HTML template, add new graph, or do whatever you like to customise it.

I hope you find it useful.

Free Electronics Video Courses

PyroElectro have decent and recent courses on electronics that cover different fields.

Each course is 10 videos ranging from 7 to minutes, no prior knowledge to electronics needed.

Currently they provide 4 courses:

They also have 2 more courses planned:

  • FPGA
  • Sensor

Courses are free so check them out:
PyroEdu

Thank you PyroEdu for the great courses, and thank you for you Kickstarter supporter who make it happens.

Ubuntu new server checklist

Create Admin User

As root create new user for management, after that you should never use root:

root# adduser myuser
root# passwd myuser
# Add user to sudo group
root# usermod -a -G sudo myuser

You should logout from “root” and login again using your new “user”

Setting Up Admin User

Add your public key to the admin user for password less access

$ mkdir ~/.ssh
$ vi ~/.ssh/authorized_keys
:
paste your public key e.g. id_rsa.pub
:

Setup

Change the default editor from nano to vi (if you want):

$ sudo update-alternatives --config editor

Setup system update without a password:

$ sudo visudo 
:
Cmnd_Alias APTITUDE = /usr/bin/aptitude update, /usr/bin/aptitude upgrade
%sudo ALL=(ALL) NOPASSWD: APTITUDE

Fix the timezone:

$ sudo dpkg-reconfigure tzdata

Install “ntp” if not already installed:

$ sudo aptitude install ntp

Change hostname

$ sudo hostname s5.rayed.com
$ sudo sh -c "echo 's5.rayed.com' > /etc/hostname "
$ sudo vi /etc/resolv.conf
:
domain rayed.com
search rayed.com
:
$ sudo vi /etc/hosts 
:
178.79.x.x  s5     s5.rayed.com
:

Update the machine

$ sudo aptitude update
$ sudo aptitude upgrade
$ sudo reboot

Configure Mail Server

I usually install Exim mail server as “internet site” but listening to localhost only, this the needed commands:


sudo aptitude install exim4-daemon-light
sudo dpkg-reconfigure exim4-config

Try sending an email to your self, and check the log:


date | sendmail rayed@example.com
sudo tail -f /var/log/exim4/mainlog

Django returning JSON for AJAX requests

In your views.py you can have a page that return JSON data for AJAX request like this:

import json
from django.http import HttpResponse

def ajax(request):
    data = {}
    data['something'] = 'useful'
    return HttpResponse(json.dumps(data), content_type = "application/json")

This would work fine if you fill data your self, but if you getting a data from a model try the following:

from django.core import serializers
def tasks_json(request):
    tasks = Task.objects.all()
    data = serializers.serialize("json", tasks)
    return HttpResponse(data, content_type='application/json')

Setting Up Python and Supervisor on CentOS

CentOS default repository is very limited, and even if you install EPEL you will get old packages, in my case I needed to install Supervisor to manage my Django application, after trying to do it manually and through EPEL I ended up with the following setup.

Install Needed Package

sudo yum install python-setuptools
sudo easy_install pip
sudo pip install supervisor

Setup Supervisor

We’ve already installed “Supervisor” globally, but we need to create its configuration, luckily it comes with default config:

echo_supervisord_conf > supervisord.conf
sudo cp supervisord.conf /etc/supervisord.conf
sudo mkdir /etc/supervisord.d/
sudo vi /etc/supervisord.conf
:
[include]
files = /etc/supervisord.d/*.conf
:

Next we need to set “Supervisor” to run automatically every time you restart your machine, we need to create /etc/rc.d/init.d/supervisord with the following content:

sudo vi /etc/rc.d/init.d/supervisord
#!/bin/sh
#
# /etc/rc.d/init.d/supervisord
#
# Supervisor is a client/server system that
# allows its users to monitor and control a
# number of processes on UNIX-like operating
# systems.
#
# chkconfig: - 64 36
# description: Supervisor Server
# processname: supervisord

# Source init functions
. /etc/rc.d/init.d/functions

prog="supervisord"

prefix="/usr/"
exec_prefix="${prefix}"
prog_bin="${exec_prefix}/bin/supervisord"
PIDFILE="/var/run/$prog.pid"

start()
{
       echo -n $"Starting $prog: "
       daemon $prog_bin --pidfile $PIDFILE
       [ -f $PIDFILE ] && success $"$prog startup" || failure $"$prog startup"
       echo
}

stop()
{
       echo -n $"Shutting down $prog: "
       [ -f $PIDFILE ] && killproc $prog || success $"$prog shutdown"
       echo
}

case "$1" in

 start)
   start
 ;;

 stop)
   stop
 ;;

 status)
       status $prog
 ;;

 restart)
   stop
   start
 ;;

 *)
   echo "Usage: $0 {start|stop|restart|status}"
 ;;

esac

Then make sure CentOS knows about it:

sudo chmod +x /etc/rc.d/init.d/supervisord
sudo chkconfig --add supervisord
sudo chkconfig supervisord on
sudo service supervisord start

Sample Supervisor App

Here is a sample of Django App to be controlled and monitored by Supervisor, just put it:

sudo vi /etc/supervisord.d/my_django_cms.conf
[program:my_django_cms]
directory=/var/www/dashboard/www
command=/home/rayed/.virtualenvs/dev/bin/gunicorn apps.wsgi:application -b 127.0.0.1:8080 --workers 8 --max-requests 1000
# UNIX Socket version (better with Nginx)
#command=/home/rayed/.virtualenvs/dev/bin/gunicorn apps.wsgi:application -b unix:/tmp/my_django_cms.sock --workers 8  --max-requests 1000
environment=DJANGO_ENV="prod"
user=rayed
autostart=true
autorestart=true
redirect_stderr=True

After that:

 sudo supervisorctl add my_django_cms
 sudo supervisorctl start my_django_cms

Setting Apache to Proxy to Gunicorn

To add virtual host to Apache that forward dynamic content to Gunicorn:

sudo vi /etc/httpd/conf.d/my_django_cms.conf
NameVirtualHost *:80
<VirtualHost *:80>
    ServerName dj.example.com
    ServerAdmin webmaster@example.com
    DocumentRoot /var/www/my_django_cms/www
    RewriteEngine on
    ProxyPreserveHost On
    RewriteCond $1 !^/(favicon\.ico|robots\.txt|media|static)/
    RewriteRule ^(.*) http://localhost:8000$1 [P]
    <Proxy *>
            Order deny,allow
            Allow from all
            Allow from localhost
    </Proxy>
</VirtualHost>

If you have SELinux enabled you might need to apply the following command:

setsebool -P httpd_can_network_connect 1

If you have issues accessing your statics files from Apache it might be SELinux also:

restorecon -Rv /var/www/my_django_cms/