FreeBSD 7 is available, many new improvements performance and feature wise, check the announcement.

One of the most impressive improvements is SMP (Symmetric multiprocessing):

Dramatic improvements in performance and SMP scalability shown by various database and other benchmarks, in some cases showing peak performance improvements as high as 350% over FreeBSD 6.X under normal loads and 1500% at high loads. When compared with the best performing Linux kernel (2.6.22 or 2.6.24) performance is 15% better

After serving Alriyadh.com the past 3 years, I am replacing Swish-e with Solr search server, Solr is a search server based on Lucene text search engine.

Solr has many advance features I didn’t utilize most of them yet, but hopefully you will see them soon. I will write a post about Solr and how it works and its integrate with PHP soon.

For now please have a look at it and tell me what you think.

Old MySQL versions didn’t have good support for Arabic, so what I used to do is to configure MySQL tables with any character set e.g. latin-1 and configure PHP to read it and write in Windows-1256 and pass it to MySQL as is i.e. MySQL think it is “Latin-1” but in fact it is “Windows-1256”.

Needless to say cheating in the character set have many bad consensuses:

• Sorting will not work as expected
• Other applications will assume you are using the correct character set and you won’t be able to read or write your data. e.g. PHPMyAdmin could display something like this:
  “ÈÓã Çááå ÇáÑÍãä ÇáÑÍíã”

How to fix it
I found this trick in MySQL documentation, what you need to do is convert any character field to “blob” then convert it back to the desired field type with the desired character set:

ALTER TABLE t1 CHANGE c1 c1 BLOB;
ALTER TABLE t1 CHANGE c1 c1 TEXT CHARACTER SET cp1256;

You can change it and test it from phpMyAdmin.

Warning: It is the same design converted to CSS.

After 5 weeks of development and testing, Alriyadh.com is now converted to CSS based design, it still not 100% complete.

You can view the design by using this URL:
www2.alriyadh.com

Some benefits of moving to CSS:

• Fonts look the same a cross browsers: I am using Yahoo! User Interface Library fonts file to guarantee that font have the same size a cross browsers.
• Page are bit smaller: because many presentation tags are now removed from the HMTL and put in the CSS.
• Printer friendly by default: we now have have separate CSS file for printing, so no need for printer friendly page since all page are automatically are.
• PDA friendly by default: same as printer CSS, not implemented yet.
• Rendering is faster.
• It will simplify future redesign. (since we are talking about redesign, suggestion are welcomed 🙂 )

After moving alriyadh.com to Lighttpd I had some problem making a CakePHP application I wrote to work properly.

I searched the net for a solution and I found this post, but unfortunately the solution they proposed didn’t work, I spent several hours trying to make it work until I gave up, I decide to do it from scratch and luckily it worked.

The mod_rewrite rules I used in lighttpd.conf file:

url.rewrite-once = (
"^/cake/(css|files|img|js|stats)/(.*)$" => "/cake/app/webroot/$1/$2",
"^/cake/(.*)$" => "/cake/app/webroot/index.php?url=$1"
)

Where cake is you cakePHP application.

When you build AJAX based form, you need to provide visual feedback to tell the user that something is really happening.

I created small template for this purpose:

Ajax Form

Hopefully you will see it in the next alriyadh.com CSS based theme 🙂

Inpired by Flickr’s Dirty Little Secret post, about how Flickr thumbnails make the photo prettier than the original photo.

The article suggest that Flick sharpen the image before it resize it.

So I did my own test, this the result and you are the judge:

Image 1, normal resize.

Image 2, sharpen before resize.

Sharpened and saturated +100:

All I need to know is how to do it from inside PHP.

Alriyadh.com website is now served by Lighttpd (aka Lighty).

I moved all static content from Apache to Lighty, Lighty “select” based design makes it ideal for I/O bound operation, i.e. disk and network operations. For this reason Lighty move all CPU bound operations to external process and communicate with them using FastCGI protocol.

This is what delayed my migration to Lighty, because I used to compile PHP with mod_php only and with FastCGI support, so I have to recompile PHP to enable it. Of course I spent few days testing, and I am still having some areas not covered completely.

Recently I have improved alriyadh.com voting system, many of them are in administration area, some of these changes will be visible in the near future.

One of the new additions is the previous votes page:
http://www.alriyadh.com/php/vote/

Check it out, and keep in mind the design is still under work.

Memo to self: replace the word “vote” with “poll”

UPDATE: If you see weird colors in IE (7?) it is fixed now.

XCache is relatively new PHP code optimizer, written by “Jan Kneschke” the same guy who wrote Lighttpd web server.

After migrating Alriyadh.com to the new dual CPU servers, I had problems with APC PHP optimizer and locking, so I disabled the optimizer altogether, after all the CPU was very fast and it handled the load without any problems.

But lately with my increasing interest on Lighttpd I noticed XCache, I never heard of it before but after reading the web site, and the problem APC have with multi processors I decided to give XCache a try, and the best of all it was ready in FreeBSD ports, so I installed it and so far it doing a good job.

Installing Java on FreeBSD (my favourite operating system) wasn’t an easy task, you have to download Java files from Sun web site put them in a special directory then start a lengthy build process that never successed.

Fortunately FreeBSD Foundation has binary package for Java, all you need is to Download and install using:

# pkg_add diablo-jdk-freebsd5.i386.1.5.0.07.01.tbz

and you are done.

With Java on FreeBSD may be I will start playing with Lucene instead of Swish-e as Alriyadh search engine.

These days I really don’t have time on the Internet, but I really want to talk about some of these issue in more depth later.

• My phone line (including DSL service) is out of service for more than month, and until now no hope 🙁
• IGW the new ISU: STC is now handling Internet gateway operation, the service is still has some issues, and we are still understaffed.
• Dev4Islam: Project to devlope open source Islamic and Arabic software and services. We aim to collect data and make it available for other so they can build better software.
• Java + Delphi = C# Anders Hejlsber the creator of Delphi, create a great language for Microsoft.
• Lucene is a full text search library, exteremly fast, and the API is awsome. I tested it Lucene .Net port it is really super fast.
• Swish-e same as lucene, this what I use for Alriadh.com, still very good.
• Alriyadh.com new feature: user login for better experince, no need to rewrite your name and email when you want to submit a comment or send article to a friend, and you can bookmark your favourite articles, any thing else we should add.

Screen Shot of Quran program:

Pingback is a method for Web authors to request notification when somebody links to one of their documents. This enables authors to keep track of who is linking to, or referring to their articles.

Implementing Pingback feature is very easy, you need to add the following code to your template:

<link rel="pingback" href="http://www.alriyadh.com/php/xmlrpc/" />


This code will make other blogging software notify your software that it is linking to it, It do this by sending “pingback.ping” XML-RPC request to the given URL, when you get the notification you can add it to the database, email it, or do what ever you want with it.

Cricket

Cricket is trend monitoring software, it allows you to monitor trend for any device you want.

I wrote 2 extensions for Cricket to monitor Lighttpd web server, and Memecached caching server.

Lighttpd web server

“lighttpd (also called “Lighty”) is a web server which is designed to be secure, fast, standards-compliant, and flexible while being optimized for speed-critical environments. Its low memory footprint (compared to other web servers), light CPU load and its speed goals make lighttpd suitable for servers that are suffering load problems, or for serving static media separately from dynamic content. lighttpd is free software / open source, and is distributed under the BSD license.” Wikipedia

For the past 9 years I used Apache Web Server exclusively, Apache is very stable and feature rich web server. The only downsize is consuming a lot of computer resources, Apache is a multi-processes server which mean you will find many Apache process running in your system and if we say that each process cosume 2MB of RAM and you have 5 Apache processes running this mean it needs 10MB of RAM.

The last 3 days we were facing many problems with Alriyadh.com web server, the server suddenly get very slow, and all requests times out. It took me very long time to identify the problem, I tried many things and every time I think it will solve the problem completely, but after few hours the problem appear again and he server start to crawl.
Until this moment I am not 100% sure the problem is solved, but during the process I learned many things, so I’ll share it with you and hopefully everything will go smoothly, if it doesn’t I have to learn new thing to figure out the problem.

A year ago Waleed Alzuhair asked me about the reason I am still using Windows-1256 Arabic character set, intead of the new standard Unicode. My reason was the lack of good editor that support Arabic and Unicode, my favourite eidtor HTML-Kit doesn’t have any support for Unicode.

But lately while looking for an editor for Mac OS I found JEdit, it is a Java based editor which means it will work with any OS, it supports many charachter sets including Unicode UTF-8, another intersting feature is its support (through a plugin) of remote editing using both FTP, and SFTP (HTML-Kit doesn’t support SFTP).

Alriyadh Newspaper is giving away 25 one year subscription, one subscription for each game day in FIFA world cup 2006, all you need to do is to guess the result of each match, of course the more matches you get correct the more chance for you to win, to vote go to alriyadh world cup site http://www.alriyadh.com/worldcup/, and it is absolutely free, no need for 5 SR SMS to subscribe, and if you get the result right you really have good chance to win.
Update:
“All” correct guesses will get 50% discount on Arliyadh Newspaper subscription, so it will become 300SR instead of 600SR.

Today I had to finish writing some code for Alriyadh Worldcup web site, and I was expecting it to take at least 5 to 6 hours, and it had to be finished today, so I planned all my evening for that coding session

Fortunately I finished the job in 2 hours, and I had some extra time in my hand, so I decided to do something I planned to do for very long time, upgrade WordPress, so I sat down and upgrade it from version 1.5 to the latest version, alhamdu lelah the upgrade process was very smoth, althogh some features didn’t work, mainy the rich text editing feature.

Alriyadh.com launched a new website for covering the FIFA world cup 2006, the website can be found here http://www.alriyadh.com/worldcup.

The new site isn’t a regular news website, it is a new web application specifically designed for the world cup, it would be also updated in real time, i.e. you can follow any match events while it is still in play, of course you will still find regular coverage like news, and photos.

Yesterday, Alriyadh.com awarded the Middle East ICT Excellence Award in the “Print and Electronic Media of the Year (Print)” category from the Middle East Excellence Awards Institute.

Alriyadh Newspaper covered the story, and we received a great response from many people, thanks everyone:

• «الرياض»تحصل على جائزةالشرق الأوسط في تقنية المعلومات والاتصالات لأفضل موقع الكتروني
• «الرياض» تحصل على جائزة أفضل موقع إلكتروني في الشرق الأوسط
• موقع «الرياض» الإلكتروني.. عمل دؤوب وتطوير دائم(screen shots for alriyadh.com since opening)
• السحيباني: انتظروا المزيد من التطوير في موقع «الرياض» الإلكتروني
• شباب سعودي يتوج «الرياض» الإلكتروني باللقب
• تقنيون سعوديون: فخورون بفوز «الرياض» .. والجائزة ذهبت لمن يستحقها
• قراء «الرياض» يتفاعلون مع الفوز

Special thanks to Ahmad Alsuwailm for the great coverage.

FreeBSD my favorite operating system released version 6.1, they finally put the new logo.

For more information about FreeBSD checkout FreeBSD@WikiPedia

Check it out:
404 Page

and give me you feedback, what to you think it should contain.

PHP 4.0.5 and above added new feature to compress your PHP pages on the fly, all you need is the zlib extension, this how I configured it.
# vi /usr/local/etc/php.ini
:
zlib.output_compression = On
:
# apachectl graceful


Alriyadh.com front page size dropped from 20.5 K to 5.6 K, according to my tests almost 50% of browser don’t support compression, IE for instance turn off gzip compression support if you have bad firewalls (Zone Alarm), and even some IE extensions (specially ad wares) turn off the compression.

Several month ago I researched caching methods to use with Alriyadh.com, it was essential because we used old and slow hardware.

The result of my test showed that MySQL based caching was superior to memcached caching.

I did a new benchmark comparing MySQL, PHP memcache extension, and PHP memcache class.

The text is done by fetching the result of a complex query 1000 times, and calculated the time it took using my bechmark library (link dead!), times in seconds:

Today alriyadh.com main website migrated to FreeBSD, now the complete website runs on FreeBSD on HP servers.

The ads appear now as if they are local images, awesome 🙂

You will notice that the maximum CPU utlization is 29% at around 7:30 am, at this time the old server would hit 100% CPU utlization.

The 29% is before enabling APC (Alternative PHP Cache) which replaced PHPAccelerator I used with the old server, you can see a major drop in CPU utilization after enabling it.

To find out how fast our new servers are I run a small test using ab Apache HTTP server benchmarking tool.

I’ll send 10,000 requests using 10 simultaneous processes, I run the same test on both old and new server:

Old Server

$ ab -c 10 -n 10000 www.alriyadh.com/zzz.txt

:

Time taken for tests:   78.172 seconds

:

New Server

$ ab -c 10 -n 10000 new-www.alriyadh.com/zzz.txt

:

Time taken for tests:   3.851 seconds

:

This mean that the new server is 20 times faster than the old server, the test is really unfair because the old server is running on production and serving other requests. I’ll repeat the same test after migration to see the real numbers.

Alriyadh.com is hosted at MeduNet, they unpacked the servers on the lab for now, and hopefully they will prepare the racks and install the servers next week.
Mean while I went to MeduNet and installed FreeBSD 6 on the 3 servers, fortunately I brought 2 installation CD with me which allowed me to work on 2 servers concurrently. The whole installation process took 1 hour for the 3 servers, of course this a basic setup to allow me to connect remotely to continue installing required software.
I went back home and continued working on the servers, it took more than 7 hours to update the system and install the required software, specially Cricket which required a lot of time and configuration.
I also started migrating some of the data from the old servers to the new ones, and to tell you the truth the link between those servers are extremely slow, I left rsync running for overnight and it only copied 1.7G out of 10G.

rsync is a really smart file transfer tool, when you want to move a large directory from machine to machine it is the best tool you can use.

The tool utilise ssh protocol (among others) to transfer files, this is a sample command:

host_1# rsync /some/dir host_2:/other

The above command will copy the directory /some/dir from host_1 to host_2, the smart thing about rsync that it check what files are already copied so it won’t copy them again.

alriyadh.com new servers arrived, we are upgrading from 4 Sun Solaris machines to 3 HP machines with FreeBSD, the machines specification as follow:

• Dual Xeon 3.40GHz
• 1GB RAM
• 4 x 36GB Hard disk

The servers should be in production next week, you should notice significant performance improvements Insha Allah.
Other website enhancements will follow that wasn’t possible with the old hardware.

It is still 10:54 pm and until now we have around 1200 comments:

• 923 Viewed.
• 65 Pending.
• 207 Rejected.

and I complain when I get 3 spam messages.

الله يعينكم والله

alriyadh.com introduced new service called “Alriyadh eBook”, basically they collected articles of selected writers and group them in one downloadable PDF file.
We used to get hundreds of request for some writers, specaially “فهد عامر الأحمد” who write very informative and in the same time amusing articles.
http://www.alriyadh.com/static/ebooks/

I really hate Alexa, and I never trust their ranking method, you must install their toolbar to get counted but I never saw any one using their toolbar, but unfortunately it is the only was to measure website popularity.

Any way I am going to use A9 toolbar (which is part of Amazon, who in turn own Alexa), and I added Alexa Badge in the bottom of my page to see how this will affect my website ranking.

I developed a new Javascript based feed for alriyadh.com that can be included easily in any website, all you need is cut-n-paste some HTML code (mostly Javascript) into your web page and you will get all the news you need.
The script is customizable you can select the following:

• What section you want to view.
• How many links in each section.
• How do you want to view the links.

Give it a try and tell me what do you think, remember it is still under development, so please don’t complain about the design 🙂
http://www.alriyadh.com/static/feedjs/

We had today a major DDoS attack (Distributed Denial of Service), we get thousands of concurrent connection from all over the globe. It took us a while to figure out the problem, the attack still active but hopefully we managed to take the site online again for Saudi Arabia users.

www.alriyadh.com added a new advertising banner to all pages, they now show 6 ads in a single page, I am not sure what they are thinking why don’t they increase the prices, and put fewer ads. I hate it.

When reading alriyadh.com readers comments, I found many readers using bad practices when posting comments for example:

• Repeating one character many times, e.g. “……”, “!!!!!”, “?????”, etc..
• Using empty lines
• Using Kasheda to format Arabic strings.
• Using more than one space between words.
• Misspelling “لكن”

So I wrote a filter to remove all these practices.

What do you think? good or bad idea, please tell me what do you think.

alriyadh.com added multiple usability improvements to comment posting form:

• To prevent duplicate posting, the submit button is disabled after the first click.
• The user name and email is now remembered, so next time it will be auto filled.
• Focus is set to the first empty field in the form.
• Client IP is now recorded to detect fake posting, some users sends multiple posts with different, this will help to detect them.

Everyday at 11pm, alriyadh.com team start working on the next day issue, the job involve copying all contents from the printed version to the alriyadh.com content management system, the process involve 2 to 3 team members, and takes about 4-5 hours to complete.

But last Thursday was a special case, due to a planned vacation for one team member, and family issues for another member, Faisal Alenzi was the only guy working that night and he carried the whole show and published the whole issue on time by him self.

My friends at Alriyadh Newspaper published my son news on todays issue:
Abdullah Alrashed
Thank you guys 🙂

I am not a “speeling bee champion” -the previous sentence proves it 🙂 -, and I usually don’t care about spelling mistakes of others, but some spelling mistakes drive me crazy.

One of the most common in Arabic is:
لاكن
This is wrong, if you write it like this please stop, it is written:
لكن

I hope Google would support Arabic in their toolbar spell checker.

P.S: If you noticed any spelling mistakes in any of my post please let me know.
P.S: I used to write “great” as “grate” for many many years, but thanks to my friend “Fayez Alazmi” who corrected me one day I don’t write “grate” that often any more 🙂 .

Alriyadh.com introduced new style to display comments, see the Sample Article

The new style uses CSS instead of tables, which make it much smaller and cleaner.

A link to each comment is also added -the time of the post- to allow bookmarking and sending URLs to specific comment.

The posting form is now viewed in the same page, it is hidden by default but after clicking “Post Comment” it will appear thank to the power of Javascript.

This is an experimental page that shows the articles with most comments:
http://www.alriyadh.com/php/comments/comments_top.php

Hopefully it will find it self in the front page shortly, I don’t where should we put it, the front page is little crowded as is.

Today Alriyadh.com opened comments on all articles, instead of articles in the “columns” section only.

In the first day of opening comments the amount of comments posted today tripled the average daily volume.

This means the work on the Internet team will need to work three times harder than they used to be, since all of user comments need to be moderated!

This is a huge step from being just an electronic copy of the printed newspaper, to becoming a real Internet website.

I tested different types of caching, here is my times:

Altought File caching is much faster that the other two methods, it can suffer from file locking problems, and can’t be shared between more than one server.

memcached client used is written in PHP which will slow it down a little bit.

So MySQL seems the ideal choice 😉

When I started building alriyadh.com website I decided to generate all pages as static HTML files, I relied heavily on Server Side Includes (SSI) to build complex pages.

The main reason for this decision was the fact that the server was very slow on processing PHP files, espcially with a site with high load.

Generating static HTML files has its advantages but it also had many problems:

• Regenerate everytime you want to test a change
• Changing the design will cause regenrating the whole website, which could take forever.
• Generating complex pages is very hard, I had to rely on SSI.
• What to do with pages that change frequently, e.g. pages with comments.
• When importing old data, you need to generate them as well.
• Static HTML files performance advantage for frequently used data, like todays issue, but for old issues it just take a lot of disk space.

Soon I’ll start testing dynamicly generated of pages with these points in mind:

Today alriyadh.com broke the world record for the largest number of ads in a single page, today they added yet another banner to become 7 banners in thier frontpage.

Swish-e (Simple Web Indexing System for Humans – Enhanced) is really nice small program that index almost anything, I am playing with it at the moment and I am thinking of dumping my own search engine and replace it with Swish-e.

Swish has many features but the one feature that stroke me is the speed:
66,548 files indexed. 164,076,410 total bytes. 26,496,024 total words.
Elapsed time: 00:02:44 CPU time: 00:02:31
Indexing done!