Application Focus

PHPStorm has released version 4 last month. Our license enables us to freely upgrade PHPStorm until Feb next year, so why not?

There are a few features that come with version 4, you can have a look at here if you are interested, but I am more interested in the one called “Code Coverage for PHPUnit”. I will detail in the following on what I did to get it working.

Firstly I enabled Xdebug on my MAMP installation by update the correct php.ini file. For my instance it is in file “/Applications/MAMP/conf/php5.3/php.ini”, find the following line in your file and uncomment it:

[xdebug]
zend_extension="/Applications/MAMP/bin/php5.3/lib/php/extensions/no-debug-non-zts-20090626/xdebug.so"

Restart your apache server and you should be able to find the xdebug section from your phpinfo();

Now open PHPStore’s preference and select PHP from left side, and choose “Xdebug” for Debugger:

Then setup PHPUnit for your project, my configuration looks like this:

Now I am able to run the test by clicking on the “Run <project> with coverage” command:

After running the test with coverage, I can see the report on my project:

And it also highlights in the file which lines are covered (green) and which lines are not covered (red) by giving appropriate colors on each line:

Now I know exactly how my code is covered and how to fix them directly inside the IDE.

I regularly need to fill in migration/tourist visa application forms for my families. I have done it numerous times and I am sure I will continue doing so in the next few years.

Initially I had to print out the PDF forms online and fill in using pens, if I made a mistake, I had to print out another and re-fill the page that had errors. This is very cumbersome and waste of paper. Recently I have found a tool called CutePDF Form Filler. What it does is to allow us to fill in the PDF forms electronically and then we can save it as the normal PDF document, so that you can just open it in normal Adobe Reader.

This is really handy as all I need to do is fill in the form on my computer and then save it on the cloud, and only print it when I have double checked everything and make sure no mistakes.

It is a commercial product, but I think you can just use it forever as long as you don’t mind the little message mentioning about CutePDF on the first page of the saved PDF document after trial finishes.

The most important thing is that it does the job.

After I replaced my old hard drive with 1TB seagate hard disk from MSY, I tried to install dual boot of Windows XP and Ubuntu with 50GB and 30GB partitions respectively, but never got success. Everytime I got Ubuntu installed, Grub2 always complained about unrecognised partition and unable to boot, I had to remove the Linux MBR to allow Windows to boot. The reason being that somehow my BIOS is unable to read the boot partition created by Linux, some people on the Internet saying that some system’s BIOS has limitations if the boot partition is too far from the beginning of the disk. I never found a solution to fix this issue.

My old hard disk has bad sectors, that’s why I have to replace it as Windows kept failing due to corrupted system files. I gave a bet and managed to re-partition the whole disk and allocate 30GB to install Ubuntu. I was lucky that the first 100GB of disk space can be re-claimed, however the rest of 220GB Ubuntu is unable to format due to bad sector. Anyway, as long as I can get Ubuntu going, I don’t worry much about the loss of this 220GB space, this hard disk was planned to be throw away anyway.

Now I have two system running on separate hard disk. Windows XP is running on the new hard disk and Ubuntu 12.04 is on the old disk. They are running very happy with each other so far. Here are some screenshots for Ubuntu:

Dash Home:

Ubuntu Software Centre:

CompizConfig Settings Manager:

Windows switching:

Workspace switcher:

I am still struggling with installing PPStream to watch videos online, but so far it is quite good, very fast start up and shut down and compiz is awesome, although with some bugs that sometimes kill itself.

Next step is to get my webcam working with Skype. Wish me luck.

I was trying to install Ubuntu 12 on my existing Windows XP installation on a separate partition, using the image downloaded and burned onto a CD-RW. However, after the installation completed and my PC rebooted, I got an error after the POST screen:

no parition found
grub rescue >

The screen stayed there and I can’t reach the point to choose which OS to boot from. Looks like the new version of Grub has trouble finding the correct partitions on my hard drive. I have installed previous version of Ubuntu without problems under the exact same partition structure on my disk.

I have tried installing it twice, one with single partition and another one with “/” and swap partition, however none of them were working.

I had no choice but needed to remove the Ubuntu’s boot loader from MBR and bring back my Windows XP, otherwise my system will not be useful at all, as I can’t boot into Windows anymore.

To remove the Ubuntu boot loader, I have to boot back into Windows installation CD and using the Recovery Console to fix the boot issue.

After system recovery console is loaded

C:\WINDOWS> cd ..
C:\> fixboot C:

Now my Windows is back, I need to find out what caused the issue and will try to install it again.

I have been with iiNet for almost 5 years. Since I switched to Naked DSL plan around 2 years ago, although iiNet had upgraded our monthly usage quota a few times from 10GB a month ( peak time ) to current 75GB a month, my experience is still very frustrating. The reason being that Naked DSL plans counts both uploads and downloads towards the monthly usage quota.

We video chat with family member overseas everyday using either Skype or QQ ( Chinese social and chatting software ), which heavily download and upload videos at the same time. I am sick of keeping worrying about the quota usage and logging into iiNet’s toolbox web interface to check the amount we have used, which does not work well most of the time ( keep redirecting to iiNet web mail for some reason ). I have been thinking about switching to TPG for a long time, after seeing its ads for unlimited ADSL2+ plans, but just lazy and worry about the downtime that we will have while switching.

But enough is enough, I have called TPG today and confirmed that there will be almost no downtime at all when switching the service from iiNet to TPG, most likely will be around 5-10 minutes. And I have also confirmed with iiNet that I will not be charged for cancellation and any unused service will be refunded at the time of cancellation.

I also checked with my friends who are TPG customers and they seem to be happy with the service. So I called TPG again tonight and they confirmed doing the transfer which will happen in about 10-20 working days.

The new service will cost $10 a month less than what I am paying currently, not taking into account the connection fee ($80), but I am very happy with it.

Now I can finally enjoy the unlimited internet and stop worrying about the damn QUOTA.

We have just done another hadoop deployment to our processing system to process our demographic data on daily, weekly and monthly basis. This is the third Hadoop based processing release in about a month. Everything goes really well and no problems found so far.

We will continue with our fourth new feature based on Hadoop’s power in the coming weeks.

Great work team!

Disk might be cheap, but when it comes to deal with TB of data, you might want to consider compression for your data storage.

When you want to create a table with compression enabled, you will need to use “STORED AS SEQUENCEFILE” when create a table in Hive:

CREATE TABLE compressed_table (data STRING)
   ROW FORMAT DELIMITED
   FIELDS TERMINATED BY '\t'
   STORED AS SEQUENCEFILE;

However, you will not be able to use “LOAD DATA” command to load data from text file into this compressed table, Hive will complain about the file format.

There is a trick to by pass this, however. What you need to do is to create a temp table to hold the data from file as “LOAD DATA” command will work for normal text file storage, and “INSERT OVERWRITE INTO TABLE” will also work for compressed table. Follow the steps below:

CREATE TABLE tmp_table (data STRING)
   ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';

CREATE TABLE compressed_table (data STRING)
   ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
   STORED AS SEQUENCEFILE;

LOAD DATA LOCAL INPATH '/tmp/file.txt' INTO TABLE tmp_table;

INSERT OVERWRITE TABLE compressed_table SELECT * FROM tmp_table; 

-- then drop the tmp table
DROP TABLE tmp_table;

Another way ( faster ) is to use the external table which will save the time to load data into the tmp_table, but you will need to put the file into HDFS first:

CREATE EXTERNAL TABLE IF NOT EXISTS tmp_table (data STRING)
   ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
   LOCATION 'hdfs://hadoop-namenode:8020/directory_name';

CREATE TABLE compressed_table (data STRING)
   ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
   STORED AS SEQUENCEFILE;

INSERT OVERWRITE TABLE compressed_table SELECT * FROM tmp_table; 

-- then drop the tmp table
DROP TABLE tmp_table;

Hope the new Hive release will make our life easier by fixing these limitations.

Hive query is very close to MySQL in many ways, and it has lots of other features that MySQL does not have. I have been playing with it for quite a few months and I quite enjoy using it so far.

There are also lots of problems that I have encountered and here is one of them.

Because Hive is build from JAVA, which is a strong typed language, when doing query on Hive tables, you should be careful about column types and the value you are comparing with, for example:

CREATE TABLE hive_table (hive_string_column string, hive_int_column int) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';

-- INSERT SOME DATA HERE

SELECT * FROM hive_table WHERE hive_string_column != 0;
SELECT * FROM hive_table WHERE hive_int_column != '';

Both of the queries above will return no results regardless, simply because the types don’t match. The correct query should be:

SELECT * FROM hive_table WHERE hive_string_column != '' OR hive_string_column != '0';
SELECT * FROM hive_table WHERE hive_int_column != 0;

It is not an easy pickup initially, but now you know!

I have Ubuntu 11.04 installed to serve my own server to host my blogs and subversion repository. I had the history of losing my whole server’s data when my VPS host company did some terrible things. So I decide to setup a cron job to backup my SVN repository on daily basis by simply using svnadmin.

I used command:

crontab -e

and then enter:

5 8 * * *          /bin/bash /home/user1/Dropbox/backups/scripts/svndump.sh > /tmp/cron.log

to the end of file and save it. However, the cron job never runs and the log file was never created. I tried on root user as well and also without any luck.

In the end I have found out this is actually a bug on the Ubuntu system ( not sure other distros though ) from Ubuntu Help:

When adding a new entry to a blank crontab, forgetting to add a newline
 at the end is a common source for the job not running. If the last line
 in the crontab does not end with a newline, no errors will be reported
 at edit or runtime, but that line will never run. 

See man crontab for more information.
This has already been suggested as a bug.

So what I did was to add a few new lines to the end of “crontab -e” output and then restart the cron server:

service cron restart

Now my cron job works perfectly. I have my backup runs everyday and it uploads to dropbox automatically, I don’t need to worry about lose data anymore.

I use GROUP_CONCAT function in MySQL quite often, but today I encountered a situation where by I need to order the data returned from the GROUP_CONCAT function call. Simply having “ORDER BY” clause at the end of the query does not help as it orders the final output rows, not the data inside the GROUP.

I have found the way to do it, very simple:

SELECT `column`, GROUP_CONCAT(DISTINCT choice_id ORDER BY bit_index DESC) AS choice_ids FROM table_bit_index GROUP BY `column`

The stuff inside the GROUP_CONCAT does the trick “DISTINCT choice_id ORDER BY bit_index DESC“.

Now I know..