I’ve been an IT professional for 6 year now and I’ve grown to love the scripting/programming language PHP. A lot of you use languages like these for easy and quick web development.

But we often forget that C/C++ is the foundation of these modern creations. Although I have had some C++ training in school but got alienated from it over the years. It was actually the love for PHP that got me back into C/C++. When compiling PHP and its set of extension I got into contact again with this fabulous language.

But this blog post isn’t entirely about C and C++. My goal is put gcc (the compiler par exelence) in the spotlight.

Gcc 101

Let’s start with the basic usage of gcc

The binaries

By default we use the gcc binary which is suited for compiling regular C programs. When programming in C++ it’s better to use the g++ binary.

A simple C++ script

It doesn’t get simpler than this: outputting Hello world!.

#include <iostream>
 
int main()
{
    std::cout << "Hello world!" << std::endl;
    return 0;
}

In PHP a simple echo would do, but in C++, you’ll need to initialize your tools:

• Include you Input/output stream library called iostream.
• Define a main function
• Call the cout language construct that resides in the std namespace
• Implement a proper return value for the main function

Compiling it into a binary

The easiest way to compile this script is by calling the g++ binary and by passing the name of the source file:

g++ test.cc

The g++ binary processes the test.cc source file and compiles it into machine code that is stored in a binary. Since we didn’t mention the name of the binary, g++ gives it the default name a.out

Running the binary

Calling the binary is simple, just execute it as follows:

./a.out

And you’ll get the expected output:

Hello world!

Gcc options

I’m just going to cover some basic gcc options. More information on gcc options can be found here.

Output

As mentioned, gcc defaults the name of its output binary to a.out. To control the name of the binary, use the -o option.

An example:

g++ test.cc -o test

This generates a binary based on test.cc called test.

Warnings

The -W option allows you to define the warning levels. Most developers set this value to -Wall. Compiling a program with these parameters goes as follows:

g++ -Wall test.cc

Binaries vs objects

By default gcc generates a single binary containing the entire program. When using a fair amount of source files, a minor modification can result in slow compile times. To avoid this, you can compile every C/C++ source file into an object in machine code. This happens via the -c option.

g++ test.cc -c

This command creates a file called test.o containing our simple program. This file is not executable, but can be linked into a binary when needed.

The upside is that you only have to recompile the source files in which modifications where made. Afterwards you can link the objects and create a binary.

g++ test.o -o test

This will start getting useful when you have a large number of source files.

Libraries

In most cases you won’t program all logic yourself unless you like to reinvent the wheel. You’ll probably include some third party code which is stored in the library folder of your operating system.

Let’s take the following example script I’ve found on the blog of Todd Papaioannou. It describes how to use the libcurl library to perform HTTP calls in a simple C++ program:

#include <string>  
#include <iostream>  
#include "curl/curl.h"  
 
using namespace std;  
 
// Write any errors in here  
static char errorBuffer[CURL_ERROR_SIZE];  
 
// Write all expected data in here  
static string buffer;  
 
// This is the writer call back function used by curl  
static int writer(char *data, size_t size, size_t nmemb,  
                  std::string *buffer)  
{  
  // What we will return  
  int result = 0;  
 
  // Is there anything in the buffer?  
  if (buffer != NULL)  
  {  
    // Append the data to the buffer  
    buffer->append(data, size * nmemb);  
 
    // How much did we write?  
    result = size * nmemb;  
  }  
 
  return result;  
}  
 
// You know what this does..  
void usage()  
{  
  cout << "curltest: \n" << endl;  
  cout << "  Usage:  curltest url\n" << endl;  
}   
 
/* 
 * The old favorite 
 */  
int main(int argc, char* argv[])  
{  
  if (argc > 1)  
  {  
    string url(argv[1]);  
 
    cout << "Retrieving " << url << endl;  
 
    // Our curl objects  
    CURL *curl;  
    CURLcode result;  
 
    // Create our curl handle  
    curl = curl_easy_init();  
 
    if (curl)  
    {  
      // Now set up all of the curl options  
      curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errorBuffer);  
      curl_easy_setopt(curl, CURLOPT_URL, argv[1]);  
      curl_easy_setopt(curl, CURLOPT_HEADER, 0);  
      curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);  
      curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writer);  
      curl_easy_setopt(curl, CURLOPT_WRITEDATA, &buffer);  
 
      // Attempt to retrieve the remote page  
      result = curl_easy_perform(curl);  
 
      // Always cleanup  
      curl_easy_cleanup(curl);  
 
      // Did we succeed?  
      if (result == CURLE_OK)  
      {  
        cout << buffer << "\n";  
        return 0;  
      }  
      else  
      {  
        cout << "Error: [" << result << "] - " << errorBuffer;  
        return -1;  
      }  
    }  
  }  
}

As you might have noticed, Todd calls the curl.h header file located in the curl directory. This directory is located somewhere on your system. The exact directory depends on the version and distribution of your OS, but in my case the header file is located in /usr/include.

This is just the header, the actual libcurl library is stored in /usr/lib/ and is called libcurl.so. The upside about using this .so file is that the compiler doesn’t compile the entire library in our small program. A shared object (so) shares its content at runtime and can be reused by several other programs.

This architecture results in smaller binaries, faster compile times and more flexibility when having to modify a library. Compiling it requires the -l option. Libraries are named libname.so where name is replaced by the actual name of the library.

Gcc only needs to know the name of the library and the following command does so:

g++ -Wall test.cc -lcurl -o curltest

Notice the fact that we don’t mention libcurl.so, but just curl. The binary we generate is called curltest and takes a command line argument that represents the URL of the website to which the HTTP request is sent.

Static libraries

In some cases your library isn’t stored in an so file or you just want all logic included in you binary. In that case you’ll want to compile your libraries in a static way.

When there’s no shared object to link dynamically, your library is called libname.a. Compiling it via the -l option will statically compile it into your program.

When specifying the -static option, you can force static compilation.

Make makes it easy

Most of you have probably used the make command on Linux. We take this for granted and often forget the complexity behind the compilation tasks this command executes.

What does make do?

Make is actually a build tool that cooperates quite well with gcc. The make command consists of easy tasks, that as a whole can represent a complex build. All input is retrieved from a file called Makefile which includes all actions to be performed.

The build actions are grouped in targets. By default the target that is first defined will be called when executing make. All other targets can be invoked by adding the target name as an argument. The make install is a common example of this.

The Makefile

This is an example of a Makefile. Before any of the targets are called, a set of variables can be set. These variables are used to setup gcc as gcc cycles through all source files and compiles them.

In this simple example I’ve set 2 variables:

• CC=g++: specify that g++ is our compiler binary
• CFLAGS=-Wall: specify that the -Wall warning option should be passed to gcc

After the initial compilation phase when all source files are compiled into objects, the targets can be executed. Our Makefile contains 3 targets:

• Main: the first target that is called by default. It links compiled objects into a single binary
• Clean: deletes the main binary and all object files
• Run: executes the binary

CC=g++
CFLAGS=-Wall
main: main.o hello.o
        $(CC) main.o hello.o  -o main
clean:
        rm -f main main.o hello.o
run: main
        ./main

Each of these targets has one or more dependencies. When the files main.o and hello.o don’t exist, the main target will not be executed and an error will occur. Similar patterns apply to the other targets.

We can now call:

• make
• make clean
• make run

References

This is merely an illustration of what gcc and make can do. I didn’t even cover 1% of the features. For a complete reference, please visit the following pages:

• GCC: http://gcc.gnu.org/onlinedocs/gcc-4.4.3/gcc/
• Make: http://www.gnu.org/software/make/manual/make.html

The concept for this talk originates from my (give or take) 6 years experience within the Combell support team. Over the years I’ve seen tons of crazy cases which have thought me a lot about PHP.

For those who don’t know us, we’re the largest independent hosting company in Belgium and we have positioned ourselves in the quality segment. We have a lot of different types of customers, ranging from small one man businesses to large enterprises. Each of these have their own expectations, but also a different way in which they deal with hosting and PHP.

The funny thing is that we have the same goals as regular developers:

• It has to work
• It has to work fast
• It has to scale
• it has to be secure

But we use different instruments to achieve our goal:

• We don’t write code
• We sometimes interprete code
• We install PHP
• We tweak runtime configurations
• We analyze logs & error messages
• We migrate applications
• We perform monitoring & profiling
• We perform disastery recovery
• We deal with hacked sites & abuse
• We offer customers general best practices

Come see me in Amsterdam June 11th. I’ll be speaking at 13h30 in track 3. Check out the schedule for more info.

Thanks to Lorna and the DPC crew for this lovely opportunity.

The enthousiams for me is even greater because for the first time I assisted in organizing a conference. In fact it’s the first time this conference is organized.

Organizing it

Who’s who

After the merge between PHPBelgium and phpGG it was made very clear that our goal was to organize a real conference. We all had our experiences with user group meetings, but this was a different thing.

I have always felt like “the new guy” in the group and I feel fortunate to be surrounded by a group of very experienced people.

• There are community icons like Michelangelo van Dam(aka dragonbe) and Stefan Koopmanschap (aka skoop).
• There are expert open source contributors like Paul Borgermans and Patrick Allaert
• There are the incredibly dedicated and talented Dutch co-members Rein Velt, Kana Yeh and Gerrit te Sligte (aka Remi)
• And of course there’s my partner in (PHP) crime Felix De Vliegher

I know I put in a lot of work and energy, but it’s nothing compared to what Mike and Stefan put in. They are community animals, so this stuff comes natural to them, but nevertheless: thank you for doing this.

The practical stuff

There’s so much stuff that you have to take into account:

• The venue
• The sponsors
• The printwork
• The announcements & communication
• The speakers
• The call for papers
• The hotel
• The income and expenses
• The pricing
• The website
• The attendees
• …

It took us a while to figure all of it out, but (as it seems) we didn’t do that bad of a job. I admit, some things were done absolutely last minute, but hey: it got done. I’m sure that due to the experience we now have, the phpbnl11 conference will be even better.

Pre-conference

Friday preparations

After lots of Skype meetings, group mails and discussions, it was finally time to meet up and make it happen.

I ordered the goodie bags, the sponsors brought all the goodies and Mike made sure there were prints to attach to the bags. Rein took care of the lanyards whereas Remi had the badges covered. You should have seen it: a real chain gang.

Picking the speakers up

When you’re organizing an international conference, you need international speakers and those people need to be picked up from airports and railway stations. There was even some carpooling going on.

My task was picking up Fabien Potencier which I had already picked up some time ago during the Symfony Bug Hunt Day. Apparently this proved to be a challenge.

I won’t bother you with the complete story, but I managed to show up in at the Central Station in Antwerp at 3 occasions without finding Fabien. Apparently there was a communication problem and the time of arrival proved to be the time of departure. During my 3 visits to the station there was one little gap I couldn’t cover due to transportation and parking and that was the exact moment Fabien did arrive.

Luckily Fabien showed up safely the day after and had to take a taxi from the station to the hotel. Although I was a little annoyed with the situation, I was actually happy to see Fabien show up. If you’re reading this, Fabien: don’t worry, I’m perfectly cool with the situation and will definitely pick you up again when there’s another PHPBenelux event.

Speakers dinner

The speakers dinner is always good fun and this one proved to be no different. We decided to eat at Da Giovanni , a colourful Italian restaurant near De Groenplaats and the cathedral.

The funny thing about this place (do check their website to see for yourself) is that everything in the place has a white/red squared design. The waiters have an attitude and know how to use. In fact, almost none of the waiters are Italians (most of them are Morrocans or so), but they all pretend to be full blown Italians. Lots of prego’s and gracie’s.

Good food, good company and a feeling that I’m slowly being part of the PHP community clique.

Drinks

After a lovely dinner, we went to De Vagant, apparently an established jenever bar.

It was rather quiet out there and except for the speakers and crew only a couple of attendees. We did have some fun and had a couple of drinks. But everyone agreed that going to the hotel was a good idea. We had to be on top of our game for the conference and there was still some last minute details that needed to be figured out.

On the way back to the hotel, I offered Kore Nordmann a drive. I only knew him as a speaker and PHP expert, but I had the opportunity to get to know him as a person too. Cool dude, I must say!

Conference day

T-minus one hour

I got up at 7 am (which was quite an effort), enjoyed a nice breakfast and joined the ever so charming Cal Evans. This guy hopped into a plane from the US to present the closing keynote at our conference and had to leave the day after. Talk about dedication. I knew my fellow PHPBenelux co-members had a high opinion of him, but now I can confirm this myself. Thanks for making it to Antwerpen, Cal !

After a quick breakfast, we had to bring in all the goodie bags. In the meanwhile Stefan was already assisting the sponsors with their boot setups. Things looked very promising at that moment.

Only one thing caught us (or should I say the attendees) off guard: the night before the conference it had snowed and a lot of roads weren’t cleared yet. This forced us to kick off with a 15 minute delay.

Kickoff

Opening keynote

Derick Rethans was our opening keynote speakers. This guy needs NO introduction. I’ve seen a lot of his talks, even a couple of keynotes, but for this occasion he had prepared an entirely new keynote called The PHP universe. Really good stuff about how the so called PHP ecosystem.

Track 1

After the opening keynote, the schedule was split up into 2 tracks. I decided to stay in track 1 for a while where I assisted with the room management.

Fabien Potencier was first up with a brilliant talk about dependency injection in PHP 5.2 and 5.3. The new features in PHP 5.3 he used to illustrate his point were excellent. I did not know that lambda functions and closures could be so convenient when using dependency injection and avoiding overhead.

David Zülke was next with a talk about the state of SOAP. I have been using a lot of SOAP webservices the last couple of years, so I was very interested. This talk confirmed lots of things I did know about SOAP and taught me a couple of new things as well. Besides being a real PHP expert and a cool guy, David’s English is exceptionally good.

Track two

I would have loved to see Juliette Reinders Folmer’s talk about regular expressions, but I felt I had to support Patrick Allaert and Davide Mendolia. Their talk was about APM and Pinba: two open source projects that deal with monitoring. It’s not about monitoring the server itself, but more about monitoring it from a PHP point of view.

Very interesting talk, but unfortunately, the message dissolved in the chaos that emerged from the technical failures. I really felt bad for you guys, but I hope you’ll learn from your mistakes.

And Juliette: I promise I will attend your regex talk in London next month. I’ll have a front row seat!

After lunch (which was great actually, much better than the luches we usually get at conferences) I decided to check out Kore Nordmann’s talk about CouchDb and PHPillow. Really advanced shizzle! I already saw a similar talk he gave at PHPVikinger, so I knew what to expect. For people who aren’t familiar with non-relational databases, this seemed like rocket science. I will definitely try some PHPillow myself.

Back to track one

Next up was Lorna Mitchell. Lorna is the community liaison for Ibuildings and she has built up a rocksolid reputation in a very short time. Here talk was great and dealt with the so called Joel test. Unfortunately, I had to leave early because Fabien needed a ride back to the station.

I’m really excited with the fact that Lorna will be in charge of The Dutch PHP Conference this year. It has always been a top conference and I’m sure this year will be a great edition as well. Good luck!

Nearing the end

We were nearing the end of the conference and I learned that Eric Ritchie couldn’t make it to Antwerpen. He got stuck somewhere, but luckily technology saved us. He managed to give his talk remotely and the audio/video was broadcasted into the conference room. I even heared that there was a microphone in the room that allowed the crowd to give real time feedback.

The closing keynote was the much anticipated Open teams talk by Cal Evans. Cal illustrated in his signature style how corporate development teams can benefit by adopting open source development principals.

I must say: I like his ideas and I would love to see this in action, but I thing a lot of managers will be frightened by some of his propositions. Nevertheless, it can happen, we only need a couple of courageous companies to lead the way.

The end

Conference social

As Cal stated: he was the one thing standing between the conference attendees and an open bar. After his keynote there was still the obligatory conference closing statement which Stefan did very well. Everyone involved received a nice thank you and gifts were handed out to the speakers.

After those formalities, people rushed to the open bar (thank you Ibuildings for sponsoring this).

I had to leave quite early, but I heared the bowling event was a great success.

Cool people I met

• The entire PHPBenelux crew
• Cal Evans
• Kore Nordmann
• David Zülke
• Derick Rethans
• Juliette Reinders Folmer
• Katrien de Graeve at Microsoft
• Rick Buitenman and Ronald Kleverlaan at WEBclusive
• Tom Claus and Stijn Janssen at Inventis
• Christian Durel at Zend
• Nick Belhomme
• Andries Seutens
• Paul Delbar
• Ivo Jansch, Lorna Mitchell, Felix De Vliegher and Harrie Verveer at IBuildings
• Jachim Coudenys and Jochen Vandendriessche at Guideline
• Filip Forrez and Jeroen Serpieters at Ausy
• Davide Mendolia
• Jeremy Coates
• And lots of other people I forgot to mention

See you next year

See you next year for phpbnl11 !

MySQL followed this trend and introduced character set support in version 4.1. Importing data from older versions is easy and SET NAMES ‘utf8′ is quite useful to ensure special characters are correctly encoded in the new databases.

But I learned first hand that the SET NAMES ‘utf8 trick is not a cure for every disease.

The story

Moving up is easy, but moving down is a real nightmare if you don’t take the necessary precautions. I had a case where a database had to be moved from a MySQL 3.23 server to a MySQL 5.0 server. We decided to store all data in UTF-8 encoding which worked out well.

We also knew that the upgrade was a temporary measure: the 5.0 was installed on a high performance dedicated box to handle a peak. Once we were passed the peak, we decided to move back to the old server, which was a shared box. I was always under the assumption that the old server had a MySQL 4.1, but little did I know that I was dealing with an old school 3.23.

The problem

Unaware of any risk, I quickly took a mysqldump and imported the data on the old server. Apparently I had saved UTF-8 data in a database server that was only ISO aware. The consequences weren’t that clear in the beginning since Dutch & English characters are usually the basic ISO characters.

Once I noticed that French characters were suddenly malformatted, I knew something was wrong. I quickly recreated the hosting on a 4.1 server and imported the data again. When reloading the page I was stunned: the old UTF-8 data was correctly displayed, but apparently all the new data since the migration from the dedicated server was still ISO.

The solution

Attempts that failed

The conclusion was that I had tables with records partially in UTF-8 & partially in ISO. I figured this was easy to solve by writing a script that reads every line from the dump and converts it to UTF-8. By doing this, I would be sure all characters were the same.

Again I was faced with a big surprise: converting ISO data to UTF-8 works fine via ‘utf8_encode’ in PHP , but UTF-8 stored in an ISO database cannot be converted to valid UTF-8 via this function.

Later I figured iconv could to the trick for me, but again the results where negative.

The solution that actually worked

Desperate for a solution, I continued browsing http://www.php.net/manual/en/ and stumbled upon a user comment describing a function to detect UTF-8 strings. That function, written by a guy named Anton Vakulchik, contained a fancy regular expression that does all the heavy lifting.

So I created a script containing the regular expression, started reading every line from my dumpfile and converted the string to UTF-8 where the preg_match returned false. To my surprise, it worked perfectly and I managed to output the data line by line in valid UTF-8

The script

The script below reads from stdin line by line and processes it. It detects UTF-8 data and converts lines to UTF-8 when necessary. This CLI script outputs every line to the stdout.

<?php
function detectUTF8($string)
{
	return preg_match('%(?:
	[\xC2-\xDF][\x80-\xBF]        # non-overlong 2-byte
	|\xE0[\xA0-\xBF][\x80-\xBF]               # excluding overlongs
	|[\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}      # straight 3-byte
	|\xED[\x80-\x9F][\x80-\xBF]               # excluding surrogates
	|\xF0[\x90-\xBF][\x80-\xBF]{2}    # planes 1-3
	|[\xF1-\xF3][\x80-\xBF]{3}                  # planes 4-15
	|\xF4[\x80-\x8F][\x80-\xBF]{2}    # plane 16
	)+%xs', $string);
}
function readStdInArray()
{
	$stdin = fopen('php://stdin', 'r');
	$out = array();
	if(is_resource($stdin)){
		while(!feof($stdin)){
			$in = trim(fgets($stdin));
			if(strlen($in)>0){
				$out[]= $in;
			}
		}
	}
	fclose($stdin);
	return $out;
}
$std = readStdInArray();
foreach($std as $line){
	if(detectUTF8($line)){
		echo $line.PHP_EOL;
	} else {
		echo utf8_encode($line).PHP_EOL;
	}
}

Using the script

The combination of stdin and stdout allow the use of pipes in Linux which makes it really easy to convert data between 2 MySQL servers.

There are several ways to use the script. Every method consists of 3 steps:

1. Dump mixed data from the old MySQL server
2. Convert mixed data into UTF-8 data
3. Import UTF-8 data into the new MySQL server

You can choose to store to results in a dumpfile or immediately transfer it from source to destination server with the conversion script in between. Here are a couple of examples:

Direct transfer between 2 database servers

mysqldump -hOldHost -uUsername -pPassword -Q -A --skip-extended-insert | php -q utf8.php | mysql -hNewHost -uUsername -pPassword

Storing the dumps in dumfiles

mysqldump -hOldHost -uUsername -pPassword -Q -A --skip-extended-insert -pPassword > mixeddump.sql

cat mixeddump.sql | php -q utf8.php > utf8dump.sql

cat utf8dump.sql | mysql -hNewHost -uUsername -pPassword

Script restriction

The script is quite powerful, but has one big restriction: it can only determine the charset of a line. So every line must be encoded in a single character set. To enforce this, I had to make sure that every database record was stored on 1 single line. The mysqldump –skip-extended-insert parameter helped me to achieve this. So please do use it!

Of course, I could have performed a UTF-8 check on every character, but that wouldn’t have been so resource friendly when using 100+ MB dumpfiles.

Check out the mysqldump manual for more information on mysqldump command line parameters.

Conclusion

What I’ve learned is that working with character sets can get tricky. I will definitely by more cautious in the future, but a least I have a conversion script that can take care of hairy situations.

Thank you Anton Vakulchik for writing the kick-ass regular expressions

To be perfectly honest, we were discussing it for a while, but we didn’t reach the planning stage. That was until the kind people at Intracto decided to take the lead. I knew they were pretty fond of Symfony, but now they decided it was time to give back to the community by sponsoring and hosting the event.

The event takes place Saturday November 14th at Frame21 in Herentals. More details can be found on the PHPBenelux website. You can still register for this event and we kindly invite you to do so by filling up the following form.

This blog post will assist you in preparing for the actual event. You shouldn’t be a rocket scientist to participate. A good knowledge of PHP and an open mind will get you a long way.

About Symfony BugHuntDay

A BugHuntDay is a community event where people gather to find/fix bugs in open source software. As I already mentioned, we picked the Symfony framework which is a well known PHP framework which delivers a full stack MVC solution. Our goal is to find/fix as many bugs as possible for the upcoming 1.3 release.

Finding bugs is quite easy, the Trac issue tracker that Symfony uses has a nice overview of all tickets. But finding a bug that you are able to fix is a different story all together.

Getting to know Symfony

The framework

You can get to know the Symfony framework by downloading and installing the Symfony sandbox. The sandbox offers a pre-packaged Symfony project you can play and experiment with. It includes the application default document structure and the complete Symfony framework library.

There’s also plenty of documentation available. Go to the Symfony documentation pages to read all about it. If you have difficulties installing Symfony or the sandbox, you should consult the Symfony installation guide.

The source

Once you’ve gained some basic experience, it’s time to take a look at the source code. Since we are working on the upcoming 1.3 release, you should check out the 1.3 Subversion branch. You can also view this branch via the Trac panel.

Coaching

Once you know your way around the framework, it’s time to find yourself a nice bug to fix. That will not be so easy. You are free to browse the tickets, but I guess some coaching will be required.

The good thing about PHPBenelux is that we have lots of contacts in the PHP community. We managed to invite some people who are important to the Symfony project. They will coordinate the event and provide proper coaching. We have confirmed the following Symfony team members:

• Stefan Koopmanschap is the community manager for the Symfony project. He’s also one of our board members, so it was quite easy to confirm him.
• Fabien Potencier is the project leader for Symfony. It’s a real honor to have him at the event.
• Fabian Lange is the release manager for the 1.2 release. His presence is also greatly appreciated.

Not only will they help us with the issues we encounter, but they’ll make a selection of bugs that are quite manageable to fix. It’s also a great opportunity for them to have a group of people who will contribute to the project. If you have difficulties finding the right bug to fix, talk to the coaches, they will guide you.

For those of you who can’t make it to Herentals: you can also participate over the internet. The entire event will also be hosted on the internet via IRC. It will be the main communication channel to get in touch with the coaches and the onsite contributors.

Fixing a bug

Once you have found a bug that you can fix, you still need to take some things into account before you start coding away. It’s very important to read the wiki article on how to contribute.

The Submitting patches chapter is an essential part that describes:
- Coding standards
- Patch styles
- Commit procedure
- procedure

When it comes to preparing, this is a crucial step that cannot be skipped. If you really want to commit a patch It will have to be according the standards.

Testing

When can you really guarantee a bug is fixed? And can you guarantee that your patch hasn’t triggered another bug?

Unit tests are the answer to that question. The assumption that your patch is absolutely flawless is not enough. It’s very easy to trip over a proverbial wire and trigger one or more additional bugs. Unit tests allow you to test your patches via a set of fixtures.

An initial testcase should make your testcase fail and should reproduce the actual bug. This is a solid way of proving that the bug is actually a bug and not a feature. Again, assumptions aren’t enough.

After fixing the bug, the testcase will succeed and this should be a proof that the patch is a valid bugfix. If the patch makes other unit tests fail, you’ve obviously did something horribly wrong and you should review the patch before submitting it.

It’s obvious that unit tests are saveguards that protect the integrity of the project. There are many tools to perform unit tests. The people at Symfony have created there own testing framework which is called The Lime Testing Framework. Again, the Trac wiki provides the necessary information on how to use this testing framework.

Hope to see you there

Don’t be afraid to participate, even if you aren’t that experienced. It will be a great opportunity to learn a lot about the framwork, about bugfixing and about PHP in general.

You will meet lots of interesting people and if your not part of the PHP community yet, this is your chance to take the first step.

See you there !
Thijs

Hello readers

It’s been a while: I’ve been busy at work, going on vacation and been lazy when at home. It’s more than about time to get into my blogging routine again.

This one is actually a short one on Tarball archives. You know, the (compressed) archiving you use to pack your files with. For people unaware of this technology: you can compare it to ZIP and RAR archives but with a little more punch.

Basic usage

Everyone who’s used to working with Linux or other Unix-like systems knows how to create and extract a Tarball archive. But for those who are new to it, here’s an example

Creating an archive

The simplest way to create a Tarball archive is with the following command:

tar cvf myTarArchive.tar folderToArchive

• Tar: the command used to archive a specified folder
• Options:
  • C: create an archive
  • V: verbose
  • F: filename to use
• myTarArchive.tar: the name of the new tar archive
• folderToArchive: the folder that will be archived and packed into the myTarArchive.rar file

In most cases you will want your Tarball archive to be compressed. The tar command allows GZIP compression by adding the Z option:

tar cvzf myTarArchive.tar.gz folderToArchive

• Tar: the command used to archive a specified folder
• Options:
  • C: create an archive
  • V: verbose
  • Z: compress using GZIP compression algorithm
  • F: filename to use
• myTarArchive.tar.gz: the name of the new tar archive (with the GZ suffix)
• folderToArchive: the folder that will be archived and packed into the myTarArchive.rar file

Extracting an archive

The exact purpose of this article is facilitating extraction and before we can get into that, I need to show you how basic Tarball extraction works:

tar xvf myTarArchive.tar

• Tar: the command used to unpack a specified tar archive
• Options:
  • X: extract an archive
  • V: verbose
  • F: filename to use
• myTarArchive.tar: the name of the tar archive to extract

If your Tarball archive was compressed, you will need to specify the Z option as well:

tar xvzf myTarArchive.tar.gz

• Tar: the command used to unpack a specified tar archive
• Options:
  • X: extract an archive
  • V: verbose
  • Z: apply GZIP decompression
  • F: filename to use
• myTarArchive.tar.gz: the name of the tar archive to extract (with GZ suffix)

The tricks

Without further ado, I will immediately continue with the actual tricks. Let’s say you have an automatic backup schedule of some crucial folders on your server. A cronjob launches a tar command that archives a specified folder every night.

Extracting a single branch

The reason one backups his files, is to eventually recover lost/corrupt data in case of emergency situations. If that archive happens to be a couple of giga’s, your server will require a lot of computing power to decompress and unarchive the data you want. In a number of cases you don’t want to recover your entire archive since you only a single subfolder.

In that case it would be nice to only extract a single branch of your entire folder tree. For all you GUI users out there, this is peanuts, you just drag and drop. On the command line, it requires an extra argument.

Take for example this folder tree:

• folder_1
  • subfolder_1
    • subfolder_1_1
    • subfolder 1_2
  • subfolder_2
• folder 2

Let’s say you only want to extract ‘’subfolder_1_2″. It would be overkill to extract the entire archive, wasting disk, resources and most important: your time. To extract this single branch, you can issue the following statement

tar xvzf myTarArchive.tar.gz folder_1/subfolder_1/subfolder_1_2

What will happen is that in the current working directory a number a folder tree will be created containing only the files of ‘’subfolder_1_2″. Of course it will reside in the hierarchy of the folder tree.

Extracting to a different folder

If can be quite annoying if you want to extract an archive containing files that already exist in your current working directory. These files will be overwritten and you don’t always want to create dummy folders where you can extract you archive. Sometimes you just want extract a file, a folder or an entire branch to a specified folder. The -C option does all that:

tar -C /path/to/selected/folder -xvzf myTarArchive.tar.gz

So this on will extract the myTarArchive.tar.gz archive in the path specified after the -C option.

Combining both tricks

Needless to say you can combine both tricks:

tar -C /path/to/selected/folder -xvzf myTarArchive.tar.gz folder1/subfolder_1/subfolder_1_2

The trip to Amsterdam started off in Gent where I traveled along with Felix De Vliegher & Steve Roelens. Comfortably cruising along the Belgian & Dutch motorways, we where looking for some good times and some expert opinions on common PHP topics. In retrospect, we got both in spades!

Welcome to Amsterdam

We all booked rooms in Novotel nearby the RAI venue and after the obligatory checkin, we quickly decided to head off to town. Everyone knows that Amsterdam is a pretty wild city, but add the fact that Holland was playing an UEFA European Cup game versus France that weekend and you have yourself a full blown circus.

The Dutch are pretty dedicated, emotional and loudmouth fans and the entire city was covered in orange ornaments. They where excited about their weekend, but so where we. After some late night shopping and a quick fastfood stop, we quickly turned to the number one tourist attraction in Amsterdam: the famous red light disctrict known as De Walletjes.

As strange as it might sound, but there are families strolling around looking at the entire phenomenon: girls luring guys in, pimps on the lookout and drug dealers hustling. Your typical family night out, right? Very exciting, but extremely fake I must add.

Tutorial Day

After a final fastfood stop, we headed back to the hotel and hit the hay. Make no mistake: Amsterdam is a nice city the visit, but the real reason for our stay was the PHP conference. The first day was tutorial day and I had signed up for the PHPUnit tutorial by Sebastian Bergmann. The good thing about this tutorial is that you’re not dealing with some speaker or coach, but the guys who actually developed the software. It doesn’t get much better than that.

Really interesting stuff and this was only the beginning. There where plenty of opportunities for networking as well and I actually met a lot of customers, so I obviously had to represent Combell.

Dutch PHP Conference 2008 (by Felix De Vliegher)

Good Stuff

If knew right from the start that this was a good conference, the organisation was superb:

• Great topics
• World class speakers
• Good venue
• Decent catering
• Kick-ass afterparty (courtesy of the Dutch national football team)

Afterparty

Let me get into more details about that friday night afterparty: all attendants were kindly invited to the Werck bar at the Prinsengracht. There was a happy our (thanks to IBuildings) and the atmosphere was ecstatic. The bar also had a dining area which was exclusively reserved for IBuildings staff. Since IBuildings has lots of employees and even a UK branch, there were lots of people.

So we had a packed bar, lots of drinks and tons of football fans. Need I say more? Ow yeah, I do: Holland crushed France that night and it was the most lovely football experience ever. Seconds after the first Dutch goal, it felt like a bomb was dropped on Amsterdam: in the bar everyone went nuts, but outside you could here the roars from an entire city. Imagine asking a million people to yell at the same time and you’d have the same effect.

Poor Fabien Potencier who supported the French team. The game didn’t just leave an impression on him or me, because Europeans knew what they’re getting themselves into. But the Americans, they were absolutely stunned ! But at least now they know what Hup Holland Hup means. Some American speakers even made remarks on that during their talks

Adieu les blues (by Derick Rethans)

Conference Day

Intro

After tutorial day and the parties it was time for some real deal PHP shizzle. Ivo Jansch was the guy that put us on track with a short introduction.

Zeev Suraski

After the intro it was time for a keynote by Zeev Suraski, the co-founder of Zend called State Of PHP. He talked about the state and evolution of PHP. It is definitely a cool thing to see how PHP evolved from PHP/FI and a small university project into to enterprise product it is nowadays.

Marco Tabini

Two keynotes in a row: we were in for a real treat! Next up was Marco Tabini, the co-founder of PHP|architect. He brought us a presentation called Software And The Taste Of Mayo. The title seems quite mysterious and vague, but in fact he explains that good software is a mixture of the right ingredients, just like mayonaise.

Gaylord Aulke

I also enjoyed the talk by Gaylord Aulke called An Infrastructure For Team Development which covers best practices when working with multiple developers on one project. Gaylord went over the common tools professional PHP developers use these days and also specified how to use them when working in teams. Good stuff there!

Lorna Mitchell

Content wise, this was the most interesting talk on the bill. Lorna Mitchell’s talk called Deployment With Subversion dealt with best practices to make deployment a little easier. It even involved a little bunny called Nabaztag that goes nuts when Subversion commits fail. Simply awesome!

Ivo Jansch

Off course I had to see Ivo in action. His talk covered practical items, but was actually quite evangelistic. It was called Enterprise PHP Development and shows how PHP is enterprise ready as a technology. He wrote a book about it and even the fiercest non-believers would crawl up in some deep dark hole after seeing this.

I strongly believe that PHP can be used in an enterprise environment and I have talked about it on some occasions myself. This talk inspired me to spread the word.

Terry Chay

Needless to say that a Terry Chay talk leaves an impression. He knows what he’s talking about, he has proven to be an expert in his domain and he’s not afraid to kick the oposition in the nuts. Did he screw with Rails? Ow yeah he did! Did he curse a zillion times? Fuck yeah he did!

His talk was called ”The Internet Is An Ogre” and was beautifully illustrated with some great slides, animations and movies. Terry, please come back to Europe some time!

DPC09

I hope this brings back some memories for the people who were there. I am convinced that this year’s edition will be as successful and I hope to see you guys there for a little chat.

At that time I was using Zend_Cache_Backend_File as my backend and cache files were created. My goal was to boost the performance by implementing the Zend_Cache_Backend_Memcached backend. Again everything seemed to be working fine because my Memcached logging displayed activity of some sort.

I was stunned to hear that in fact all of those cache files where created, but never read. So here I am optimizing everything, but in reality I was creating extra overhead. So I started my quest and found the solution (sorry for the spoiler).

Page caching

We use Zend_Cache_Frontend_Page as the frontend of our caching mechanism because it is so simple to use:

$cache = Zend_Cache::factory('Page',
    'File',
    $frontendOptions,
    $backendOptions);
 
$cache->start();

The start method just triggers the caching mechanism and doesn’t need any more attention. It just takes care of itself. The output buffering stores the output in the caching backend. Caching keys are used to identify cached records. Via save() and load() we can interact with the caching backend.

The frontend options

In the previous example I didn’t specify the frontend options, but actually they are essential to this blog post. That’s why I dedicate a chapter to it.

$frontendOptions = array(
    'lifetime' => 86400,
    'default_options' => array(
        'cache' => false
        ),
    'regexps' => array(
        '^/(nl|fr|en)(.*)$' => array('cache' => true,
            'cache_with_get_variables' => true,
            'cache_with_session_variables' => true,
            'cache_with_cookie_variables' => true,
            'make_id_with_get_variables' => true,
            'make_id_with_session_variables' => true,
            'make_id_with_cookie_variables' => false
            )
        ),
        'debug_header' => false
    );

The code example above illustrates the extra options that were passed to the frontendOptions array. The cache_with_… option keys are quite important because they make sure that caching is still enabled when cookies or get variables are passed. Sometimes you want to caching to be disabled when there’s dynamic content involved. We decided to continue caching when there are cookies.

The make_id_with_… option keys don’t decide if caching is enabled or disabled, but they decide how dynamic content is stored. The next chapter explains how Zend_Cache_Frontend_Page stores its data.

Behind the scenes

To understand how Zend Framework works with page caching and how I ran into my problem, we should dig into the framework code.

Data storage

As mentioned, the start method takes care of everything and below you’ll find the source code of that method.

public function start($id = false, $doNotDie = false)
{
    $this->_cancel = false;
    $lastMatchingRegexp = null;
    foreach ($this->_specificOptions['regexps'] as $regexp => $conf) {
        if (preg_match("`$regexp`", $_SERVER['REQUEST_URI'])) {
            $lastMatchingRegexp = $regexp;
        }
    }
    $this->_activeOptions = $this->_specificOptions['default_options'];
    if ($lastMatchingRegexp !== null) {
        $conf = $this->_specificOptions['regexps'][$lastMatchingRegexp];
        foreach ($conf as $key=>$value) {
            $this->_activeOptions[$key] = $value;
        }
    }
    if (!($this->_activeOptions['cache'])) {
        return false;
    }
    if (!$id) {
        $id = $this->_makeId();
        if (!$id) {
            return false;
        }
    }
    $array = $this->load($id);
    if ($array !== false) {
        $data = $array['data'];
        $headers = $array['headers'];
        if (!headers_sent()) {
            foreach ($headers as $key=>$headerCouple) {
                $name = $headerCouple[0];
                $value = $headerCouple[1];
                header("$name: $value");
            }
        }
    	if ($this->_specificOptions['debug_header']) {
            echo 'DEBUG HEADER : This is a cached page !';
        }
        echo $data;
        if ($doNotDie) {
            return true;
        }
        die();
    }
    ob_start(array($this, '_flush'));
    ob_implicit_flush(false);
    return false;
}

A couple of things that are worth mentioning about this method:

• Via $this->_makeId(); an identifier is created and the identifier is the storage key
• The $this->load($id); method loads the cached data using the key which was returned by the makeId method
• Via ob_start(array($this, ‘_flush’)); output buffering is started. The _flush() method is a callback method that is triggered when the output buffer is flushed.

public function _flush($data)
{
    if ($this->_cancel) {
        return $data;
    }
    $contentType = null;
    $storedHeaders = array();
    $headersList = headers_list();
    foreach($this->_specificOptions['memorize_headers'] as $key=>$headerName) {
        foreach ($headersList as $headerSent) {
            $tmp = split(':', $headerSent);
            $headerSentName = trim(array_shift($tmp));
            if (strtolower($headerName) == strtolower($headerSentName)) {
                $headerSentValue = trim(implode(':', $tmp));
                $storedHeaders[] = array($headerSentName, $headerSentValue);
            }
        }
    }
    $array = array(
        'data' => $data,
        'headers' => $storedHeaders
    );
    $this->save($array, null, $this->_activeOptions['tags'], $this->_activeOptions['specific_lifetime'], $this->_activeOptions['priority']);
    return $data;
}

The _flush() method we were talking about does most of the storage: when the output buffer is flushed (which happens at the end of the script execution) the save method is called and this one also triggers the saving mechanism of the backend.

The actual problem

After some intense debugging I noticed that the reason why caching went all wrong was because every page hit resulted in a new cache key. Result: 0% cache hits !

Making id’s

To figure out why the previous key isn’t re-used, we should take a look at the _makeId() method that is responsible for creating those id’s:

protected function _makeId()
{
    $tmp = $_SERVER['REQUEST_URI'];
    $array = explode('?', $tmp, 2);
  	$tmp = $array[0];
    foreach (array('Get', 'Post', 'Session', 'Files', 'Cookie') as $arrayName) {
    	$tmp2 = $this->_makePartialId($arrayName, $this->_activeOptions['cache_with_' . strtolower($arrayName) . '_variables'], $this->_activeOptions['make_id_with_' . strtolower($arrayName) . '_variables']);
        if ($tmp2===false) {
            return false;
        }
        $tmp = $tmp . $tmp2;
    }
    return md5($tmp);
}

Did you notice that foreach loop? Yes: that’s the key to everything, because we’re looping through the different superglobals. In this loop, we’re checking if we can create id’s based on cookie values. Off course, if the cookie value would change, a new id is generated. And that is actually what was happening on my system.

What does Google have to do with it?

To summarize: the values of cookies can be used to create cache keys. If cookie values change, our caching key changes as well and we have a cache miss.

So I started looking at my cookies and noticed that there were some cookies which had a different value each time I reloaded the page. After some research, I finally figured out that these cookies were created by my Google Analytics implementation. The reference page shows a definition of the cookies set by Google Analytics. Some of those cookies contain timestamps that change on every page hit.

The solution

The solution is actually simple: disable the make_id_with_cookie_variables key in the frontendOptions and you’re safe!

The final summary

1. Caching is based on identifiers
2. Rules can be defined to allow caching based on cookie values
3. A new cookie value means new caching identifier
4. A new caching identifier means a cache miss
5. When cache misses occur, the ‘miss’ is stored to result in a hit the next time
6. Google Analytics creates cookies that have dynamic values
7. By disabling make_id_with_cookie_variables, we can solve the problem

The item that came up which surprised me the most is the fact that the PHP core developers will choose reverse compatibility above a more than necessary source code cleanup.

This post is actually written in support of the one Chris Ramakers from Inventis wrote yesterday about this very same topic. I surely hope these kind of blog posts will re-open the discussion on this matter and I hope the PHP core developers will reconsider their strategy.

Please read Chris’ blog post and spread the word. Off course it’s the PHP developers who should share their opinion and if the majority feels reverse compatibility is a priority, well then the core guys should stick with it, otherwise a cleanup wouldn’t be a bad idea.

We’re talking about the release of a major version and I think this would be the ideal opportunity to fix some inconsistencies. The argument Derick and Scott used is that they don’t want to piss off PHP developers, but I’m pretty sure some of them will already be pissed off with the fact that register_globals, safe_mode & magic_quotes_gpc will disappear as well.

Think about it …

As some of you might have noticed: my blog series “CLI, the other SAPI” is actually based on this talk. There are still some episodes in the making, but you can have sneak preview of what’s yet to come by checking the slides.

A large portion of the presentation contains information on process control. Some people advised me to make a separate presentation on that topic. I might consider that, but in the mean time this is the result: