One of the most difficult parts of making a search engine, whether it is a small search for a single website or something as large as a web search engine, is tweaking the ability to get as much relevant results from a user's query as possible. The best way to tweak this with Lucene and Zend_Search_Lucene is the correct use of analyzers for both indexing and querying an index.

Analyzers prepare and normalize text to be indexed and are also used to parse queries. For example, unless you want your search engine to be case sensitive you would want your analyzer to have a filter that lowercases everything as it goes into the index; that same analyzer needs to be used against the query to lowercase any terms in order for it to match the terms in the index. That way a search for "query" will match "Query, qUery, query, etc..", and a search for "Query" will match the same.

Now we come to the subject of Stemming. Stemming is the act of getting related words to "stem" to the same word. For example "nationalize, nationalization, nations" should all stem to "nation". Why is this important? say for example you are searching for Ipods, if you aren't making use of stemming, your search engine will match only "ipods", but not the singular "ipod". By making use of stemming you will get more relevant results. Stemming doesn't actually have to return a term that is spelled correctly. For example "ponies" and "pony" both stem to poni. The important thing is that all related words stem to the same thing, spelled right or not.

Stemming algorithms differ from language to language. A English stemmer will NOT work for German or any other language. The most used stemming algorithm for English is the Porter Stemmer, made by Martin Porter. You can read about the Martin Porter Stemmer here. I have ported the Porter Stemmer from the Java Lucene Project for use with Zend Search Lucene. You can download it from my downloads section or directly here. This is pretty much a direct port, so If you can find some optimizations, feel free, and let me have some patches so I can update the package.

And now some super simple example code of the stemming filter in action:

 
<?php
class Search {
 
  private $_index;
 
  public function __construct(){
    $this->_index = Zend_Search_Lucene::open('/path/to/index');
    $analyzer = new Zend_Search_Lucene_Analysis_Analyzer_Common_Text_CaseInsensitive();
    $analyzer->addFilter(new CodeAngel_PorterStemmerFilter());        
    Zend_Search_Lucene_Analysis_Analyzer::setDefault($analyzer);
  }
 
  public function find($query){
    return $this->_index->find($query);
  }
 
  public function addDocument($link, $content){
    $doc = new Zend_Search_Lucene_Document();
    $doc->addField(Zend_Search_Lucene_Field::Keyword('link', $link));
    $doc->addField(Zend_Search_Lucene_Field::Unstored('contents', $content));
    $this->_index->addDocument($doc);
    $this->_index->commit();
  }
}
?>
 

Using this class is as easy as 1-2-3:

 
<?php
  $searcher = new Search();
 
  //index a document
  $searcher->addDocument('some.html', 'Hello, how about some ipods and some nationalization');
 
  //search
  $hits = $searcher->find('Ipod');
?>
 

Codeangel 0.5 has the following new features:

• Trackback and Pingback support (both recieve and sending).
• Feeds are reimplemented, Also notice the syndication symbol under each article links to this comment feed
• Reimplementation of the Search Engine. Articles are searched with better granularity, Comments are now indexed as well. Also I made a Porter Stemmer so one should get more relevant full results. I'll post more about this Porter Stemmer and release it soon.

Probably won't notice much on my next planned release: Splitting my admin and the main part of the site into modules. Also going to make a better article manager. Then I'm going to add published/unpublished article support. After I get done with that I will implement Skin support and implement skins with Zend_Layout. The big thing you'll notice is a new look for codeangel.org!

• Now Powered By Zend Framework 1.0.1!
• Improved Security Auditing and logging
• 404 and 500 error pages
• Cacheing, Front Page now twice as fast!
• Other Refactor using more features of Zend Framework, and code cleanup for faster execution.

Codeangel 0.5 the user will finally notice some feature updates, including linkback support, comment subscription, and a better searching experience.

The default file logging format for Zend_Log File Writer is as folows:

%timestamp% %priorityName% (%priority%): %message%

Which is fine for error logging. For other sorts of logging like security auditing, We need more, like an IP of the visitor and a hostname. Other sorts of logging you might want to log things like the request where the error occurred. This is very easy to do with Zend_Log, however this really isn't documented and I've found people doing weird things like extending Zend_Log to achieve this. Let's look how to do this right.

 
//first we create a file writer
$stream = new Zend_Log_Writer_Stream('./path/to/logs/sec.log');
//then we create a log formatter, note that we changed the default
//fields to add the user ip and the user host of the request.
$formatter = new Zend_Log_Formatter_Simple('%userip% - %userhost% - [%timestamp%] %priorityName%: %message%' . PHP_EOL);
//now we add the new formatter to the writer
$stream->setFormatter($formatter);
//then we add the writer to create a new logging instance
$logger = new Zend_Log($stream);
//now we give meaning to the newly added fields.
//I also changed the default format of the timestamp
$logger->setEventItem('userip',$_SERVER['REMOTE_ADDR']);
$logger->setEventItem('userhost',gethostbyaddr($_SERVER['REMOTE_ADDR']));
$logger->setEventItem('timestamp', date('m-d-Y H:i:s', time()));
 
//now in our code we can log away
$logger->emerg("User Failed Login");
 

In our log we'll get nice messages with IP and Host:

127.0.0.1 - localhost - [08-22-2007 21:54:30] EMERG: User Failed Login

I was surprised the other day when someone came into ##php on freenode wanted a strange feature when he hit a wall designing his app with exceptions. He wanted something like the following:

//NOTE THIS IS NOT POSSIBLE, IT'S BULLCRAP!!
try {
  throw new Exception("throw an exception");
} catch (Exception $e) {
  //do something with a thrown exception.
} else {
  //do something if there wasn't an exception thrown.
}

That's right, he wanted an 'else' for a try/catch block. My mind was blown, surely he was using exceptions wrongly. After I did some inquiring on why he wanted such a ridiculous feature, I found out he was indeed misusing exceptions. He was using exceptions to control his code execution flow. Using them to drive his application by recoverable errors. He was quite defensive about it too! I decided if there weren't enough blog articles about correct exception usage, there needed to be one more...

This tutorial isn't specific to PHP, It applies to all languages that support exceptions. Error checking is very important in any language. You should be checking for any possible place where an error can occur.

 
$db = new my_db('host','user','pass','db');
if($db->connect_error()){
  die("Error Connected to the Database");
}
$stmt = $db->prepare("SELECT * FROM table WHERE field = ?");
if($db->error()){
  die("Could not prepare query");
}
$stmt->bind_param(1,'value');
$stmt->execute();
if($db->error()){
  die("Could not execute query");
}
 

With exceptions this becomes a lot cleaner. The error checking can all be moved to one place.

 
try {
  $db = new my_db('host','user','pass','db');
  $stmt = $db->prepare("SELECT * FROM table WHERE field = ?");
  $stmt->bind_param(1,'value');
  $stmt->execute();
} catch (Exception $e) {
  die($e->getMessage());
}
 

Exceptions are great. But they're meant to track one kind of error only: Exceptional Errors. Exceptional errors are non-recoverable. Exceptional errors are things that are unexpected. Exceptional errors should not have happened in any normal sense. If you remove all Exception handling in your code, the code should still run (that is if nothing exceptional happened.)

Here are some examples where an exception is not warranted:

• User enters an incorrect password, you should expect this
• A user enters alpha characters in a phone number field that should only contain digits, WRONG, but you should expect this behavior
• Checking that a file exists that you have no idea if it should exist or not.
• A Search returns no results.

Examples of where exceptions may be warranted:

• A database connection fails that should be able to connect.
• A SQL query cannot execute because it is malformed.
• Checking that a file exists that should be there.
• A Parser cannot parse because the data is malformed.
• Checking for existence of required properties in a class that should have been set before a method was run.

Why this distinction of when to use exceptions? The biggest reason is that exceptions moves the code execution to a catch statement much like goto moves code execution to a different part of the program, and can create spaghetti code if used incorrectly. A normal day to day catch statement should show a friendly error to the user, log or show the exception, and halt the execution flow.

I hope this article helps readers understand when to use exceptions, and if it can prevent just one person from designing their code like the user in the first paragraph, it did it's job.

References:
Hunt, Andrew and Thomas, David The Pragmatic Programmer Addison-Wesley, Feb 2006.

• Added links for digg, furl, stumbleupon, del.icio.us, technorati, etc... on each article page.
• Changed code highlighting class from Pear Text::Highlight to GeSHi
• Refactored the models a little bit with better error reporting. Not completely the way I want them yet, mostly due to the limitations of the older Zend Framework I'm working on.
• Fixed various small bugs

0.4 well be a big change, upgrading to Zend Framework 1.0. Because of the large differences between 1.0 and 0.8, there will be alot of code refactored (for the better).

• Changed homegrown js to the jquery library
• fixed a bug with md5 files on the download page(thanks to David for pointing it out)
• Completely changed my article linking method, the old way was a failed experiment.
• Made warnings to check when my posts weren't valid XML.
• truncated lists of tags and archives in the sidebar.

plans for 0.3 include adding links for dig this, technorati, delicious and what not for the articles and replacing the code highlighting lib with GeSHi. In 0.5 I plan on refactoring the app to work with Zend Framework 1.0 and to include things like cacheing and logging. Also I hope to replace some of my home-grown components with Zend Framework's components.

The Zend Framework 1.0 has been released! I have been keeping an eye on/involved in this project since I first heard about it almost 2 years ago. My involvement started out learning the framework and providing feedback. Lately I have joined in the cause to help out with documentation and will continue to do so from now on.

This very blog has been written in Zend Framework. It's 0.8 and I need to upgrade it and refactor it, and I don't have time with my current projects. It will happen soon though!

If your a web application developer, you really have to try Zend Framework out, it has many libraries for fast application development, but allows for much more degree of control than most other frameworks, making it more enterprise family.

The Zend Framework is more a collection of libraries ready for your use, It also gives plenty of abstract classes and interfaces to add on or change the framework to meet your needs. The most notable components of the Zend Framework are it's MVC implementation, a PHP port of Lucene, and a stand alone PDF class. And that's not all!

So give the Zend Framework a look and try it out for your next project! A word before you judge to early, one of the missing components that we couldn't get in to the 1.0 release was a partial view renderer, If you see this as a loss, don't worry it'll most likely be out by the next realease.

One could tweak Zend Framework's Router_Route to meet almost all your routing needs. But what if you want something beyond what that package can offer? You can make your routing dreams come true with making your own custom router, all you need to do is implement Zend_Controller_Router_Route_Interface (that's a mouth full).

Case example, I needed a website that could have an arbitrary amount of hierarchal categories. I wanted my URI path to reflect the full path to that category. For example:

http://www.example.org/category/clothing/hats/dress/fedora

This URL would represent the category controller. Each category is a child of the category to the left of it, clothing being the top most parent.

For this particular router I ignored the action component. The category controller will only have an index action, for all that is needed is to view the contents of the category. Administration of the categories happen in a completely different app, so I really don't need the usual scaffolding behavior. I just need the Route to route to the correct controller and then put the rest of the URL into a flat array that I can retrieve with _getParam(). Looking at the example below however, it would be very easy to figure out how to make action routing work right :)

Zend_Controller_Router_Route_Interface requires to implement three methods: match(), assemble(), and getInstance(). The match() method is what does all the magic, it parses the url, decides if it's the right router for this url, and then passes an array of info to the dispatcher. The assemble() method is used to reconstruct an URL given an array of data. This is mainly used by some helpers like Zend_View_Helper_Url. I won't show how to implement this method since it's slightly beyond the scope of this article. The assemble() is not necessary for the Router to work. Since the Interface requires it, I'll put it in the class as a dummy method. You can play with Url Helper on your own to get assemble() to return what is needed for this Helper. The getInstance() method is simply a factory method to create instances of itself from Zend_Config.

 
<?php
class MyApp_CatRoute implements Zend_Controller_Router_Route_Interface
{
	public $defaults = array();
        public $name = null;
 
        //this class sets the defaults and the name of the controller
        public function __construct($name, $default = array())
        {
            	if(!empty($default)){
            		$this->defaults = $default;
            	} else {
            		$this->defaults = array(
                        'controller' => 'category',
                        'action' => 'index'
                    );
            	}
                $this->name = $name;
        }
 
        public function match($path)
        {
            //splits the URL into an array
            if (preg_match_all('#/([^/]*)#', $path, $matches)) {
                //gets the array from the preg_match
                $segments = $matches[1];
                /*pops off the first element and checks if 
                 *this is the right router, if not, return   
                 *false, Zend framework will continue looking 
                 *for the right router
                 */
                $category = array_shift($segments);
                if ($category != $this->defaults['controller']){
                	return false;
                }
                //creates an array, the array with all the categories are indexed by 'cats'
                $return = array(
                    'category' => $category,
                    'cats' => $segments 
                );
                //merges the array above with the defaults and returns it.
                $return = array_merge($return,$this->defaults);
                return $return;
            }
            return false;
        }
 
        public function assemble($data = array())
        {
            return $route;
        }
 
        public static function getInstance(Zend_Config $config){
            $defs = ($config->defaults instanceof Zend_Config) ? $config->defaults->toArray() : array();
            return new self($config->route, $defs);
        }
}
?>
 
 

Basically the idea is to have match() return an array with atleast the 'controller' and 'action' indexes set. If your using a modular layout you would also need a 'module' index in the array. Any other index you can set is completely up to you, and those indexes are accessible with _getParam() in the controller. Here is how I set up this Route in the bootstrap:

 
require_once('MyApp/CatRoute.php');
$rwRouter->addRoute('cat', new MyApp_CatRoute('category',array('controller' => 'category', 'action' => 'index')));
 

Finally one can grab the array of categories easily in your action:

 
$cats = $this->_getParam('cats', array());
 

If you have any questions you can find me on irc on freenode.com in #zftalk. You can also find me on the ZF General Mailing list. Have Fun!

Tomcat 6 has some neat new features. The one feature I am eyeballing the most is it's advanced IO features like the Comet Servlet Interface. In an upcoming article I'll be explaining the Comet Servlet Interface some more, Plus a neat proof of concept!

In order to get these advanced IO features, one has to enable APR listener support. APR is Apache's portable runtime that gives developers an API which behavior is consistent across many platforms. This runtime is the magic that makes the latest versions of Apache's httpd so darn portable. It's also needed for some advanced features in Tomcat!

I am assuming you have tomcat 6 installed at this point and are trying to figure out how to get APR working. I am also assuming you are running Linux (If not you better!). To install APR support you need JNI headers, OpenSSL, and of course the APR lib. If you have a Java JDK 1.4+ installed on your system, the JNI headers should already be there. If you don't have this, go to the Java web site, grab the latest JDK, install it, and point your JAVA_HOME environment variable at it. Depending on your distribution, you may have to do some other voodoo to get your default Java switched over. For me all I had to do was move a symlink. Most likely you already have OpenSSL libs installed on your Linux Distribution, you might need to grab the headers so that you can compile against the libraries, usually these are in a package named such as libssl-dev. You'll also need the APR library and headers. Some distributions already have APR available through packages. Go check your package repos and install libapr and libapr-dev packages. For me, I had to install APR from the source packages here. You Should only have to do the familiar ./configure && make && make install, to install APR from source.

Now that we got the deps out of the way, time to get APR working in Tomcat. First thing we need to do is compile the JNI wrappers. The source for this comes with Tomcat, cd into your Tomcat $CATALINA_BASE/bin/ directory, and do the following: (paths may vary)

tar -xvzf tomcat-native.tar.gz
cd tomcat-native-1.1.10-src/jni/native
./configure --with-apr=/usr --with-java-home=/usr/lib/java --with-ssl=yes
make
sudo make install

After the make install it should give the path where the wrappers were installed, for me they were in /usr/local/apr/lib/. Now the hard part was getting Java to find these wrappers, since they are not in the usual java.library.path. To get them in the java.library.path, set the LD_LIBRARY_PATH environment variable on server startup. Edit tomcat's startup.sh script and add a line near the beginning (I did mine after the uname case stuff).

export LD_LIBRARY_PATH=/usr/local/apr/lib

Restart Tomcat and enjoy! You will be able to tell that APR support is installed by going to your /manager/status webapp, there should be a new section called 'OS' that gives OS memory and CPU information.

Mathematica 6.0 has been released today! Mathematica 6 has gone in a different direction than any of it's predecessors. It allows users to create dynamic interfaces quickly and easily. It also allows interactivity beyond what any previous version offered.

Mathematica is a highly interactive functional programming language that allows almost anyone to produce mathematical models and simple code with little knowledge of computer science. With Mathematica 6 it allows users to create interactive dynamic interfaces in a few lines of code that would take a traditional programming language thousands of lines to produce. Mathematica has the ability to connect to the internet and databases, parse xml, has regular expression functionality, and anything else that I would need for my day to day programming tasks. Even though it was originally built for mathematicians and scientist, almost anybody can benefit from using Mathematica for their programming needs.

Along with Mathematica 6, Wolfram Research also released a free Mathematica Player that allows you to download and run free Mathematica notebooks from Wolfram's Demonstration Project and allows you to experience what Mathematica 6 can do!

This movie was a great way to re-live my childhood. A story about 4 Teenage Mutant Ninja Turtles. Ninjas + Turtles = Win. I have been waiting for the release of this film since I first heard of it 9 months ago. I was enthralled when I saw the first trailer about 3 months ago. I had to see it on opening day.

When I was a wee lad, I watched the cartoon religiously. I had the toys. In fact I recall that a Michaelangelo action figure is the first toy I bought with my own money. I dressed up like Splinter for Halloween one year as a kid. I probably drove my parents crazy watching the movies over and over.

I remember the first three movies. Live action movies with men in big rubber suits and makeup that they could hardly even move was the theme to all of the movies. Big rubber suits are not conducive to ninja like skillz, not to mention make the turtles look really really fat.

Then this new movie came out. Done with computer animation, the way it was meant to be. Turtles could actually act and move like ninjas. The fight scenes were awesome. The characters were done very well. The villains were cool. This movie just all out rocked.

Only complaint I heard was the fact that April O'Neil has a unbelievable skinny waist and hoooge boobs that made her look like an alien. Those complaints were from the girls I went to see the movie with. I, however, approved. ^_^

So, if you grew up with the turtles, this movie is awesome. If you have kids, take them to see the movie, they'll love it. For everyone else, well, you'll find it entertaining at least. This movie seems to target either kids or people who grew up with turtles on their T.V. But, I think most audiences would love this movie.

I give this movie 4.5 angel wings out of 5.

Last week I took my family and friends out to see the Engineering Open House (EOH) at the University of Illinois. The open house showcased the different aspects of engineering in a science fair format. The exhibits ranged from things interesting and cool for adults, to things fun and exciting for the kids. One exhibit, titled R.A.G.E. Beyond The Ether, located in the Computer Science section was the highlight of my day. Second place would have to go to a Distributed Music Library system which I might highlight in another article.

R.A.G.E. Beyond the Ether was, in my opinion, an overly pompous name. Cool for kids I guess. I believe the main point of the exhibit was to see how computer vulnerabilities were discovered, and what was at risk. Unfortunately I believe the college kids at this exhibit were misunderstood, as an older gentleman asking questions quickly labeled the students as Virus Writers. The students, and most certainly I, dismissed such an ignorant statement. The students, rather than writing and distributing virii primarily aimed at selling male enhancement pills, were actually working on a potentially important discovery in computer security.

The actual demonstration consisted of an attack between two Macintosh laptops. The attacking mac was connected via a Firewire cable to the victim mac. A walking by spectator would inquire on what the exhibit was about, interrupting the poor student's Magic: The Gathering game. The students would ask the spectator to enter in a password on the the attacking machine and hit enter. When the spectator hit enter, a python script was run. When the script finished, the spectator was directed to go to the victim machine and hit shutdown on the login panel. Instead of the machine shutting down, the login screen blinked. The spectator was then asked to log in as an adminstrator on the victim's machine using the password he entered in on the attacking machine. Low and Behold, it worked! The attacking machine was able to change the admin password on the victim by just hooking up to another computer via Firewire and running a script. If that doesn't scare you, you haven't heard the least of it yet.

After seeing this demonstration I grilled the students on the details of this attack. Given that there won't be a vulnerability report or any sort of release until more testing is done on the nature of this attack, I will only divulge on how the attack works. I will also discuss potential theories and hazards that this discovery could uncover.

Don't worry Mac Users, the vulnerability isn't necessarily a Mac problem. The issue is the implementation of Firewire. Firewire uses Direct Memory Access (DMA). The exploit took advantage of this feature and used DMA to directly write to the memory and change the behavior of the shutdown button on the login screen. Instead of shutting down, the code was replaced with a program that changed the root password on the mac.

The theory is we now have a potential security problem with any operating system and any technology that uses DMA and is capable of a remote connection to another computer or device. Firewire? USB 2.0? Some Gigabit Ethernet chipsets? BlueTooth? Are you scared yet?

Moral of this story? Think twice when somebody wants to connect their iPod to your computer. The iPod, or any computer device, could potentially be modified to run the same code that changed the computer's password.

It's Here at last! CodeAngel.org is a project I have been working on, on and off, for almost a year! CodeAngel.org is my blog. The software behind this was written by me, and of course those who wrote the libraries behind it. The application base was written with the Zend Framework. I started writing this blog when Zend Framework was on 0.1.2 and at this time of writing is running on 0.8.0, I'll continue to update this blog based on Zend Framework perhaps perpetually.

The main reason I wrote my own blog software, instead of say use any of those 100s of other blog and content management software out there, is for two reasons. One reason is that I wanted a project to better understand MVC and how it applied to Web Development. The other reason why I reinvented the wheel, all the wheels out there didn't fit my car.

The main features I wanted for a cms/blog software is something aimed at developers, coders, nerds etc. First I wanted something that allowed for multiple or single page blog posts, allowing to write short little blurbs famous amongst blogs, or to pour on page after page in some sort of technical tutorial (I already have quite a few planned). Also I added on Text Highlighting, with Pear Text::Highlight:

 
{ package Animal;
  use Carp qw(croak);
  
  ##constructors
  sub named {
    ref(my $class = shift) and croak "class name needed";
    my $name = shift;
    my $self = { 'Name' => $name, 'Color' -> $class->default_color };
    bless $self, $class;
  }
}

The best thing about Text_Highlight is that it is extensible. I have plans for adding on some modules to it so that it supports Bash, C#, and maybe VB.Net.

Next page I touch on the future plans CodeAngel