Password Security

How can someone gain access to your e-mail account without knowing your password? Have you ever signed up for a new website or service you wanted to try out? Did you give them your e-mail address during registration? Did you use the same password you used for your e-mail account?

Newer websites and services that haven’t been globally trusted may be scraping your passwords during registration and testing them against the e-mail account you provided. Once gaining access, it’s easy to scan e-mail for messages from other services and begin requesting access to other accounts associated with it.

You may have also registered with a new website or web service that is not necessarily malicious. They may just be storing your password insecurely and unencrypted. A potential hacker may have discovered a security exploit allowing them to access that password data.

The solution isn’t to simply use only trusted sites. Using new services and trying new concepts is what helps the Internet grow. Plus, even large trusted services can have their security compromised by badly written code allowing third parties to gain access to your data. The better solution is to use unique password for every website. How would you remember a unique password for every website? You’d use a password algorithm.

Password Algorithms

Many Internet users typically use the same password for multiple systems. Some people use a slightly more secure technique of having different levels of passwords. They may use one for unsecured sites, such as social networking or instant messaging, another for secure sites such as e-mail and a very strong one for computer and bank passwords. Although this is better than using the same password everywhere, it’s not replacement for the security provided by a unique password algorithm.

A good password algorithms can be based off the website the password is intended for. For instance, you can take the first or last letter of the website, combine it with a pin number or short password, and then tack on a character representing something else about the site. For example, you could add a letter representing the top level domain (e.g. The letter ‘A’ for .com, ‘S’ for .net, ‘D’ for .org and ‘F’ for others).

You could also use something on the site itself, like the company’s primary color or the background color of the website. Although using something on the page itself may change over time, this may also force you to continually change and rotate out passwords.

You can be creative and find many different ways to create a good password system. However there are a few basic rules you should try to follow to ensure good password selections.

• Make sure your pattern will always give you a password with at least one capital letter, one lowercase letter and one number
• Ensure your system always gives you a password between 8 and 9 characters long. Many websites have restrictions on length and 8 or 9 characters ensures your password will never be too short or two long
• Avoid special characters. Many websites ban these (due to ignorance) for security and it’s much more difficult to remember exceptions to your password system.
• Chose a system you can remember easily, but that’s not easy to understand should someone get a hold of one of your passwords.

In addition to individual passwords for each website, you should also have secure passwords for non-web related systems such as company computers/networks and home computers.

You don’t need to change every password at once when you come up with an algorithm. Just start changing passwords as you access sites you use more frequently. Whenever you access a website with your old password, go into that website’s account options and change it. You can then gradually adjust to a new password algorithm over the course of a few weeks.

Additional Tips

• Never use your web browser’s password store to remember passwords for you. Turn it off, remember your password system and commit them to memory.
• Always remember to log-off your accounts on computers that are not yours or that you cannot secure.

Security Questions

One serious piece of contention in the network security industry are the increasing use of security questions. Although they are used to alleviate customer services calls, the questions themselves are easily guessable.

In an 2009 IEEE Symposium on Security and Privacy, researchers from Microsoft and Carnegie Mellon University showed that in a study involving 130 people, 28% of the people who were known and trusted by the studies participants could guess the correct answers to participants’ security questions. Even people not trusted still had a 17% chance of guessing a participant’s answers¹.

Some security experts suggest typing in random letters for security questions, going through the additional burden of calling the company should one’s user account become locked. Others suggest using a similar password algorithm as described above to generate a separate password for the security question. Without doing one or the other, the system a user is accessing gains an automatic backdoor to his or her account, defeating the purpose of having password based security in the first place.

A better solution to using security questions is to use reset codes sent to user’s e-mail addresses, or combining an e-mailed reset code with both security questions and locking an account after several bad attempts. While this is a better solution, many websites cannot implement these e-mail based password reset systems because they allow multiple accounts to use the same e-mail address.

Authentication without Password

Another item to watch out for are services which ask for a password to another services. For example, a new social networking site that may ask for your e-mail password in order to search for friends that might be using the new service. While at one time this was the only way for third parties to access information on your account, it isn’t any longer.
Now almost all major services support some type of single sign-on authentication. Twitter uses an OAuth based system, Facebook has their own Facebook Connect system, Microsoft has LiveID and Google has third party account authentication. These systems work by directing you to a given service (Facebook, Twitter, Google, etc.) with a security token. On that site, you log-in and authorize the token granting the other website limited access to your account.

The advantage of this system is that if the 3rd party service does start doing something malicious, such as sending spam messages to your contacts, you can simply revoke the applications access to your account without having to change any passwords. Most services even allow their users to report malicious applications, which could cause their application tokens to be rejected permanently for all users.

Conclusions

Password security is critical in a world where so much of your personal information an accounts can be accessed electronically. Developing and protecting a good password system is more important in many ways than defending your social security number.

At a minimum, you should have different levels of passwords: An insecure password for social networking websites and other trivial services, a high security password for e-mail, payment accounts and other high security services, and finally an ultra secure password for critical websites such as on-line banking.

Ideally, you should create a password algorithm for all web passwords. Algorithms should always generate 8 to 9 character passwords that include at least one capital letter and one number, and the system should be easy to remember yet difficult to understand should one or more password become compromised.

If a user suspects a malicious website or application has gained access to one of his or her accounts, the user should change the password on that account immediately. Using an algorithm for generating passwords ensures that other accounts won’t be comprised. Without it, a user may have to change password on several websites afterwords.

Switching to a password system from a single universal password may seem a bit cumbersome, but the transition is a lot easier than one would anticipate. I switched from a set of three passwords to an algorithm three years ago and now have a much greater degree of confidence about the security of my information.

¹ Are Your Secret Questions Too Easily Answered?. Lemons. Technology Review. May 18, 2009.

public class  MyServiceBean {

    public void startProcess() {
    
        DataSource ds = (DataSource) new InitialContext().lookup("jdbc/ds1");
        con = ds.getConnection();
        
        //do something with connection        
    }    
}

The DataSource is simply an interface which the underlying application container implements in order to allow applications to get connections. In this way, a web application server can control the way connections are handed out as well as keep connections in a pool to be reused. The information about the data source, including the driver, host name, user name, password and database name are all setup on the web application server. The way they’re configured varies depending on the server (most have a web administration console or XML configuration files), but all application servers provide an initial context containing references to these data sources for all running web applications.

If we simply tired to test this bean using a main function in its own stand-alone application like so:

public static void main(String[] args) {        
    MyServiceBean b = new MyServiceBean();
    b.startProcess();              
}

We would get the following exception:

javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file:  java.naming.factory.initial
	at javax.naming.spi.NamingManager.getInitialContext(Unknown Source)
        .....

In order to run code which looks up a DataSource this way in a stand-alone application, we must create our own implementation of a DataSource object as well as an InitialContextFactory, plus several intermediary objects, and then tell our currently running JRE to use our context object for all subsequent calls to new InitialContext(). Because we’re running locally, we don’t need to implement everything out of all these interfaces, just the bare minimum in order to provide DataSource and Connection objects.

First, we’ll look at a very quick and dirty solution. In this solution, we declare new classes directly within the main function using the final keyword and hard-coding the driver and connection strings. This is a quick drop to simply test a single bean.

public static void main(String[] args) throws SQLException, ClassNotFoundException, NamingException
{
    final class LocalDataSource implements DataSource , Serializable {

            private String connectionString;
            private String username;
            private String password;
            
            LocalDataSource(String connectionString, String username, String password) {
                this.connectionString = connectionString;
                this.username = username;
                this.password = password;
            }
            
            public Connection getConnection() throws SQLException
            {
                return DriverManager.getConnection(connectionString, username, password);
            }
    
            public Connection getConnection(String arg0, String arg1)
                    throws SQLException
            {
                return getConnection();              
            }
    
            public PrintWriter getLogWriter() throws SQLException
            {
                return null;
            }
    
            public int getLoginTimeout() throws SQLException
            {
                return 0;
            }
    
            public void setLogWriter(PrintWriter out) throws SQLException {}
    
            public void setLoginTimeout(int seconds) throws SQLException {}

    }
    
    final class DatabaseContext extends InitialContext {

        DatabaseContext() throws NamingException {}

        @Override
        public Object lookup(String name) throws NamingException
        {
            try {
                //our connection strings
                Class.forName("com.mysql.jdbc.Driver");
                DataSource ds1 = new LocalDataSource("jdbc:mysql://dbserver1/dboneA", "username", "xxxpass");
                DataSource ds2 = new LocalDataSource("jdbc:mysql://dbserver1/dboneB", "username", "xxxpass");

                Properties prop = new Properties();
                prop.put("jdbc/ds1", ds1);
                prop.put("jdbc/ds2", ds2);
                
                Object value = prop.get(name);
                return (value != null) ? value : super.lookup(name);
            }
             catch(Exception e) {
                 System.err.println("Lookup Problem " + e.getMessage());
                 e.printStackTrace();
             }  
             return null;            
        }        
        
    }

    final class DatabaseContextFactory implements  InitialContextFactory, InitialContextFactoryBuilder {

        public Context getInitialContext(Hashtable<?, ?> environment)
                throws NamingException
        {
            return new DatabaseContext();
        }

        public InitialContextFactory createInitialContextFactory(
                Hashtable<?, ?> environment) throws NamingException
        {
            return new DatabaseContextFactory();
        }
        
    }
    
    NamingManager.setInitialContextFactoryBuilder(new DatabaseContextFactory());   

    MyServiceBean b = new MyServiceBean();
    b.startProcess();
}

In this example we’ll start from the bottom. Before we run our bean we see a call to NamingManager.setInitialContextFactoryBuilder(). This function sets the factory that will be called in our environment by all subsequent calls to new InitialContext() throughout our application. Underneath the hood of a web application server, this is one of the many things that happens well before a web application is loaded.

The DatabaseContextFactory is a class we create that not only serves as a ContextFacotry but also a ContextFactoryBuilder through appropriate interfaces. Thanks to interfaces, we can simplify these two functionalities into a single class. It’s sole purpose is to return a DatabaseContext.

The DatabaseContext extends a regular InitialContext and we simply override the one function typically used by web server code, the lookup() function. It is here we can inject our own LocalDataSource objects, which return our Connection objects. On an actual web application server, the DataSource object would typically have some type of pooling mechanism that could return existing connections into a queue once the web application calls the close() function on them.

Although is is a decent quick solution, it’s really unclean and isn’t reusable without copying and pasting. It also ignores any environment parameters passed into the InitialContext and discards them. For a more permanent solution, each of the classes should be separated out and compatibility should be maintained with the environment properties passed in to our custom Context object.

For our clean and reusable solution, we’re only going to have one class with public visibility. It’s our factory class. All other classes will have default/package visibility because they shouldn’t be instantiated independently outside of our factory. Let’s start with the factory:

package org.penguindreams.db.local;

import java.util.HashMap;
import java.util.Hashtable;
import java.util.Map;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import javax.naming.spi.InitialContextFactory;
import javax.naming.spi.InitialContextFactoryBuilder;

public class LocalContextFactory {
	/**
	 * do not instantiate this class directly. Use the factory method.
	 */
	private LocalContextFactory() {}
	
	public static LocalContext createLocalContext(String databaseDriver) throws SimpleException {

		try { 
			LocalContext ctx = new LocalContext();
			Class.forName(databaseDriver);	
			NamingManager.setInitialContextFactoryBuilder(ctx); 			
			return ctx;
		}
		catch(Exception e) {
			throw new SimpleException("Error Initializing Context: " + e.getMessage(),e);
		}
	}	
}

In the above code, we’re creating a new LocalContext. We’re also initializing our JDBC driver. As with the previous example, the NamingManager.setInitialContextFactoryBuilder function ensures the new context we’re creating will be given to all subsequent calls made to new InitialContext(). Also, as with the previous example, the LocalContext takes both the role of an InitialContextFactory and an InitialContextFactoryBuilder through the use of interfaces.

package org.penguindreams.db.local;

import java.util.HashMap;
import java.util.Hashtable;
import java.util.Map;

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import javax.naming.spi.InitialContextFactory;
import javax.naming.spi.InitialContextFactoryBuilder;

class LocalContext extends InitialContext implements InitialContextFactoryBuilder, InitialContextFactory {

	Map<Object,Object> dataSources;
	
	LocalContext() throws NamingException {
		super();
		dataSources = new HashMap<Object,Object>();
	}
	
	public void addDataSource(String name, String connectionString, String username, String password) {
		this.
		dataSources.put(name, new LocalDataSource(connectionString,username,password));
	}

	public InitialContextFactory createInitialContextFactory(
			Hashtable<?, ?> hsh) throws NamingException {
		dataSources.putAll(hsh);
		return this;
	}

	public Context getInitialContext(Hashtable<?, ?> arg0)
			throws NamingException {
		return this;
	}

	@Override
	public Object lookup(String name) throws NamingException {
		Object ret = dataSources.get(name);
		return (ret != null) ? ret : super.lookup(name);
	}	
}

In the above example, we also see that we’ve allowed for some default behavior. For instance, properties that are given to initialize our LocalContext are stored in our local HashMap. If a lookup fails, we call the parent’s lookup method as well. The important function we add above is the addDataSource() method which creates new LocalDataSource to be looked up by the passed in name argument. Finally we have the LocalDataSource itself.

package org.penguindreams.db.local;

import java.io.PrintWriter;
import java.io.Serializable;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;

import javax.sql.DataSource;

class LocalDataSource implements DataSource , Serializable {
	
	private String connectionString;
    private String username;
    private String password;
    
    LocalDataSource(String connectionString, String username, String password) {
        this.connectionString = connectionString;
        this.username = username;
        this.password = password;
    }
    
    public Connection getConnection() throws SQLException
    {
        return DriverManager.getConnection(connectionString, username, password);
    }

	public Connection getConnection(String username, String password)
			throws SQLException {return null;}
	public PrintWriter getLogWriter() throws SQLException {return null;}
	public int getLoginTimeout() throws SQLException {return 0;}
	public void setLogWriter(PrintWriter out) throws SQLException {	}
	public void setLoginTimeout(int seconds) throws SQLException {}
}

As you can see, many of the functions for the DataSource have been left unimplemented. We’ve only implemented enough functionality to get lookups working and to create simple, non-pooled connections for the end client. So our main function should now look something like the following:

public static void main(String[] args) {        
    
    LocalContext ctx = LocalContextFactory.createLocalContext("com.mysql.jdbc.Driver");
    ctx.addDataSource("jdbc/js1","jdbc:mysql://dbserver1/dboneA", "username", "xxxpass");
    ctx.addDataSource("jdbc/js2","jdbc:mysql://dbserver1/dboneB", "username", "xxxpass");
    
    MyServiceBean b = new MyServiceBean();
    b.startProcess();              
}

The call to LocalContextFactory.createLocalContext() initializes our environment with the LocalContext as the InitialContext. Then we can add the DataSource objects we need later using the addDataSource() method. All of our data objects are kept within their own packages and have limited visibility to ensure they can only be instantiated and fully initialized using the factory method.

There are some limitations to this clean implementation. For one, you can only use one type of database depending on what driver you specify with the LocalContextFactory. Also, not all of the DataSource methods are fully implemented, so you may run into problems in environments that depend on other functions and more complex implementations.

Still, the above code will work in a testing environment and, in a pinch, you can use the final classes used at the beginning of this tutorial directly in a main function for some quick and dirty testing. If you need a more complete implementation, the clean version of the code is an excellent starting point to begin building a full local implementation of your own custom DataSource objects.

Our web.xml for our WAR file is fairly basic. It contains a single servlet and a mapping to that servlet.

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" id="WebApp_ID" version="2.5">
  <display-name>SimpleRest</display-name>
   
  <servlet>
  	<servlet-name>RestService</servlet-name>
  	<servlet-class>org.penguindreams.service.ServiceHandler</servlet-class>
  </servlet>
  
  <servlet-mapping>
	<servlet-name>RestService</servlet-name> 
	<url-pattern>/models/*</url-pattern>
  </servlet-mapping>	
	
</web-app>

Likewise, our application.xml, which is used to build an EAR file which can contain multiple WAR files, is very basic. We see it only contains support for one web module.

<?xml version="1.0" encoding="UTF-8"?>
<application xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:application="http://java.sun.com/xml/ns/javaee/application_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/application_5.xsd" id="Application_ID" version="5">
  <display-name>SimpleRestEAR</display-name>
  <module>
    <web>
      <web-uri>SimpleRest.war</web-uri>
      <context-root>rest</context-root>
    </web>
  </module>
</application>

Our servlet overrides the doGet method, sets the current time as an attribute and then passes on control to a JSP to finish processing the request.

public class ServiceHandler extends HttpServlet {
	
	@Override
	protected void doGet(HttpServletRequest request, HttpServletResponse response)  { 		
		try {			
			request.setAttribute("timeStamp", new Date().getTime() );			
			getServletConfig().getServletContext().getRequestDispatcher("/hello.jsp").forward(request,response);			
		} catch (Exception e) {
			System.err.println("Fatal Servlet Error");
			e.printStackTrace();
		}			
	}
	
}

Likewise, our JSP page is very simple. It just displays a header image and the resulting time.

<%@ page language="java" contentType="text/html; charset=ISO-8859-1"
    pageEncoding="ISO-8859-1"%>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html  xmlns="http://www.w3.org/1999/xhtml" dir="ltr" >
<head>
	<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" />
	<title>Sample Page</title>
</head>
<body>
	<p>
		<img src="images/penguindreams.gif" alt="PenguinDreams" />
		<br />
		TimeStamp: <%= request.getAttribute("timeStamp") %>
	</p>
</body>
</html>

So now that we have our application out of the way, let’s focus on the building process. The first part of the build.xml contains our variables. The sample application is built for JBoss. Depending on your web application server, the paths and variable names should be adjusted. Notice we’re pulling in both the jar libraries within our web application as well as the libraries found on the application server itself.

Eclipse does this automatically when you specify a server runtime. When building from an ant file, these libraries need to be specified for the build to complete. Be careful not to include libraries in your local lib folder that are all ready present on the application server (i.e. JSF libraries, Servlet API, Struts, log4j, et cetera). This could cause complications. For instance, deploying a WAR that contains a log4j.jar within it will cause the logging handlers within JBoss to stop working for your web application.

    <property name="build" value="./build" />
    <property name="dist" value="./dist" />
    <property name="conf"  value="./conf" />
    <property name="src"   value="./src" />
    <property name="lib"   value="./lib" />
    <property name="web"   value="./web" />
    <property name="version" value="0.1" />
    <property name="jbossLocation" value="/Users/myhomedir/jboss-6.0.0.M1/" />
    <property name="servletLib" value="${jbossLocation}/common/lib" />
    <property name="deployDir" value="${jbossLocation}/server/default/deploy/" />
    <property name="warFile" value="${dist}/${ant.project.name}.war" />
    <property name="earFile" value="${dist}/${ant.project.name}-${version}.ear" />

    <path id="build.classpath">
	<fileset dir="${lib}" includes="**/*.jar" />
	<fileset dir="${servletLib}" includes="**/*.jar" />
    </path>

We need to create some simple cleanup and initialization targets to clear out old bytecode as well as setup our build environment. The compile task will depend on the initialization being complete and make use of the libraries we specified above.

    <target name="clean">
    	<delete dir="${build}" />
    	<delete dir="${dist}" />
    </target>

    <target name="init">
    	<tstamp />
    	<mkdir dir="${build}" />
    	<mkdir dir="${dist}" />
    </target>
    
    <target name="compile" depends="init">
	<javac srcdir="${src}" destdir="${build}" optimize="on">
	    <classpath refid="build.classpath" />
	</javac>
    </target>

The WAR file task is simply an extension of the ZIP file task that places certain types of files within the correct location of the WAR file. Here we specify our web.xml configuration file, our Java classes, the web application libraries and the static content.

    <target name="war" depends="compile">
    	<war destfile="${dist}/${ant.project.name}.war" webxml="${conf}/web.xml">
    	  <lib dir="${lib}" />
    	  <classes dir="${build}"/>
    	  <zipfileset dir="${web}"  /> 
    	</war>
    </target>

The resulting JAR file has the following structure:

META-INF/
META-INF/MANIFEST.MF
WEB-INF/
WEB-INF/web.xml
WEB-INF/lib/
WEB-INF/classes/
WEB-INF/classes/org/
WEB-INF/classes/org/penguindreams/
WEB-INF/classes/org/penguindreams/service/
WEB-INF/classes/org/penguindreams/service/ServiceHandler.class
images/
hello.jsp
images/penguindreams.gif

An EAR file is also very simple. It’s just a collection of WAR files with an XML describing each individual web module and it’s Context Root. For this example, there is only one web module.

    <target name="ear" depends="war">
    	<ear destfile="${dist}/${ant.project.name}-${version}.ear" appxml="${conf}/application.xml">
    		<fileset dir="${dist}" includes="*.war" />
    	</ear>
    </target>

Finally, we have a deployment task. Most application servers can be configured to monitor a deployment directory and automatically install EAR files copied into that directory. If your application server is on a different machine than your build environment, you may need to use a remote deployment task, such as those included with IBM WebSphere to deploy EAR files using either their SOAP or RMI protocols. For this example, our server is on the same machine, so we can do a simple copy:

    <target name="deploy" depends="ear">
    	<copy todir="${deployDir}">
    		<fileset dir="${dist}" includes="*.ear" />
    	</copy>
    </target>

So our final build.xml is shown below. Notice the project root element where we specify the project name as well as the default task to run.

<?xml version="1.0"?>
<project name="SimpleRest" basedir="." default="ear">

	<property name="build" value="./build" />
	<property name="dist" value="./dist" />
	<property name="conf"  value="./conf" />
	<property name="src"   value="./src" />
	<property name="lib"   value="./lib" />
	<property name="web"   value="./web" />
	<property name="version" value="0.1" />
	<property name="jbossLocation" value="/Users/myhomedir/jboss-6.0.0.M1/" />
	<property name="servletLib" value="${jbossLocation}/common/lib" />
	<property name="deployDir" value="${jbossLocation}/server/default/deploy/" />
	<property name="warFile" value="${dist}/${ant.project.name}.war" />
	<property name="earFile" value="${dist}/${ant.project.name}-${version}.ear" />
	
	<path id="build.classpath">
		<fileset dir="${lib}" includes="**/*.jar" />
		<fileset dir="${servletLib}" includes="**/*.jar" />
	</path>
	
	<target name="clean">
		<delete dir="${build}" />
		<delete dir="${dist}" />
	</target>
	
	<target name="init">
		<tstamp />
		<mkdir dir="${build}" />
		<mkdir dir="${dist}" />
	</target>
	
	<target name="compile" depends="init">
		<javac srcdir="${src}" destdir="${build}" optimize="on">
			<classpath refid="build.classpath" />
		</javac>
	</target>
	
	<target name="war" depends="compile">
		<war destfile="${dist}/${ant.project.name}.war" webxml="${conf}/web.xml">
		  <lib dir="${lib}" />
		  <classes dir="${build}"/>
		  <zipfileset dir="${web}"  /> 
		</war>
	</target>
	
	<target name="ear" depends="war">
		<ear destfile="${dist}/${ant.project.name}-${version}.ear" appxml="${conf}/application.xml">
			<fileset dir="${dist}" includes="*.war" />
		</ear>
	</target>
	
	<target name="deploy" depends="ear">
		<copy todir="${deployDir}">
			<fileset dir="${dist}" includes="*.ear" />
		</copy>
	</target>
</project>

We now have a complete project that can be built and deployed using a simple ant command. If you prefer to deploy your project within Eclipse, you can go to Project>Properties and then select Builders and New to create a new Ant Builder. Once it is configured, preforming a build within Eclipse will launch an external delegate to run the build.xml file. Here is the output from running the deploy task.

$ ant deploy
Buildfile: build.xml

init:
    [mkdir] Created dir: /Users/myhomedir/Documents/workspace/SimpleRest/build
    [mkdir] Created dir: /Users/myhomedir/Documents/workspace/SimpleRest/dist

compile:
    [javac] Compiling 1 source file to /Users/myhomedir/Documents/workspace/SimpleRest/build

war:
      [war] Building war: /Users/myhomedir/Documents/workspace/SimpleRest/dist/SimpleRest.war

ear:
      [ear] Building ear: /Users/myhomedir/Documents/workspace/SimpleRest/dist/SimpleRest-0.1.ear

deploy:
     [copy] Copying 1 file to /Users/myhomedir/jboss-6.0.0.M1/server/default/deploy

BUILD SUCCESSFUL
Total time: 1 second

Immediately afterward, we can see JBoss pickup the new EAR file by watching its log file.

01:58:53,696 INFO  [Http11Protocol] Starting Coyote HTTP/1.1 on http-127.0.0.1-8080
01:58:53,743 INFO  [AjpProtocol] Starting Coyote AJP/1.3 on ajp-127.0.0.1-8009
01:58:53,759 INFO  [AbstractServer] JBossAS [6.0.0.M1 (build: SVNTag=JBoss_6_0_0_M1 date=200912040958)] Started in 34s:58ms
01:59:05,970 INFO  [TomcatDeployment] undeploy, ctxPath=/rest
01:59:06,231 INFO  [TomcatDeployment] deploy, ctxPath=/rest

Finally, we can open a web browser, open the appropriate URL to our servlet as determined by both the application.xml and the web.xml.

Creating a web application with an ant build file is just the beginning. The build.xml can be modified with additional variables and tasks to assist in deploying web applications to different environments or for running automated unit tests prior to deployment. Variables defined within the build file can be overwritten using the -D command line option for ant. Scripts can be created on UNIX servers to automatically retrieve your projects from a source control system such as CVS, Subversion or Mercurial and run the ant script within the project with a particular set of arguments based on the environment (e.g. development, test and production).

Using ant to build projects is essential to streamlining development of J2EE web application and ensures consistency in builds and deployments. This tutorial just scratches the surface of building J2EE applications and is part of a series of tutorials for building solid enterprise services and web applications from the ground up. The full source code for the example application shown above can be downloaded as j2ee-ant-example-penguindreams.tar.bz2 or j2ee-ant-example-penguindreams.zip

try {
    sendMail(from,to,subject,body);
}
catch(AddressException e) {
    //Handle This Exception 
}
catch(UnknownHostException e) {
    //Handle This Exception
}
catch(MessagingException e) {
    //Handle This Exception
}

In some projects I would often catch the base exception class and use the instanceof keyword to clean up the exception handling as seen below. Although the code for this technique may be cleaner, it’s more difficult to add new exception types that may be introduced later by code changes:

String error = "";
try {
    sendMail(from,to,subject,body);
}
catch(Exception x) {

    error = "Internal Error: ";

    if(x instanceof AddressException) {
            error += "Invalid Internal Address";
    }
    else if(x instanceof UnknownHostException) {
            error += "Could Not Find Mail Server";
    }
    else if(x instanceof MessagingException) {
            error += "Messaging Problem";
    }
    else {
           error += "Unexpected: " + x.getMesssage();
    }
}

In both instances, programmers typically just log the exception and display a general error to the end user. Catching the individual exception types doesn’t really help in any meaningful way. Also, the exception message itself is often human readable and a clear indication of what error occurred. In my current projects, I use a base exception class and simply place any code that produces checked exceptions within a large try block and rethrow any exceptions as a property of my base exception class.

public class SimpleException extends Exception {

	private Throwable nestedException;
	
	public SimpleException(String s) {
		super(s);
	}
	
	public SimpleException(String s, Exception nestedException) {
		super(s);
		this.nestedException = nestedException;		
	}
	
	public Throwable getNestedException() {
		return this.nestedException;
	}
	
} 

Using this class, the previous example would change to the following:

String error = "";
try {
    sendMail(from,to,subject,body);
}
catch(Exception e) {
    throw new SimpleException("Error Occurred Sending E-mail",e);
}

This cleans up our code significantly. Under the old model, say that the implementation changed and new potential exceptions could possibly be thrown. Each one of those exceptions would have to be added to the throws clause to the appropriate function. If the class implemented a interface or extended and abstract class, those base methods would have to change as well, along with any code implementing the class. If we throw only one type of exception that wraps all other exceptions, we never have to worry about the method signature changing.

Some programmers may oppose this approach and classify it as an anti-pattern, similar to wrapping all checked exceptions in blocks that rethrow runtime exceptions. However, in my experience in working with large projects, using this approach is much cleaner and more maintainable. Furthermore, it allows Aspect Orientated Programming (AOP) interceptors to have a lot more information when dealing with exceptions.

This article is the first in a series of tutorials on J2EE development. The use of the except pattern I’ve used will be further expanded with seeing how it can integrate into web applications using Spring and AOP.

The actual exam’s questions barely intersected with the material on the practice exams. Many of the questions weren’t even covered in the PDF that Zend labeled a “Study Guide.” I checked online to see if there were similar reactions over the certification test. I found one post on the Zend forms by a user named Nick A Williams that looks as if it were lost after Zend preformed an upgrade to their bulletin board software. The following is taken from the search engine cache of the forum¹.

I must say I am very disappointed with the certification program so far. I have found the practice exams and study guide to be very misleading while preparing for the exam.

ZCE Study Guide
The very existence of a study guide suggests its purpose is to adequately prepare the reader for the real exam, covering everything the exam questions will ask. This is extremely misleading and actually reduces the value of the book, as it is clear that there are things asked in the exam that are not touched upon in the study guide.

Without explicit indication of this, a potential exam taker such as myself would have no reason to believe further studying beyond the scope of the book is necessary.

Online Practice Exams
I have found these practice exams to provide absolutely no value whatsoever. I have yet to fail a practice exam (having taken 6 so far), 3 of which were scored as “Excellent” yet I have now failed the actual exam twice. This would suggest that the practice exam does not help one to determine his/her readiness for the actual exam. If this is the case, what is the purpose of the practice exams and why does it cost money to take them?

Zend has established the expectation that if one passes the practice exam, the likelihood of passing the actual one is very high. A quote from Zend’s website:

Quote:

The questions on the practice test are different from the ones used in the real exam, however, they are also more difficult and complex-thus, if you score well on this exam, you should be comfortable that you are well-prepared to take the real test.

Combine this statement with the scores I received, and one can see how easily one can be convinced they are prepared for the real exam when, in fact, they are not.

Purchasing practice exams is supposed to help mitigate the potential for having to purchase re-takes of the actual exam (spend $20 for a 10-pack, aovid another $125 for a re-take). Instead, it has achieved the opposite – providing test-takers with a false sense of confidence (which is quickly shattered upon completing the actual exam).

So the nutshell version?

Test takers: Don’t bother with the study guide or the practice tests. Just RTFM and maybe read everything in the manual realted to the study guide’s section headings. Do that and you should be fine.

Zend: Adjust the descriptions for both the practice exams and study guide in your store to avoid confusion. Inform customers that the study guide is no substitute for the PHP manual. Remove your current practice exam description, and suggest how to properly use the practice exams for studying.

I hope the adoption rate of this certification program is indeed more important to Zend than the sales volume of books/practice exams/real exams. More certified PHP developers will add far more value to your business objectives than a few pointless sales¹.

I don’t want to sound whiny about this at all. After all, certification exams should be somewhat difficult, be challenging and show an aptitude for the technology or language being tested over. Still, I do agree with Mr. Williams statement about how the practice material didn’t relate accurately to the actual exam. Whether this is simply due to Zend not updating their training material or if it’s an intentional attempt by Zend to gain money out of retests is unknown.

I also found some interesting concerns about PHP5 certification in the comments to a post from Michael Kimsal’s weblog back in 2008², showing that if the material is simply out of date, it has been for a very long time.

I’ve tried the Zend Certification Training, I thought it was absolute bullshit (about 1/2 years ago). The exam tested on things like order of parameters and lots of stuff that can be found in the PHP manual. It’s much more important for new hires if they have problem solving skills, including being able to go to the manual, instead of memorizing it. Real skill comes from experience and being a good PHP developer means writing maintainable, secure and scalable code. Not being able to know the manual inside-out.

FYI: I took the PHP architect PHP 5 certification class, studied the PHP 5 Zend certification guide, took the online practice tests and passed. I have been coding PHP for 6 years. Yet I failed the Zend certification test. The test feedback was non-existent. I feel I am a better coder because I have studied. I will not waste my time to take the Zend Certification exam again.

Having been in Java web development for over three years, I also feel like I’m slowly being priced out of the market. Although tools like Spring have eased the burden of Java development in some ways, I still feel like scripting languages like PHP, Python and Ruby are vastly superior in web development to Java.

The horrors of Java Server Faces (JSF) attempt to turn the web into a component based system that HTML was never intended for. Like its .NET competitors based around C# and VisualBasic.NET, I think JSF abstracts way too much of the underlying HTML and makes it very difficult to accomplish the same tasks accomplished easily and cleanly in scripting languages and frameworks.

Although I feel gaining PHP5 certification would open me to more opportunities in that job market, I do not believe it will make me a better programmer at this point. I may simply focus on creating more tutorials and release some of my own custom framework designs to show my knowledge in the field. I’d hope that good potential employers would look past my resume to my portfolio.

¹ Certification Exam – Very Disappointed – PHP Certification – PHP support & tips – Zend.com. Retrieved from Google Cache on March 4, 2010. Original cached December 30, 2009

² PHP certification views? Michael Kimsal. May 27, 2008. Retrieved on March 4, 2009.

This project uses a lot of different open source libraries including Smarty, twitter-async, jQuery, Blueprint, TableSorter. The main engine is built upon parts of a PHP framework I started developing almost a year ago. I stopped development on the framework and started utilizing WordPress for all my content-based websites.

I may go back and start development on my framework again and experiment with new web application ideas that don’t fit into the standard content model of a blog.

The following tutorial goes through how to install and configure the free and open source web statistics program Awstats to be used with Media Temple’s grid-servers to provide analytic data from the Apache logs per each individual domain.

This tutorial assumes that you have a Media Temple grid-server account. It also assumes you have some basic knowledge about using Linux including shell commands, shell scripting and editing files. You should also know how to use SSH and work with .htaccess files. If any of this is unfamiliar to you, you may want to start with installing Linux on a local computer or virtual machine and practicing basic shell commands and setting up a basic web server before continuing with this tutorial.

In a typical grid-server account, there is a symbolic link created in the primary user’s home directory called domains which points to a data directory containing all the individual domain’s document roots. For this tutorial, we’ll create a a WebApps folder within the domains folder, install awstats into that folder and configure it for each domain. We’ll then create the data directories, create symbolic links from each website to awstats, protect those directories and setup a cron script to updated the stats automatically. Let’s get started.

First you’ll want to SSH into your grid-server. Once there, create the directories needed for awstats, download the latest source code and then untar it. Finally, create a symbolic link to aid in easier upgrades further down the road.

ssh example.com
mkdir domains/WebApps
cd domains/WebApps
wget http://downloads.sourceforge.net/project/awstats/AWStats/6.95/awstats-6.95.tar.gz?use_mirror=softlayer
tar xvfz awstats-6.95.tar.gz
ln -s awstats-6.95 awstats

Awstatus uses it’s own file format to store statistics in. These files are created and updated by running Awstats against a set of Apache log files. Next, we’ll create the data directory and setup the configuration files. We’ll also move all the other web support files into into the cgi-bin directory, which will be necessary for correctly displaying the web statistics later.

cd awstats
mkdir data
cd wwwroot/cgi-bin/
mv ../classes ../css ../icon ../js .

Within this cgi-bin directory is the actual awstats.pl script. It’s used both to display the HTML stats page via a CGI interface and also to update the data files from the command line. Here is where you will create configuration files for each website.

The official installation documentation suggests using the command awstats_configure.pl located in the tools directory. It generates a configuration file and places it in the config directory, similar to the awstats.model.conf file located in the cgi-bin directory. It doesn’t matter which file you use as a template, but you must make sure the following attributes are set:

LogFile="/home/XXXXX/users/.home/scripts/mt_logfix.py example.com |"
LogType=W
LogFormat=1
LogSeparator=" "
SiteDomain="example.com"
HostAliases="example.com www.example.com"
DNSLookup=1
DirData="/home/XXXXX/domains/WebApps/awstats/data"
DirIcons="/awstats/icon"
DNSStaticCacheFile="dnscache.txt"
DNSLastUpdateCacheFile="dnscachelastupdate.txt"

Be sure to replace the XXXXX in this and other code examples with the number which leads to your home and data directories. The example.com should be replaced with your domain. The HostAliases should include any subdomains that you want included in this set of statistics. The other settings in the configuration file can be adjusted to your liking as well.

The file in the above example should be named awstats.example.com.conf and placed in the cgi-bin directory. Additional domains will need their own configuration file, placed in the same directory with the same naming convention.

You’ll notice in the above configuration file, there is a reference to mt_logfix.py. The trouble with the standard Apache logs for the way Media Temple sets up their grid-server is that it places the domain name for the virtual host in the actual path of each log entry. For example:

xxx.xxx.xxx.xxx - - [29/Oct/2009:12:08:24 -0700] "GET /example.com/somepage.php HTTP/1.1" 200 1232 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google
.com/bot.html)"

The following python script extracts all the log entries for one particular website and removes the name of the virtual host from path. It uses the logresolvemerge.pl script that comes with Awstats to combined all the log files and process them sequentially. Awstats is designed so that it can process the same log file(s) multiple times by skipping over all the previous recorded entries, so you don’t need to keep track of which files you’ve processed. I’ve placed this file in a directory called scripts under my home, but you can place this anywhere you like, just be sure to adjust the configuration file(s) you created previously accordingly. Also be sure to set the two initial variables in the script to the correct path.

#!/usr/bin/env python

mergecmd = '/home/XXXXX/domains/WebApps/awstats/tools/logresolvemerge.pl'
logfiles = '/home/XXXXX/logs/access_log*'

import sys
import re
from subprocess import Popen, PIPE, STDOUT

#check for domain argument
if len(sys.argv) != 2:
  print '\nMT LogFix\n\nUsage: ./mt_logfix.py <domain>\n\n\tSumit Khanna - PenguinDreams.org\n'
  sys.exit()

domain = sys.argv[1]

#open logresolver
pipe = Popen([mergecmd,logfiles],stdout=PIPE,stdin=PIPE,stderr=STDOUT)


#filter for domain
for line in pipe.stdout:
   if re.search('GET /'+domain+'/',line) :
     print re.sub('GET /[a-z.]*/', 'GET /', line).strip()

For this to work, logging must be enabled for your grid-server. To do this, login to the Media Temple account center, click on your server and go to the section titled “Report Settings and Logs.” Be sure the that the “Keep Logs For” drop down has a value in it. I would suggest keeping it at the maximum unless you start to run out of space. At a minimum, you must keep this setting higher than the cron settings you’ll establish for the batch script later so that no log data is lost.

Next, setup the batch process to run Awstats at regular intervals and update our database files with the latest Apache logs. I created a simple script and placed it in the scripts directory located in my home directory. Be sure to adjust the path and domains.

#!/bin/sh

SCMD="/home/XXXXX/domains/WebApps/awstats/wwwroot/cgi-bin/awstats.pl" 

domains="example.com example.net example.org"

for i in $domains; do
  $SCMD -config=$i
done

It would be nice if we could just schedule the above script to run through cron. Unfortunately running the above script via cron on Media Temple produces the following results

Create/Update database for config "/home/XXXXX/domains/WebApps/awstats/wwwroot/cgi-bin/awstats.example.com.conf" by AWStats version 6.9 (build 1.925)
From data in log file "/home/XXXXX/users/.home/scripts/mt_logfix.py example.com |"...
Phase 1 : First bypass old records, searching new record...
Searching new records from beginning of log file...
Jumped lines in file: 0
Parsed lines in file: 0
 Found 0 dropped records,
 Found 0 corrupted records,
 Found 0 old records,
 Found 0 new qualified records.

None of the log files are read and none of the data files are written. It would seem like a permission issue, however the cron scripts should run under the same permissions as your user account. I wrote the following script and ran it both through cron and through SSH.

#!/bin/sh

echo -e -n "Who:\t"
whoami
echo -e "Home:\t$HOME"
echo -e "UID:\t$UID"
echo -e "EUID:\t$EUID"
echo -e "User:\t$USER"

Here are the results running the script via SSH:

Who:    example.com
Home:   /home/XXXXX/users/.home
UID:    4----9
EUID:   4----9
User:   example.com

And here is the output of the same script when run via cron:

Who:	example.com
Home:	/home/XXXXX
UID:	4----9
EUID:	4----9
User:	

It’s important to note that although the username is the same, the home directory when running the script via cron is incorrect. Furthermore, the $USER shell variable is not established. Clearly the cron shell is running in a very different mode than a standard login shell, possibly due to security restrictions or certain patches for a hardened environment. A less than elegant solution is to establish a set of keys for SSH and have the cron script ssh to the local host and run the generation script. To do this, create a script called generate_stats_ssh_fix.sh and place it in the scripts directory you created earlier.

#!/bin/sh

echo "Using SSH Hack:"
ssh -F /home/XXXXX/users/.home/.ssh/config -i /home/XXXXX/users/.home/.ssh/id_dsa localhost "/home/XXXXX/users/.home/scripts/generate_stats.sh"

In order for this script to work in cron, ssh must be able to login to the local machine without needing a password. Therefore it is necessary to create a DSA authorization key set and add the generated public key into the local authorized key store. If you are all ready familiar with SSH keys, this should make sense to you. If you don’t and have never setup any of the configuration files within the .ssh directory on your webserver before, then simply run the following:

ssh-keygen -t dsa #press enter for all questions which will select the default answers
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh localhost #type yes when prompted to add the key fingerprint

If you don’t understand what is going on here, please read up on SSH keys and authorization to fully understand the security implications presented. Normally you would create a set of SSH keys and add the public key to a remote server’s authorized key store in order to SSH to it without needing a password. Here we are essentially doing the same thing, except we are connection to our local machine. We preform the connection once to add the local machines fingerprint into the known_hosts file.

Next, we take this SSH wrapper script and create a cron task out of it. In the account center, navigate to the cron section and add the generate_stats_ssh_fix.sh script as a new job. I would suggest setting up the notification e-mail initially to make sure the task is running correctly. After you’re sure it’s going, you can remove the e-mail from this field. The schedule time is up to you. I’d suggest running it once per day.

Now that we have have log collection taken care of, we need to work on displaying the results. We can do this with the awstats.pl script. Let’s start by creating a .htaccess file within the /home/XXXXX/users/.home/domains/WebApps/awstats/wwwroot/cgi-bin directory:

AddHandler cgi-script .pl
Options +ExecCGI

AuthName "Restricted Area"
AuthType Basic
AuthUserFile /home/XXXXX/domains/WebApps/awstats/wwwroot/cgi-bin/.htpasswd
AuthGroupFile /dev/null
<LIMIT POST GET>
  require user someuser
</LIMIT>

With this .htaccess file, we’re doing two things. The first is granting this directory permissions to execute CGI scripts. The second is restricting access based on an .htpasswd file. In the above example I place the password file within the same directory as the CGI script, but for security you may want to move it to some other location out of your web accessible folders.

Information on creating the actual .htpasswd file can be found in the Password Protecting Directories article in Media Temple’s knowledge base.

Finally, we can create the symbolic links to our statistics pages.

cd /home/XXXXX/domains/example.com/html/
ln -s /home/XXXXX/domains/WebApps/awstats/wwwroot/cgi-bin/ ./awstats

You can create this link for each domain you’ve created configuration files for. Once created, you should be able to navigate to http://example.com/awstats/awstats.pl, enter in the username and password you established earlier with the .htaccess and .htpasswd files, and view the analytics for that particular domain.

Congratulations. You should now have Awstats setup on your Media Temple grid-server account. To add new domains, simply create a new configuration file and update the generate_stats.sh script. To upgrade to a newer version of Awstats, untar the newer version in the WebApps directory, copy over the data and configuration files and adjust the symbolic link. Although the example presented is for Awstats, the mt_logfix.py can be used to get the Apache web server logs in the correct format for many other log file analysis based analyzers.

Awstats is a decent free analytics tool. Although not as powerful as commercial solutions, it is much better than the horrid Urchin tool provided by Media Temple. Most other web hosting providers have an installation of Awstats in their basic packages. Although the process for installing it yourself with Media Temple is a bit cumbersome, it is worth it. Media Temple does have awstats installed on their dedicated virtual (dv) solutions, and hopefully they will eventually add this application into their grid-server account center, removing the need for this manual installation.

A mixture of emotions came across as I realized I wasted an entire day on a software decoder when a hardware solution was available for $30 to $40. The software solution was fairly disappointing, so I decided to try one of these new cards, an nVidia GeForce 8500 GT, to see if it provided a better solution. It took some work with my setup, but the results were worth it.

The new interface from nVidia for hardware video decoding on Linux is known as the Video Decode and Presentation API for UNIX or VDPAU. It’s been available officially since the release of the 180.22 nVidia Linux drivers in January of 2009, but has been available in beta drives since last November starting with 180.06. Drivers which support VDPAU will install special header files in /usr/include/vdpau/ which are required to compile media players with VDPAU support.

The installation process is fairly straightforward. The patches for mplayer can be found on the nVidia FTP site. The tarball contains not only the patch sets, but also a build script named checkout-patch-build.sh. As the name suggestes, the script checks out all the appropriate revisions of mplayer and its dependencies (libavcodec, libdvdread, etc) from a subversion repository, applies nVidia specific patches and compiles mplayer. The README.txt file in the package also suggests several free sample videos that can be used to test the new mplayer build.

To prevent conflicts with the copy of mplayer that may come with the Linux distribution and that is typically controlled by a package manager, I suggest manually installing the VDPAU version of mplayer by itself with its own name. The following are the commands I ran for both compiling and installation. Be sure to replace the tarball with the latest version available on the ftp site.

wget ftp://download.nvidia.com/XFree86/vdpau/mplayer-vdpau-3402051.tar.bz2
tar xvfj mplayer-vdpau-3402051.tar.bz2
cd mplayer-vdpau-3402051
./checkout-patch-build.sh
su 
cp ./mplayer /usr/local/bin/mplayer-vdpau

As stated earlier, the README.txt providers several sample files available to download to test out the decoder. I’d also suggest the following options when playing H.264 files:

mplayer-vdpau -vo vdpau -vc ffh264vdpau -framedrop -nocorrect-pts <filename>

Both the -framedrop and -nocorrect-pts will help provider for smoother playback and may be helpful on older CPUs. On machines that can handle the playback, they shouldn’t degrade the video quality.

You may get an error such as the following along with no video window being displayed:

Cannot find codec matching selected -vo and video format 0x31637661

This is most likely the result of having a ~/.mplayer/codecs.conf. Current versions of mplayer do not require a codecs.conf as one is compiled into the player itself. A stray configuration file will override the built-in defaults and remove the nVidia provided codecs. You may see this error if you had followed my previous tutorial on implementing software H.264 decoding.

Fixing this issue involves either deleting the file, renaming the file or combining the codecs.conf in your home directory with the patched one located in nVidia’s mplayer-vdpau build directory.

Another issue can arise if you have a multi-seat system; that is a system with multiple X servers each with their own monitor, keyboard and mouse configuration. In such an environment, it is essential that the X server you want to use the VDPAU card with comes up first. The video card does not need to be in the first PCI-E slot. Mine resides in the second:

 # lspci | grep nVidia
01:00.0 VGA compatible controller: nVidia Corporation NV42 [GeForce 6800 XT] (rev a2)
02:00.0 VGA compatible controller: nVidia Corporation GeForce 8500 GT (rev a1)

However in the greeter configuration file, the server that holds the GeForce 8500 card I want to use for decoding must be listed first. In my case, I use gdm on Gentoo Linux which uses /etc/X11/gdm/custom.conf to define server startup:

[servers]
0=Plasma
1=Console

[server-Plasma]
name=Plasma server
command=/usr/X11R6/bin/X -layout Plasma -novtswitch -sharevts -isolateDevice PCI:2:0:0
flexiable=true

[server-Console]
name=Console server
command=/usr/X11R6/bin/X  -layout Console -novtswitch -sharevts -isolateDevice PCI:1:0:0
flexiable=true

When I got the VDPAU mplayer up and running, I was very impressed by the results. Some video still jumped and sputtered in placed, but nothing too distracting from the video considering the high bitrates and quality. There was also occasional screen tearing where one frame partially overlapped a previous one. I’ve read turning off the X composite extension may help alleviate this problem although I haven’t tried this yet.

I’m glad nVidia is realizing the importance of capitalizing on the Linux market. ATI is trying to catch up with their own RadeonHD drivers. Linux support will be essential as it’s an ideal platform for embedded systems such as nVidia’s Ion. It could open up an entire realm of possibilities for small, affordable, set-top box and media PC solutions.

Unfortunately, the installation documents for coreavc-for-linux were months old, out-of-date and had few corrections for new bugs. The following are some of the common errors I found as well as the solutions I’ve found to get the trail version of the CoreAVC codec working on my Gentoo Linux system.

Installation

To start the installation, I created a new working directory within my home directory and checked out both the latest versions of mplayer and coreavc-for-linux (revision 82 for coreavc-for-linux and revision 28381 for mplayer). I then ran the configuration script for mplayer, applied the direct show patch from the coravc-for-linux project and proceeded to build mplayer:

mkdir coreavc
cd coreavc
svn checkout http://coreavc-for-linux.googlecode.com/svn/trunk/ coreavc-for-linux
svn checkout svn://svn.mplayerhq.hu/mplayer/trunk mplayer
cd mplayer
./configure
patch -p0 < ../coreavc-for-linux/mplayer/dshowserver.patch
make

At this point the new version of mplayer is ready. Rather than use the make install command, I wanted to keep this version of mplayer totally separate. So I copied the two executables I wanted directly to /usr/local/bin with new names like so:

su
cp mplayer /usr/local/bin/mplayer-coreavc
cp mencoder /usr/local/bin/mencoder-coreavc

At this point we have mplayer installed, but not the codec itself or the directshow server. We’ll start with the codec first. You can signup for a free trial on the official CoreAVC website. I would suggest the trial version first to make sure the codec works on your system. The download contains an executable installer, so you can either install it onto a windows system or install it using wine. After the install, copy the installed codec to the /usr/lib/win32 directory. The following example assumes you used wine. Your directory names may be different.

cd ~
su
cp ~/.wine/drive_c/Program\ Files/CoreCodec/CoreAVC\ 14\ Day\ Trial\ Edition/CoreAVCDecoder.ax /usr/lib/win32/

Before we go further we’ll install the directshow server. The wiki has instructions for compiling the server or using a precompiled binary. If you’re like me, you’re running a 64-bit system and can therefor not compile the binary from source. Doing so will give you a pretty massive compile error with several warning: ‘stdcall‘ attribute ignored messages from interfaces.h.

If you have a 32-bit linux system available, either real or a VM, or if you have somehow mastered the magic of cross compiling (I gave up on this years ago when I tried building a cross compiler for PPC), you can probably make this binary yourself. If you used the latest precompiled binary in the downloads section (r63 for Gentoo at the time of this writing) along with the 1.8.5 release of CoreAVC, you will get the following fault when trying to run it:

/usr/local/bin/dshowserver -c CoreAVCDecoder.ax -s 1280x720 -g 09571a4b-f1fe-4c60-9760de6d310c7c31 -b 12 -f 0x34363248 -o 0x30323449
No id specified, assuming test mode
Opening device
len: 992
ProductVersion: 1.8.5
Called unk_LoadImageA
Segmentation fault

This is due to a know issue with the 1.8.5 release of CoreAVC adding a dialog box in the initialization which is documented in Issue #62 and in a posting on Acmelabs’ Blog. In a later post, Acmelab gives his readers a precompiled version of release 77 which I’ve made a copy of for download: dshowserver-ia32-r77-acme.tar.bz2

Untar this file and copy the dshowserver/dshowserver and loader/registercodec binaries to your /usr/local/bin.

Now that you have the directshow server installed, you’ll need to add the trial serial number into mplayer’s virtual windows registry. The following command is known to work with version 1.7.0 and 1.8.5 of CoreAVC. Additional registry options can be found on the wiki

export REGISTRY=$HOME/.mplayer/registry32
registercodec -r $REGISTRY -k 
"HKLM\\Software\\CoreCodec\\CoreAVC Trial\\Serial" -v "55555-55555-CORE-55555-55555"

If you’re using the full version, you’d replace the word Trial with Pro. At this point you should be able to run the following command to test the codec:

/usr/local/bin/dshowserver -c CoreAVCDecoder.ax -s 1280x720 -g 09571a4b-f1fe-4c60-9760de6d310c7c31 -b 12 -f 0x34363248 -o 0x30323449

You should get the following response:

No id specified, assuming test mode
Opening device
Called unk_IsDebuggerPresent
len: 992
ProductVersion: 1.8.5
Decoder supports the following YUV formats: YUY2 UYVY YV12 I420 
Decoder is capable of YUV output (flags 0x2b)
Setting fmt
Starting
Initialization is complete

You may get a serial number issue here. If you do, go back and check your registry key. Now the only thing left is the ~/.mplayer/codec.conf

Using the 28381 build of mplayer, I constantly got the following error using the default instructions for the codec.conf:

==========================================================================
Forced video codec: coreserve
Cannot find codec matching selected -vo and video format 0x3267706D.
Read DOCS/HTML/en/codecs.html!
==========================================================================

Also, using the default instructions on a current built of mplayer can also lead to seeing this error in the mplayer output:

This codecs.conf is too old and incompatible with this MPlayer release!

This is because newer version of the codecs.conf must have a release line at the top of the file to prevent conflicts with other codec configurations as stated in issue #61

After playing with my configuration file a lot and with no results, I eventually found a configuration file in comment 28 on issue 14 that worked. I’ve made the configuration file available here as well: codecs.conf

Playback

When running mplayer-coreavc, you made need the following options to force mplayer to use coreavc and also to have smooth playback. Obviously, replace the example file I have with an HD movie you have on your file system.

/usr/local/bin/mplayer-coreavc -vc coreserver -framedrop -nocorrect-pts  /media/files/movies_HD/Wall-E_720p_DTS_BluRay.mkv

If you’re like me, you don’t use a frontend to mplayer and prefer to have the terminal output. If so, you can create these little scripts that will pop-up mplayer in a terminal and then use them from the “open with” menu in your favorite file manager:

/usr/local/bin/coreavc-ac3

#!/bin/sh
gnome-terminal -e "mplayer-coreavc -vc coreserver -framedrop -nocorrect-pts -ac hwac3 \"$1\""

/usr/local/bin/coreavc-dts

#!/bin/sh
gnome-terminal -e "mplayer-coreavc -vc coreserver -framedrop -nocorrect-pts -ac hwdts \"$1\""

The -ac options for AC3 or DTS allow for media that has embedded multi-channel audio in either Dolby Digital (AC3) or Sony (DTS) format to go through the digital output of your sound card and directly to a 5.1 surround sound receiver (if you have both a digital out on your sound card and such a receiver).

Conclusions

I made this guide because many of the guides I’ve seen were out of date and the steps I had to undertake spanned multiple websites, blogs and wikis. Keep in mind that if you’re reading this six months from now, this guide is probably out of date itself and there are probably never versions of CoreAVC, coreavc-for-linux and mplayer, all with new an fun interesting bugs that will take you an hour to figure out. If you find any other helpful information, please post a comment.

As far as the actual performance of the CoreAVC codec, it is considerably faster than ffmpeg. There is somewhat of a lag or frame delay effect I see with movies which I don’t see in ffmpeg. I’m guessing this is coming from the directshow server. However with 1080p content, the video decodes much faster and is very smooth without constant bumps and clicks, even when going through the considerable amounts of translation required to run a windows 32-bit native codec in a 64-bit Linux environment.

There are still a couple of artifacts I’ve found and distorted/garbage video during fast motion scenes. This may be due to the codec specific registry settings which I’ll need to play with, or it may be I’m exceeding the limitations of my processor. Your own mileage may vary.

The Battery

Looking at the simplest upgrade first: the battery. There is an excellent free utility called Coconut Battery which allows you to not only see the current battery capacity, but look at the original capacity and the age of the laptop. Newer versions of the application allow saving data points to track the capacity and degradation of the battery.

Lithium Ion (Li-Ion) batteries don’t suffer from the same memory effect as other types of rechargeable such Nickel Cadmium (NiCd) or Nickel-metal Hydride (NiMH). However, as can clearly been see by the screenshots above, Li-Ion batteries still degrade naturally over a period of time. After two and a half years, my laptop battery could barely hold over a third of its original capacity. In addition, it took much longer to charge as well.

Since the new version of Coconut Battery can save data points, I’m hoping by the time I retire this laptop that I’ll be able to be able to chart the capacity pattern of this new battery. I’m predicting it will be a natural logarithmic regression.

One additional thing I learned about the MacBook battery. It can not be removed while the laptop is on, even if it is plugged in. Unlike my old Dell Laptop which had two expansion slots providing for the ability to hot swap two batteries, removing the MacBook battery while hot caused the MacBook to crash and it had to be rebooted.

The Hard Drive

The previous hard drive was the standard 80GB that came with the laptop. I had found myself deleting unused applications and moving media from my laptop just so I could log and transfer video clips for editing. I eventually had to use a 40GB external USB drive as my video scratch disk. With dropping prices for SATA internal drives, I decided to upgrade to a 500GB, giving me more than enough room for all my applications and media clips. In addition, a low cost external SATA to USB enclosure allowed me to still utilize my old 80GB hard drive as an external disk.

Replacing the hard drive was about as simple as replacing the memory. They are both behind a plate on the side of the battery compartment. The hard drive did require a hex-screwdriver, which I did not expect.

The Operating System

One of the biggest reason I wanted to upgrade to MacOS 10.5 was the addition of “spaces” which is basically another term for virtual desktops. In 10.4, third party software such as Desktop Manager had to be used to add in support for virtual desktops. They were buggy at best and different applications would react different to being on a virtual desktop.

Virtual desktops aren’t anything new or innovative. They’ve been available naively on UNIX/Linux systems for over a decade and as third party addons to Windows and Mac for years. The advantage of having “spaces” as part of the operating system layer itself is that newer version of applications will develop around and for the “spaces” rather than third party applications having to accommodate for other programs.

In addition OS X 10.5 does seem to add some speed increases. The visual additions and other small features are nice as well.

Conclusions

The upgrades I made were primarily to make the laptop useful again as a laptop and not just an overpriced desktop. However I did notice speed improvements when playing HD video within editing software. I’m not sure whether to attribute this to a higher efficiency from OS X 10.5 or if the larger hard drive allows the OS to have more swap space to play with (or both). In any case, these small upgrades, which cost under $200, seem to have given me some more life and years out of my laptop.