A Database for node.js Projects

LevelDb is a key/value store written three or four years ago by engineers at Google.  Since then it has been adopted by many projects including Google Chrome, Bitcoin, Apache ActiveMQ, and others.  It was introduced to the node community as levelup/leveldown in July of 2012.  Since then there have been over 13K downloads and many extensions.

When I first started experimenting with LevelDb I was pleased with it's speed and ease of use but also surprised as how easy it was to configure. Simply give it a file name and you are good to go.  Or if you want an in-memory database, use memdown as the database backing store.

SimpleNodeDb is an extension to LevelDb that adds query, insert, update, delete and backup/restore capabilities.  The keys are domain specific and data values are stored as objects with JSON encoding.

The project is published to the npm registry and can be installed using this:

npm install simple-node-db

Here is a more complete description as pulled from the published npm page:

A database implementation on top of levelup, leveldown, and memdown. SimpleNodeDb leverages the document store aspects of level up to provide a data-model centric implementation.

Models are stored as JSON strings with domain-scoped keys. For example a user data model's key of '12345' would have an associated domain key of 'user:12345'. So querying for users as opposed to orders or inventory parts is as easy as including records where keys begin with 'user:'.

Automatic model attributes include dateCreated, lastUpdated and version. The version attribute is used to enforce optimistic locking.

Typically SimpleNodeDb is well suited for small to medium datasets (less than 100K rows) or data stores that don't require complex querying. It also provides robust caching when used as an in-memory data store. To support more than 100K rows you should probably create alternate indexing schemes or stick with redis, mongo, or a traditional SQL database.

The project includes a full suite of unit tests and examples.  If you get a chance to try it out, let me know what you think.  For more info on leveldb, check out this video from Richard Astbury.

Javascript Asynchronous Operations

Javascript is single threaded.  That's good.  But to take advantage of our ever increasing multi-processor world, we need to adopt functional solutions.  Thats where asynchronous operations come in.

Javascript applications that run in the browser or in node have asynchronous events occurring all the time.  This is especially true in node where most actions return results through asynchronous callbacks rather than direct return values.  This takes some time to get used to.

Consider the following recursive function/snippet:

var list = getList(); 
var nextListItem = function() {
    var item = list.pop();
 
    if (item) {
        // process the item...
        nextListItem();
    }
};
nextListItem();

The recursive list item processor will work fine if the list is a reasonable size. But, if the list is huge, then one has to worry about the size of call stack. With every iteration it grows by one call, so it may exceed memory limits and fail (in a very bad way) if there are too many iterations.

Now consider this small change:

var list = getList();
var nextListItem = function() {
    var item = list.pop();
 
    if (item) {
        setTimeout(function() {
            // process the item...
            nextListItem();
        }, 0);
    }
};
nextListItem();

The stack over-flow problem is now eliminated. Why? Because javascript's event system handles the recursion–not the call stack. When nextListItem runs, if 'item' is true (not null) then the timeout function is defined and pushed to the event queue, then the function exits–and the call stack is now clear. When the event queue runs it's timed' out event, the item is processed and nextListItem is recursively called.

Again, the method is processed from start to finish without a direct recursive call, so the call-stack is again clear–regardless of the number of iterations.

Now consider this example:

var userList,
    cleverProgrammerBonus = 10000;

var userUpdateCallback = function(err) {

    if (err) return updateCompleteCallback( err );
    nextUser();
};

var nextUser = function() {
    var user = userList.pop();
    if (user) {
        user.pay += cleverProgrammerBonus;
        dao.updateUser( user, userUpdateCallback );
    } else {

        updateCompleteCallback();

    }
};

var userFetchCallback = function(err, list) {
    if (err) return updateCompleteCallback( err );
    userList = list;
    nextUser();
};

dao.fetchUsers( userFetchCallback );

As with most functional, asynchronous solutions, it's best to start at the bottom–the dao fetch of users. The fetch callback sets the user list then invokes the recursive nextUser() to get the iteration started. Then the user is updated asynchronously, and when it completes updateUpdateCallback is invoked. This callback continues the recursion by invoking nextUser. This is repeated until the list is empty.

There isn't a need for setInterval ( or nextTick() ) because the dao calls to update the user are asynchronous–always invoked from the event queue, not the call stack. So the call stack remains static no matter how many users are in the list.

Its also important to realize that the function dao.updateUser doesn't really start working until nextUser has exited. It's queued to do some work, but not at the direction of the call stack–just the event queue.

Asynchronous event driven operations.  Another reason to love coding in javascript.

Javascript Coding Standards

I love javascript. I love how it's flexible enough to be as expressive as you want.  The power of closures used in the traditional way or simply as function pointers like in the old C days.

Javascript's flexible nature enables an infinite combination of styles.  From prototypical, IIFE, to classical where functions are actually called "classes" when no actual classes exist.

This flexibility can lead to endless discussions of "what's best" or "what's most efficient" and worse can lead to project code that can resemble a big steamy bowl of pasta with brownish marinara.  As a team leader, the last thing you want is to have to code walk dozens of coding styles.  As a coder and team member the last thing you what to do is traverse hundreds of lines of code that has bad indentation or horrible style.

So, as a manager, you need to adopt a strict coding standard and ruthlessly enforce it.

Coding Standard Objectives

Here is a short bullet list.  The standards should...

• match the team's current experience and talent,
• should closely follow at least one major published standard,
• should have justifications for each decision,
• and include team consensus.

Team Experience

Javascript isn't new, but it's new to many programmers.  If your team has years of java and c++ experience, the wide-open flexibility of javascript will come as a frustrating surprise.  If your team has a ruby influence, then you have a leg up.  Understanding closures, dynamic typing, etc will ease the transition to production ready javascript.  If your team is primarily PHP based, then they know a bit about javascript but probably don't understand its full capability.

For the Java Team

So lets begin with the java programming team--what standards would suite them best?  The choice is easy--use a classical implementation with factory generated components that implement IoC through parameter injection.  Think of it this way, every file is a class defined by a named function variable (the function is still technically anonymous).  Its injection is always through construction (think ruby opts), and always factory built.  Files are organized as pseudo-package folders that may (or may not) have package namespaces.

The coding style is very similar to the java specs.  K&R formatting, title cased classes, mixed case variables, no underscores.  Complete separation of concerns, i.e. MVC plus services, delegates and factories.  And, unit tests--either TDD or BDD for everything.  And not just the normal tests, but also tests to find bugs that the compiler would normally find.  (mocha is probably the best test platform for both client and server javascript today).

Published Standards

Douglas Crockford.  JSLint.  Javascript, the good parts.  If you can only read one javascript book, please read "the good parts".   Coding conclusions aren't drawn on what is best--just what the alternatives are.  But, he does describe what he feels are some best practices for coding style, indentation, curly braces, etc.  And, if you have the time, watch his video series on javascript.

For good published standards, start here:

• google standards.
• yahoo standards.

Standards you probably want stay away from:

• if you are writing applications, the don't follow these standards; utility modules maybe, but...
• M$ standards--known only to a few Seattle east-siders.

Our Standards Document

Here is a link to our current javascript coding standards.  For the most part, this is for client applications, but most parts apply to node/server projects as well.

Conclusions

My best advise is to find a good published standard, modify it for your team's abilities, then stick with it.  You will need to add rules as time goes on--because coders will always find differing ways to interpret the rules.  So be prepared for additions--not so much changes, just clarifications.

Happy coding...

Easy Mac Installs--MySQL & NetBeans

I just went through two very easy installs on my new mac; the first, MySQL 5.1.5. A few years ago, when I installed on my old mac it was the typical tar ball install that linux users are familiar with. Not bad, especially if you have a canned script to remind you of all the steps.

But now there are two dmg package files that do most of the work for you. And, it installs in /usr/local with a dynamic link to mysql--just like it's supposed to be. The only steps left were to remove the anonymous users, supply the root and admin passwords and reboot to start the service. Easy.

Next was NetBeans 6.9. I have been using 6.8 for a few projects, and have loved the support for groovy and especially grails, but I have held off upgrading while in full development mode. But now, with a new grails project in it's infant stages, I decided to make the move.

Installation was easy. I did have to manually replace the 6.8 item with 6.9 on my doc, but other than that all went easy. It found my current projects and I was able to run unit tests from within the IDE. Once again, easy.

Happy coding...

Grails 1.3.4, Groovy 7 and NetBeans 6.8

After about a year of doing nothing but Adobe Action Script projects, I'm finally back to a java/groovy/grails project. So, I downloaded the latest grails and fired up my version of NetBeans (6.8) and generated some code. Here's what I discovered...

First, NetBeans is very groovy/grails friendly. I was able to generate the project inside the IDE. I then dropped to the command line to generate domain classes and tests. I was able to run the full test suite inside the IDE, but I still prefer the command line.

I noticed that the generated code doesn't include an ant script, but just as well. I'm so sick of the XML crap, and the grails command line does everything you need, so...

Next step is to generate controllers, and maybe a few views, but I usually use grails as a JSON or XML server with a Flex/Action Script UI. I'm also creating all the unit tests as I go and running them from inside the IDE and from the command line.

Next post will discuss the controllers and any problems related to getting the application into production. So far everything works as advertised--better than previous versions, and very productive. My new best friend...

UMLet UML Diagraming

Just had to comment on a very useful new(ish) project for editing UML documents called UMLet. No frills or bs* just a good productivity tool in the open source space. Human readable file structure to enable old timers like me to hack in attributes with vi.

Converts your drawings to png, jpg, eps, pdf, svg, etc from inside the UI or even from the command line (a real programmer's tool).

It comes in two flavors 1) eclipse plugin, and 2) standalone Swing application. They have a few skins that help ease the swing pain but who cares about the L&F, I just want a tool that works with me instead of against me.

So, thanks to the UMLet team for a great product!

HTML5, CSS3, Apple, Google and the W3C

About a year ago, I remember responding to a question from a colleague concerning Flex in comparison to HTML5 by saying that HTML5 was not ready yet. For the most part, I still hold that opinion but with additional qualifications.

We are currently bidding on a job that requires iPad as the target device and another that needs a private label application that will work on the desktop, iPhone and eventually Android and Blackberry. After logging many hours researching objective-c and HTML5/CSS3 my first thoughts were that all development needed to be done in objective-c. But, after viewing some of the other videos from Apple, MIT, and Stanford, and doing some small tests, I think that most applications for the iPad/iPhone can be implemented completely in HTML5/CSS3. So here is my revised/updated response...

If you have complete control over the target web browser (i.e., webkit/khtml based) then HTML5 and CSS3 can't be beat. Using the 'canvas' tag/object enables drawing anything that you can draw with flex/flash and also includes drawing text (not possible with flex). Using CSS3 enables all the transitions that are currently available in flex including easing methods. Using WebSocket, provides peer-to-peer real time communications. So if your target includes Safari, iPhone, iPad, iPod/Touch, Google Chrome, Opera, Firefox, as well as the Android, Sprint and Blackberry browsers, the new technology is ready today.

The missing piece is IE, currently at 70% of desktop browser installations. An application that needs to run only in a standard client browser is still better off using Flex (95% penetration) until Microsoft complies with the latest W3C specs. But, the flash player will probably never make it to small devices (it's a hog), so if the requirement is to run on desktop and phone, especially with touch gestures, HTML5/CSS3 is probably a better choice. The down side is waiting for Microsoft to catch up with the fully developed world and hoping they will fully comply with the W3C.