Sunday, May 25, 2014

A Database for node.js Projects

LevelDb is a key/value store written three or four years ago by engineers at Google.  Since then it has been adopted by many projects including Google Chrome, Bitcoin, Apache ActiveMQ, and others.  It was introduced to the node community as levelup/leveldown in July of 2012.  Since then there have been over 13K downloads and many extensions.

When I first started experimenting with LevelDb I was pleased with it's speed and ease of use but also surprised as how easy it was to configure. Simply give it a file name and you are good to go.  Or if you want an in-memory database, use memdown as the database backing store.

SimpleNodeDb is an extension to LevelDb that adds query, insert, update, delete and backup/restore capabilities.  The keys are domain specific and data values are stored as objects with JSON encoding.

The project is published to the npm registry and can be installed using this:
npm install simple-node-db
Here is a more complete description as pulled from the published npm page:

A database implementation on top of levelup, leveldown, and memdown. SimpleNodeDb leverages the document store aspects of level up to provide a data-model centric implementation.
Models are stored as JSON strings with domain-scoped keys. For example a user data model's key of '12345' would have an associated domain key of 'user:12345'. So querying for users as opposed to orders or inventory parts is as easy as including records where keys begin with 'user:'.
Automatic model attributes include dateCreated, lastUpdated and version. The version attribute is used to enforce optimistic locking.
Typically SimpleNodeDb is well suited for small to medium datasets (less than 100K rows) or data stores that don't require complex querying. It also provides robust caching when used as an in-memory data store. To support more than 100K rows you should probably create alternate indexing schemes or stick with redis, mongo, or a traditional SQL database.

The project includes a full suite of unit tests and examples.  If you get a chance to try it out, let me know what you think.  For more info on leveldb, check out this video from Richard Astbury.

Wednesday, January 22, 2014

Javascript Asynchronous Operations

Javascript is single threaded.  That's good.  But to take advantage of our ever increasing multi-processor world, we need to adopt functional solutions.  Thats where asynchronous operations come in.

Javascript applications that run in the browser or in node have asynchronous events occurring all the time.  This is especially true in node where most actions return results through asynchronous callbacks rather than direct return values.  This takes some time to get used to.

Consider the following recursive function/snippet:

var list = getList(); 
var nextListItem = function() {
    var item = list.pop();
    if (item) {
        // process the item...

The recursive list item processor will work fine if the list is a reasonable size.  But, if the list is huge, then one has to worry about the size of call stack.  With every iteration it grows by one call, so it may exceed memory limits and fail (in a very bad way) if there are too many iterations.

Now consider this small change:

var list = getList();
var nextListItem = function() {
    var item = list.pop();
    if (item) {
        setTimeout(function() {
            // process the item...
        }, 0);

The stack over-flow problem is now eliminated.  Why?  Because javascript's event system handles the recursion–not the call stack.  When nextListItem runs, if 'item' is true (not null) then the timeout function is defined and pushed to the event queue, then the function exits–and the call stack is now clear. When the event queue runs it's timed' out event, the item is processed and nextListItem is recursively called.

Again, the method is processed from start to finish without a direct recursive call, so the call-stack is again clear–regardless of the number of iterations.

Now consider this example:

var userList,
    cleverProgrammerBonus = 10000;

var userUpdateCallback = function(err) {
    if (err) return updateCompleteCallback( err );
var nextUser = function() {
    var user = userList.pop();
    if (user) { += cleverProgrammerBonus;
        dao.updateUser( user, userUpdateCallback );
    } else {
var userFetchCallback = function(err, list) {
    if (err) return updateCompleteCallback( err );
    userList = list;
dao.fetchUsers( userFetchCallback );

As with most functional, asynchronous solutions, it's best to start at the bottom–the dao fetch of users.  The fetch callback sets the user list then invokes the recursive nextUser() to get the iteration started.  Then the user is updated asynchronously, and when it completes updateUpdateCallback is invoked.  This callback continues the recursion by invoking nextUser.  This is repeated until the list is empty.

There isn't a need for setInterval ( or nextTick() ) because the dao calls to update the user are asynchronous–always invoked from the event queue, not the call stack. So the call stack remains static no matter how many users are in the list.

Its also important to realize that the function dao.updateUser doesn't really start working until nextUser has exited. It's queued to do some work, but not at the direction of the call stack–just the event queue.

Asynchronous event driven operations.  Another reason to love coding in javascript.

Tuesday, January 7, 2014

Javascript Coding Standards

I love javascript. I love how it's flexible enough to be as expressive as you want.  The power of closures used in the traditional way or simply as function pointers like in the old C days.

Javascript's flexible nature enables an infinite combination of styles.  From prototypical, IIFE, to classical where functions are actually called "classes" when no actual classes exist.

This flexibility can lead to endless discussions of "what's best" or "what's most efficient" and worse can lead to project code that can resemble a big steamy bowl of pasta with brownish marinara.  As a team leader, the last thing you want is to have to code walk dozens of coding styles.  As a coder and team member the last thing you what to do is traverse hundreds of lines of code that has bad indentation or horrible style.

So, as a manager, you need to adopt a strict coding standard and ruthlessly enforce it.

Coding Standard Objectives

Here is a short bullet list.  The standards should...
  • match the team's current experience and talent,
  • should closely follow at least one major published standard,
  • should have justifications for each decision,
  • and include team consensus.

Team Experience

Javascript isn't new, but it's new to many programmers.  If your team has years of java and c++ experience, the wide-open flexibility of javascript will come as a frustrating surprise.  If your team has a ruby influence, then you have a leg up.  Understanding closures, dynamic typing, etc will ease the transition to production ready javascript.  If your team is primarily PHP based, then they know a bit about javascript but probably don't understand its full capability.

For the Java Team

So lets begin with the java programming team--what standards would suite them best?  The choice is easy--use a classical implementation with factory generated components that implement IoC through parameter injection.  Think of it this way, every file is a class defined by a named function variable (the function is still technically anonymous).  Its injection is always through construction (think ruby opts), and always factory built.  Files are organized as pseudo-package folders that may (or may not) have package namespaces.

The coding style is very similar to the java specs.  K&R formatting, title cased classes, mixed case variables, no underscores.  Complete separation of concerns, i.e. MVC plus services, delegates and factories.  And, unit tests--either TDD or BDD for everything.  And not just the normal tests, but also tests to find bugs that the compiler would normally find.  (mocha is probably the best test platform for both client and server javascript today).

Published Standards

Douglas Crockford.  JSLint.  Javascript, the good parts.  If you can only read one javascript book, please read "the good parts".   Coding conclusions aren't drawn on what is best--just what the alternatives are.  But, he does describe what he feels are some best practices for coding style, indentation, curly braces, etc.  And, if you have the time, watch his video series on javascript.

For good published standards, start here:

Standards you probably want stay away from:
  • if you are writing applications, the don't follow these standards; utility modules maybe, but...
  • M$ standards--known only to a few Seattle east-siders.

Our Standards Document

Here is a link to our current javascript coding standards.  For the most part, this is for client applications, but most parts apply to node/server projects as well.


My best advise is to find a good published standard, modify it for your team's abilities, then stick with it.  You will need to add rules as time goes on--because coders will always find differing ways to interpret the rules.  So be prepared for additions--not so much changes, just clarifications.

Happy coding...