rain city digest

Easy Mac Installs--MySQL & NetBeans

2010-09-23T08:04:00.000-07:00

I just went through two very easy installs on my new mac; the first, MySQL 5.1.5. A few years ago, when I installed on my old mac it was the typical tar ball install that linux users are familiar with. Not bad, especially if you have a canned script to remind you of all the steps.

But now there are two dmg package files that do most of the work for you. And, it installs in /usr/local with a dynamic link to mysql--just like it's supposed to be. The only steps left were to remove the anonymous users, supply the root and admin passwords and reboot to start the service. Easy.

Next was NetBeans 6.9. I have been using 6.8 for a few projects, and have loved the support for groovy and especially grails, but I have held off upgrading while in full development mode. But now, with a new grails project in it's infant stages, I decided to make the move.

Installation was easy. I did have to manually replace the 6.8 item with 6.9 on my doc, but other than that all went easy. It found my current projects and I was able to run unit tests from within the IDE. Once again, easy.

Happy coding...

Grails 1.3.4, Groovy 7 and NetBeans 6.8

2010-09-22T20:50:00.000-07:00

After about a year of doing nothing but Adobe Action Script projects, I'm finally back to a java/groovy/grails project. So, I downloaded the latest grails and fired up my version of NetBeans (6.8) and generated some code. Here's what I discovered...

First, NetBeans is very groovy/grails friendly. I was able to generate the project inside the IDE. I then dropped to the command line to generate domain classes and tests. I was able to run the full test suite inside the IDE, but I still prefer the command line.

I noticed that the generated code doesn't include an ant script, but just as well. I'm so sick of the XML crap, and the grails command line does everything you need, so...

Next step is to generate controllers, and maybe a few views, but I usually use grails as a JSON or XML server with a Flex/Action Script UI. I'm also creating all the unit tests as I go and running them from inside the IDE and from the command line.

Next post will discuss the controllers and any problems related to getting the application into production. So far everything works as advertised--better than previous versions, and very productive. My new best friend...

UMLet UML Diagraming

2010-08-10T18:56:00.000-07:00

Just had to comment on a very useful new(ish) project for editing UML documents called UMLet. No frills or bs* just a good productivity tool in the open source space. Human readable file structure to enable old timers like me to hack in attributes with vi.

Converts your drawings to png, jpg, eps, pdf, svg, etc from inside the UI or even from the command line (a real programmer's tool).

It comes in two flavors 1) eclipse plugin, and 2) standalone Swing application. They have a few skins that help ease the swing pain but who cares about the L&F, I just want a tool that works with me instead of against me.

So, thanks to the UMLet team for a great product!

HTML5, CSS3, Apple, Google and the W3C

2010-04-16T00:43:00.000-07:00

About a year ago, I remember responding to a question from a colleague concerning Flex in comparison to HTML5 by saying that HTML5 was not ready yet. For the most part, I still hold that opinion but with additional qualifications.

We are currently bidding on a job that requires iPad as the target device and another that needs a private label application that will work on the desktop, iPhone and eventually Android and Blackberry. After logging many hours researching objective-c and HTML5/CSS3 my first thoughts were that all development needed to be done in objective-c. But, after viewing some of the other videos from Apple, MIT, and Stanford, and doing some small tests, I think that most applications for the iPad/iPhone can be implemented completely in HTML5/CSS3. So here is my revised/updated response...

If you have complete control over the target web browser (i.e., webkit/khtml based) then HTML5 and CSS3 can't be beat. Using the 'canvas' tag/object enables drawing anything that you can draw with flex/flash and also includes drawing text (not possible with flex). Using CSS3 enables all the transitions that are currently available in flex including easing methods. Using WebSocket, provides peer-to-peer real time communications. So if your target includes Safari, iPhone, iPad, iPod/Touch, Google Chrome, Opera, Firefox, as well as the Android, Sprint and Blackberry browsers, the new technology is ready today.

The missing piece is IE, currently at 70% of desktop browser installations. An application that needs to run only in a standard client browser is still better off using Flex (95% penetration) until Microsoft complies with the latest W3C specs. But, the flash player will probably never make it to small devices (it's a hog), so if the requirement is to run on desktop and phone, especially with touch gestures, HTML5/CSS3 is probably a better choice. The down side is waiting for Microsoft to catch up with the fully developed world and hoping they will fully comply with the W3C.

Action Script Threads for Report Generation

2010-02-11T21:01:00.000-08:00

My requirement: to create a complex report that may take several seconds to run without blocking the user's interaction with the application. In java the solution is simple: create a thread and invoke start(). In action script, it's a bit more difficult.

Action Script doesn't support threads directly out the the box but it is run in a multi-threaded framework, so I knew there must be a solution. I took some time to research how other flex developers were implementing Action Script threads and concluded that the best solution was to simply use the Timer class to create a new threads.

The timer class has an asynchronous event called TIMER_COMPLETE that is perfect for creating a bloated, heavy weight thread. But, if all you need is a separate thread to run while creating a huge report, this does the trick.

One of the practical considerations is that you will probably want a supporting class to encapsulate run parameters for each thread you create. For reports this works well because there is enough logic in each individual report to warrant its own class. And as long as the report class implements the timer start method then you are in good shape.

The simplest implementation looks like this:


public function start():void {
var timer:Timer = new Timer(1, 1);
timer.addEventListener(TimerEvent.TIMER_COMPLETE, run, false 0, true);
timer.start();
}

// override this base class to implement the run worker
protected function run():void {
// create the report or whatever you need...
}

Implement the run handler and you are set. The class can encapsulate any run time parameters that you require including a common dispatcher to dispatch a 'COMPLETE' event when the process is finished. Easy.

This is an acceptable work-around for a language that does not support threads right out of the box. But for the rare occasion that you really need a thread to enable interaction while a heavy-weight, time consuming task is running, this really does the trick.

Happy coding...

Transforming Average Programmers into Power Developers

2010-01-24T22:09:00.000-08:00

The term software crisis coined in the late 60s described the ratio of software required by industry to the inability of software engineers to deliver correct, understandable, and verifiable programs. Back then the problem was blamed on the relative high power of new computers and the general low quality of programmers.

And now 40 years on, our industry continues to suffer the software crisis, but not because computing power has outstripped programmer ability. And our crisis cannot be blamed on a lack of programmers; the world has provided an over abundance of programmers, so that's not an issue. But, as an industry, we still fail to deliver on a consistent basis. Here are some tips designed to help alleviate our crisis and improve the quality of our programming community.

Learn UNIX. Ok, linux, mac osx, or cygwin; the main idea is to use the command line in a well thought out shell. Good programmers program all the time, just like musicians practice scales. Every time you enter a series of commands into the command line you are excising your skills. When you learn the power of command pipelines to solve basic problems your are flexing your ability to solve problems pragmatically. Learn and use tar, jar, grep, tail, find, sed, awk, lex, yacc, etc. Use curl, svn, git, ant, make, ssh and scp. Find a good shell like zsh, bash, ksh or whatever and write scripts to help you organize your work. Write aliases that tame the command line. Write sed or egrep scripts to exercise your Regular Expression patterns. The important thing is that "Programmers program--always".
Expose yourself to as many languages as you can. You probably have one or two primary languages that you use on the job. Don't limit yourself to just those. Create a sandbox using subversion or git and create scripts in ruby, php, perl, javascript and python. Create java, action script, c++ sandboxes with either ant, gant, rake, or make files to build small applications. Learn new, or newly recognized languages like Scala, OCaml, Erlang, Haskell and Go.
Contribute to an open source project. There are a gazillion open source projects on the market now, most of them throw away. Rather than creating your own, join one that has a chance of succeeding. Read the project's standards and rules and begin by submitting documentation updates. Search the bug list and submit proposed fixes, complete with unit tests (don't bother with projects that ignore tested designs). After a while, if you pass the initial stages, you will become a committer--your right of passage to the open source community.
Read and re-read the APIs. The programming language that you primarily work in was created by several brilliant people that have devoted thousands of hours to create a useful platform. Don't ignore their work. Even if the platform solution may not exactly fit your situation, at a minimum, know what it does first before creating your own. Dive deep and read the platform source code.
Read the expert blogs. Posts to Object Mentor, InfoQ, Artima and experts like Martin Fowler, Ward Cunningham, Dave Thomas, Kent Beck, and a host of others. Find the blogs that are specific to your language. Look for web or pod casts.
Keep up with technology changes. IoC from Spring Source or Google, Agile and Extreme Programming, Rails Active Record, Hibernate, Apect Oriented Objects, JUnit 4, Flex Unit 4, DSLs, Patterns, Idioms. Lots to keep up with, some hype, others real.
Write readable, clean code. The definition of clean code is 1) code that is understandable by someone other than the original author, 2) package, class, and method names that are accurate and fully describe their intended purpose, and 3) code that is fully tested with a suite of automated test tools. Always choose readability over writability. As Knuth says, "A program is essentially a work of literature".
Read. Most programmers are in the habit of purchasing lots of technical books. That's great, but don't let them sit on the shelf. Read and re-read them. Dog-ear or sticky note sections that you want to re-visit. Highlight important points that resonate with you. And, don't just read the cook-book tutorials. Your library should contain a healthy mix of literature from publishers like Prentice Hall and Addison Wesley to offset your APress, O'reilly, Manning, and Wrox vocational studies.

I'm sure that there are more items, but these eight represent a good start. Happy coding!

Maximizing FlexUnit4 with Mocks and Closures

2010-01-10T20:48:00.000-08:00

After spending the past few months using FlexUnit4 that comes with the new version of Adobe Flex called Flash Builder, I have a few tips on how to maximized your testing using mocks and closures. This comes as second nature for java/groovy/ruby programmers, but may be new to some in the Flex/Action Script community, so here goes.

Mocks: The project that I am currently on uses Swiz for IoC wiring and configuration of controllers, delegates, components, services and loggers. But, it's not good practice use IoC for tests, so we use mocks. Mocks that configure themselves, expose protected members, and reassign closures to methods that modify behavior for test purposes.

Example: The Shape factory: We have a factory class that generates various shapes such as lines, rectangles, ellipses, comment boxes, polylines, curves, etc. The method closure that creates specific shapes is called createShapeFromType. In the class constructor this closure is mapped to the factory's local private method createShape that creates and returns a shape object for the specified type. During some tests, this closure is mapped to createMockShape which returns a mock shape that can be tested independent of the factory. Other tests still use the mock factory but with the default createShape method in place to test and verify that it's behaving as expected. Think of it like a second level of polymorphism without having to create complex and ridged hierarchies (class bloat reduction).

MockShapeFactory: This mock extends ShapeFactory to inherit all of it's normal behaviour. So when I write a unit test against one of it's methods, I know that I'm exercising the real thing. But the mock allows me to keep track of what's going on inside the class as the tests run. And, the mock does it's own configuration mimicking what IoC performs in production.

MockShape: This mock has two primary purposes; 1) to configure itself for use, logs, delegates, etc, and 2) to expose protected methods and variables. For example, the shape has mouse event listeners that can be invoked through the test to see if the object moves to a specific set of coordinates. Or if it's a non-scaling shape, you can verify that the target shape performed a move an resize rather than a simple scaleX/scaleY operation. You can also test that a vetoed request for focus does indeed ignore the request and remain unfocused. There are many possibilities.

Some of this is very basic for test junkies, but as I said earlier, the Flex community seems to be just catching on to test driven development. And, with the new Flex/FlashBuilder and FlexUnit4 there is no excuse to ignore testing any longer.

Flex 4 BitmapImage

2009-10-26T12:31:00.000-07:00

Flex 4, and specifically the new spark framework has a BitmapImage object in the spark.primitives package. It's associated mxml tag is . I found this after trying to add a Bitmap object to a VGroup only to get errors. So, after searching a bit, I found the BitmapImage and simply set the source property to my BitmapData. Very easy.

My code problem was to read a remote image then scale it to as close to a target width & height as possible without changing the original aspect ratio. From there, the user can rescale the image and drag a focus rectangle over the image to select the desired image section. Then, with the click of a button, the image is scaled and cropped to the optimum size. I used JPEGEncoder to write the data back out to a new image file.

Flex 4/Flash Builder Beta

2009-10-20T17:09:00.000-07:00

Adobe released Flash 4 Beta 2 a few weeks ago with many changes and improvements over Flex 3. The new lightweight spark framework has many advantages over the older mx components and separates skinning in an attempt to facility and advocate better separation of concerns and improved maintainability.

There are also some significant improvements to the Flash Builder IDE that take much better advantage of the eclipse framework. It was always a big mystery to my why Flex 2 and 3 Builders seemed to strip so many of the basic features out of eclipse when they created their IDE. Many of those features, although not all, are now included.

One basic but vital feature is the ability to create your own standard templates for AS3 classes and MXML applications and components. The approach follows the eclipse use of internal variables, like ${user}, ${date}, ${year}, etc.

Another feature is conditional breakpoints and watchpoints. There is also a way to do remote debugging.

So it looks like the new Flash Builder comes closer to providing the basic eclipse functionality that has been missing since Flex 1.x was introduced. I'll post later with other features that have been included while I work my way through the latest Max 2009 presentations and the on-line "Flex 4 in a week" training series. And, in subsequent posts, I will share what is offered by the new Spark platform.

Hotlinks External Page Viewer

2009-09-04T18:23:00.000-07:00

I first heard about twitter in 2007 while listening to a Ruby On Rails podcast. It took a while, but I finally got on board, mainly as a result of the mandatory SEO stuff required for my current project. So, I'm now all a twitter.

Twitter's 140 character limit leads to a mandatory link for almost every post. Some links are huge, but with the help of tinyurl.com or bit.ly you can hash down your URL to a very manageable size. At the same time, the link reduction sites allow you to track the number of hits the link has received.

There are a couple of companies that have mashed up the URL reducers with the ability to display a personalized reference at the top of the target page. So, If I tweet about Hillary Duff's new utube video I would be able to display a header that includes a link to my site at the top of the page. Rather than use one of the off the shelf products, I decided to build my own, mainly because 1) it was trivial to implement, and 2) I (and my customer) wanted custom graphics displayed on the banner without the third party interference.

So, here is a sample of what I mean. The original tweet would look a bit like this: "check out Hillary Duff's new utube thingy here: http://bit.ly/30YbdB". When the tweet recipient clicks on the link, they see the video as described, but they also see the Rain City Software banner at the top of the page, complete with links to my web site and twitter account. Go ahead, give it a try.

As it turns out, this is very simple to do with a simple iframe to display the target URL. The banner is small and hopefully not offensive and can be closed by clicking on the 'X' icon. Or, the user can click on the logo to get to the Rain City site or click on the tweety birds to get to Rain City's twitter account. All sorts of possibilities (including incorporating swf's, but lets keep it simple for now). And, the clicks are tracked not only by bit.ly but for google analytics as well.

Creating the Hotlink: I made the hotlink creation easy with a small web based UI. The user simply enters a title and pastes in the link then clicks create (after selecting the correct server). The hotlink is created and can be copied to bit.ly, tinyurl or your favorite URL shortener. Here is what we get for the Hillary Duff link:

The user can test the link by clicking 'Test Link' to see if all is in order. If this looks ok, then copy the link to the shortener, make the tweet and that's it.

Adobe/Flex UI Code: As you can see from the code below, the code is quite simple. Click handlers for create and test, and the use of escape to get the title and target URL (hotlink) to work correctly as http parameters. I thought about using he open APIs provided by many URL shorteners, but decided to just let the user copy/paste the target URL. Here is the Flex xml code:

The Hotlink Service Code: I decided to implement the service code in ruby but php or java would have worked just as well. The objective is to pull the title and link from the URL parameters and use those inside the standard template code. The template code should probably be separated out, but it was easy enough to keep everything all in one file. The banner can be any combination of backgrounds, links, whatever. Anyway, here is the ruby code:

So that's it. Easy to use and all the tracking that an SEO could hope for. And, very customizable to suite the customer's requirements. Let me know if you want help setting up your own. I would be happy to help.

Social Network Common Login in Flex

2009-08-27T13:54:00.000-07:00

JanRain provides a free version of their industry-leading OpenID solution. RPX Basic enables websites to accelerate user registration and login success by allowing visitors to easily sign in with one of their existing third party accounts from AOL, Facebook, Google, MySpace, Yahoo! and more.

Their product includes a drop-in widget written in Javascript that does all the communications to their open id server and third party providers. But, for Flex applications (or Java Applets) they provide a set of API calls that do the job.

Here is what our custom login dialog looks like:

The user simply clicks the preferred provider, and the appropriate API call is made to RPX to open a new window. For example, if the user clicks the facebook icon, the following page is displayed.

If the users successfully logs into facebook, a token is returned to the Flex application. The next step is to use this token to access the user's profile, email, photo, etc. Here's what the user see's after they log in.

The same thing happens if the user logs in with any of the other providers. Subsequent logins don't require the approval, but simply log in without the intermediate page. So, if I'm already logged into facebook and have approved access from my Flex application, the click skips the facebook login and returns the token. Simple.

The Code: The code adheres to the MVC paradigm and is short and sweet. A single controller talks to the RPX API and a login dialog mxml handles the display and clicks. The controller publishes two events loginComplete and accessFailed. The dialog processes the provider click through the controller while listening for the response event.

Configuration: When you sign up with JanRain for RPX Basic, there are a few configurations that need to be made to get facebook and other third party authorization platforms to work. The process is simple and only takes a few minutes.

Testing was easy for all six of the providers that our application uses. And, there are more coming on line all the time. It's a great example of distributed processing that eliminates user resistance to signing up for your latest social network application.

Simple Digest to Test File Uniqueness

2009-07-17T16:25:00.000-07:00

Here is a simple way to use the groovy Digest class from org.raincity.glib.crypto package. It's limited to reasonably small files, but could be used with chunked sections of files. In any case, here is the test script:


void testStringDigest() {
  def algorithm = 'SHA-512'
  def s1 = 'this is test 1'
  def s2 = 'this is test 2'
               
  def hash1 = new Digest().createHashString( s1, algorithm )
  def hash2 = new Digest().createHashString( s2, algorithm )
               
  println "h1 = ${hash1}"
  println "h2 = ${hash2}"
               
  assert hash1 != hash2
}

s1 = the byte stream from the first file, s2 is the byte stream from the second file. If the hashes match, then the files are identical, i.e., not unique.

Digest's default algorithm is SHA-256, but SHA-512 is available in java > 1.4 and is much stronger. The 256 default was designed to match the SHA-256 limit for Adobe/Flex.

Thanks to Brad Rhoads for inspiring this post...

Amazon S3 Access with JetS3t

2009-06-04T17:53:00.000-07:00

Back in my ruby/rails days, I always accessed Amazon S3 from the command line. No that I'm coding in groovy/grails, I thought I would search for a groovy solution. Well, I found one, but it's not ready for prime-time. But, I did come across JetS3t, written in java with examples in groovy, so I thought I would give it a try.

After downloading, I opened the README and found that JetS3t comes with a nifty set of stand-alone and web based UI application/applets for accessing, browsing and uploading files. The application is called Cockpit and I was able to easily use it inside Firefox and as a standalone Java application.

Find out more about JetS3t suite of applications and be sure to read Andrew Glover's tutorial.

Lame Flex IDE

2009-04-22T20:44:00.000-07:00

Ok, Adobe Flex, or AS3 is an OK language. But their commercial Flex IDE sucks. Adobe uses (abuses) the eclipse framework for their commercial IDE. They charge big bucks for a somewhat visual editor that amounts to a bucket of shit.

After wrestling with it for over two years I'm tempted to go back to vi and the command line. Anyone else feel this way?

Parsing ISO 8601 Dates in Flex

2009-03-23T14:30:00.000-07:00

Flex has many capabilities, but parsing dates is not one of them. As a member of the JSR-310 team I am focused on date formatting and parsing across many formats, so it comes as a bit of a surprise that Flex has very little support for this. I guess it's time to roll my own.

Unlike Java date formatters, Flex doesn't let you set the parse format to accept custom inputs. The formats must match one of the seven standard parse formats that Flex supports. Some third party examples try to use Date.parse() after massaging the input string, but they are a bit lame.

A good way to parse dates in Flex is to use RegExp to extract the numbers. Then, the numbers need to be validated prior to assignment. First, lets look at a few code snippets to see how to pull the numbers out of a local date (no time zone offset).


// ISO 8601 date time as YYYY-MM-DDThh:mm:ss.SSS
var dateTimeArray:Array = inputDate.split('T')
var dateString:String = dateTimeArray[0]
var timeString:String = dateTimeArray[1]

// parse the date from YYYY-MM-DD
var pattern:RegExp = /(^\d{4})-(\d{2})-(\d{2}$)/
var result:Object = pattern.exec( dateString )
if (result == null)
  throw(new Error("invalid date format"))

var year:uint = new uint( result[1] )
var month:uint = new uint( result[2] )
var day:uint = new uint( result[3] )

if (!validateDate(year, month, day))
   throw(new Error("invalid date format"))

var date:Date = new Date(year, month - 1, day, 0, 0, 0, 0)

// now parse the time from HH:MM:SS.SSS
pattern = /(^\d{2}):(\d{2}):(\d{2}).(\d{3})/
result = pattern.exec( timeString )
if (result == null)
   throw(new Error("invalid time format"))

var hour:uint = new uint( result[1] )
var minute:uint = new uint( result[2] )
var second:uint = new uint( result[3] )
var milli:uint = new uint( result[4] )

if (!validateTime(hour, minute, second, milli))
   throw(new Error("invalid time format"))

date.setHours(hour, minute, second, milli)

Lenient Dates and Times: It appears that in Flex world, March 32nd == April 1st. And April 2nd at 25 hours == April 3rd at 1am. Is this an April fools joke? No, it's just the lenient way that Flex handles date parameters. No problem with that until you start parsing, then you need to be strict.

Strict Dates: Lets say a vendor sends an electronic invoice to you with a payment due date of 2009-May-31. Does he want his money on June 1st? Probably not. The best response is to reply that his date is invalid and he must submit a valid date.

That's what being strict means. Unfortunately, Flex does not have a way of controlling strict dates, so you need to do this manually.

Validation: Without validation constraints a Flex date can range from 0100-01-01T00:00:00.000 to beyond the year 10,000. Dates earlier than the year 100 are coerced to 1900. This is clearly, not what we want. For parsing purposes there should be a range of minimum/maximum acceptable dates. For discussion purposes, I'll set the range to start at the recognized Gregorian start date of 1582-10-17T00:00:00.000 and end at an arbitrary future date of 2999-12-31:23:59:59.999.

Now that we have a date range, the years are easy, 1582..2999. Months for Flex follow the old C convention of zero indexing, so months range from 0..11 but we will use 1..12 for validation then decrement the month prior to creating the flex Date object.

Days are a bit trickier, but lets use 31 as the default, then 30 for Apr, Jun, Sep, Nov and 28/29 for Feb, after determining the leap year. Here is the standard calculation for leap year lifted from the white book:


var leap = ((year % 4 == 0) && ( !(year % 100 == 0) || (year % 400 == 0)))

Hours, minutes, seconds and milliseconds are bound by 0..23, 0..59, 0..59, and 0..999. This ignores time zone changes and the occasional leap-second, but otherwise works fine.

The Final Code: I created a demo application that allows you to plug in and parse dates and run the unit tests. The demo and source code are available here. I'm still working on a parser for time-zoned dates, but this is a good start.

JSR-310 javax.time Periods

2009-02-28T07:30:00.000-08:00

The proposed JSR-310 date/time API comes with many representations of date and time including Instant, Duration, LocalDate, LocalTime, LocalDateTime, OffsetDate, OffsetTime, OffsetDateTime, ZonedDateTime, MonthDay, YearMonth and others. One of the more interesting classes is Period. It represents a quantity of time, not fixed to any time in space--just a quantity. This entry will discuss periods, how they are parsed, and examples of use.

Parsing: Periods are parsed from string using formats that conform to the ISO-8601 duration format PnYnMnDTnHnMn.nS. Variations of this format are parsed to create Period objects. Parsing is done in PeriodParser, a standalone helper class that is easily accessed through the static method Period.parse(). Here are a few examples:


assert Period.parse("PT0S") == Period.ZERO
assert Period.parse("P1Y") == Period.years(1)
assert Period.parse("P10Y8M22DT3M") == Period.period(10, 8, 22, 3)
assert Period.parse("PT1M") == Period.minutes(1)
assert Period.parse("P-4Y") == Period.years(-4)

As you can see, the parsing scheme is robust and flexible. The main thing to keep in mind is that the Period object is more of a value container as opposed to a process unit. So, 60 minutes won't directly translate to 1 hour, and 24 months does not directly equal 2 years. But this aside, there are many uses for the Period class.

For example, lets say I want to periodically archive old temp files from a system. I could set the period to minutes, hours, days, whatever, then based on an arbitrary start time, begin the archive sweep. In the same moment, I could compute the next archive sweep by adding my pre-defined period object to the current time.

A more elaborate implementation would be to self-adjust the scheduled periods based on some criteria. Sticking with the archive sweep, lets say I set the initial period to 2 hours, or better yet, 120 minutes to give a finer resolution. Then, at the end of the sweep, I tally archived files. The smaller the number, the longer the period, on a sliding scale, up to 240 minutes. A large number would decrease the period to zero minutes, or a continual sweep, so I want to make sure that this is the very worst case scenario.

The Period class makes this easy to implement and maintain. To keep it simple, lets just use a linear equation. The formula for calculating the number of minutes between sweeps is reduced to period in minutes = mx + b, where b = 240 minutes (our maximum amount), x = the number of files, and m = the slope. Lets say that the maximum number anticipated required archives were 2 per minute, so after 240 minutes we would have 480 files that needed to be archived. If that is our worst case then m = -0.5 (delta y divide delta x, or -240 / 480). So the curve looks like this:

Now each time I do an archive sweep, I tally the files and calculate the next sweep interval using:


maxMinutes = 240
slope = -0.5
periodInMinutes = slope * fileCount + maxMinutes

Or, as a single groovy closure:

def periodToNextSweep = { fileCount, slope = -0.5, maxMinutes = 240 ->
 Period.minutes( slope * fileCount + maxMinutes )
}

You might think there is a danger in allowing the returned period to be a negative number of minutes. But in all practicality this is acceptable because the objective is to determine the next instant when a sweep should occur. If this time is in the past, simply do it now. Of coarse you would design the slope to target the worst case, so a negative time should seldom if ever occur. And the good part is that the parameters are easy to modify to fit changing environments.

Testing with a Fixed Time Source: My previous entries have demonstrated using TimeSource and Clock tied to the System clock. But, for these tests, I think a fixed time source would be more appropriate. The syntax is like this:


millis = 1234920035991L // 2009-02-17T17:20:35.991-08:00 Tuesday...
timeSource = TimeSource.fixed( Instant.millisInstant( millis ) )
clock = Clock.clockDefaultZone( timeSource )

So now when I create a date, time or date/time object from the clock, the time is always the same. Not very meaningful for real life, but great for testing.

If I create a class that uses clock, I can inject the TimeSource based on the system clock. And for testing, I can inject a fixed TimeSource, run tests, and not have to worry about the specific time, but simply base my tests on a static source. Here is the class:


class SweepController {
 def clock = Clock.systemDefaultZone()
 def slope = -0.5
 def maxMinutes = 240
 
 def periodToNextSweep = { fileCount ->
  int x = (int)(slope * fileCount + maxMinutes)
  x < 0 ? Period.ZERO : Period.minutes( x )
 }
 
 def nextSweepTime = { fileCount ->
  def period = periodToNextSweep( fileCount )
  
  clock.offsetDateTime() + period
 }
}

And here is the test script:


millis = 1235721600000L // 2009-02-27T00:00-08:00
fixed = TimeSource.fixed( Instant.millisInstant( millis ) )
clock = Clock.clockDefaultZone( fixed )

sweep = new SweepController( clock:clock )
println "now -> ${clock.offsetDateTime()}"
source = [ 
 [ 480, '2009-02-27T00:00-08:00' ],
 [ 0, '2009-02-27T04:00-08:00' ],
 [ 240, '2009-02-27T02:00-08:00' ],
 [ 120, '2009-02-27T03:00-08:00' ],
]
source.each { count, value ->
 println "count: ${count} -> ${sweep.periodToNextSweep( count )}, ${sweep.nextSweepTime( count )}"
 assert value == sweep.nextSweepTime( count ).toString()
}

When I run the script, here is what I get:


now -> 2009-02-27T00:00-08:00
count: 480 -> PT0S, 2009-02-27T00:00-08:00
count: 0 -> PT240M, 2009-02-27T04:00-08:00
count: 240 -> PT120M, 2009-02-27T02:00-08:00
count: 120 -> PT180M, 2009-02-27T03:00-08:00

The main advantage is that I can run this independent of the current date, but still use the clock object without changing anything inside the class.

Conclusion: This quick look at Period and PeriodParser to see how it fits into the JSR-310 from the groovy coder's perspective. Next time well look closer at Date, Time and DateTime math capabilities and how they work with groovy plus/minus operator overloading.

Adjusting Date and Time with javax.time

2009-02-22T09:00:00.000-08:00

Java's JSR-310 date and time API, co-lead by Michael Nascimento Santos and Stephen Colebourne is a natural spinoff from the venerable joda time. The implementation has many advantages over java util's Date and Calendar. Compared to java.util.Date the API has many types of Date and Time including LocalDate, LocalTime, LocalDateTime, OffsetDate, OffsetTime, OffsetDateTime, and ZonedDateTime.

The new JSR-310 date, time, and date/time classes are immutable and thread safe, unlike java.util.Calendar, but at the same time they offer many basic math, adjusters, and matchers that enable date and time calculations missing in java.util.Date. They are created from a TimeSource that can be tied to the System clock, fixed, or offset in time. This makes the classes extremely test friendly.

This entry discusses how the basic date/time math work and how the date and time adjusters can be used to solve common problems. Lets look first at the basic math.

Look Ma, no Setters: The JSR-310 date/time classes are immutable and thread safe. To accomplish this, the API doesn't include any setXX methods. Days, Hours, Years, etc are manipulated through "plus", "minus", and "with" methods that return new objects of the same type. Here are a few examples, first with date then time:


// tomorrow and 5 years from now
clock = Clock.systemDefaultZone()
today = clock.today()
assert clock.tomorrow() == today.plusDays(1)
fiveYearsFromNow = today.plusYears( 5 )
assert today.plusMonths( 60 ).year == fiveYearsFromNow.year

// two hours ago
now = clock.timeToSecond()
twoHoursAgo = now.minusHours( 2 )
assert now.minusMinutes( 120 ) == twoHoursAgo

// use offset datetime to get today at noon
noon = clock.offsetDateTime().withTime( 12, 0, 0 )
assert noon.hourOfDay == 12
assert noon.minuteOfHour == 0
assert noon.secondOfMinute == 0

The Adjusters: Here is a tricky problem: how do you compute the specific date of a week numbered day of the month, for example the 3rd friday or 4th tuesday. Bay area residents know that not being able to calculate these simple problems can cost real money (street sweep adys). So here is how JSR-310 handles this:


dt = clock.offsetDate()

thirdFridayAdjuster = DateAdjusters.dayOfWeekInMonth( 3, DayOfWeek.FRIDAY )
fourthTuesdayAdjuster = DateAdjusters.dayOfWeekInMonth( 4, DayOfWeek.TUESDAY )

thirdFriday = dt.with( thirdFridayAdjuster )
println "third friday -> ${thirdFriday}, ${thirdFriday.toDayOfWeek()}, ${thirdFridayAdjuster}"
assert DayOfWeek.FRIDAY == thirdFriday.toDayOfWeek()

fourthTuesday = dt.with( fourthTuesdayAdjuster )
println "fourth tuesday -> ${fourthTuesday}, ${fourthTuesday.toDayOfWeek()}, ${fourthTuesdayAdjuster}"
assert DayOfWeek.TUESDAY == fourthTuesday.toDayOfWeek()

These examples just scratch the surface of the many date/time manipluation methods for JSR-310's javax.time package. The next entry will discuss how JSR-310 matchers help determine if a date lands on a leap year, leap day, last day of the month, etc. and how to use this in groovy.

JSR-310 javax.time and Groovy Tests

2009-02-21T11:00:00.000-08:00

The development and implementation of JSR-310 is continuing to move ahead at a good pace. Once delivered, (hopefully by JDK 7) the java community will have a solid replacement for the outdated and comparatively buggy Date and Calendar classes. At the same time, I have created a series of groovy tests that exercise the JSR-310 capabilities.

Formal unit tests for JSR-310 are written in testng and cover all the low level details including TimeZone resolving, serialization, etc. My tests simply verify a subset of the current API and demonstrate use with groovy. The tests cover the basic API, adjusters, matchers, peroids, formatters and parsers. This post demonstrates the basic tests.

Basic API Tests: These tests take advantage of groovy's invokeMethod() to test methods that take zero or more parameters. They demonstrate typical use of TimeSource, Clock, LocalDate, and LocalTime classes. With these basics it's easy to extend to LocalDateTime, OffsetDate, OffsetTime, or ZonedDateTime. Here is the output trace generated from running BasicTests.groovy:


TimeSource to Date/Calendar conversion tests ---------------------------------------
Timesource.system().instant() -> {ts.instant()}
Clock Method Tests ----------------------------------------------------------------
clock = Clock.systemDefaultZone() -> TimeSourceClock[SystemTimeSource, America/Los_Angeles]
Clock methods...
method: clock.getSource() -> SystemTimeSource
method: clock.getZone() -> America/Los_Angeles
method: clock.today() -> 2009-02-21
method: clock.tomorrow() -> 2009-02-22
method: clock.yesterday() -> 2009-02-20
method: clock.dateTime() -> 2009-02-21T08:56:35.512
method: clock.dateTimeToMinute() -> 2009-02-21T08:56
method: clock.dateTimeToSecond() -> 2009-02-21T08:56:35
method: clock.time() -> 08:56:35.515
method: clock.timeToMinute() -> 08:56
method: clock.timeToSecond() -> 08:56:35
method: clock.offsetDate() -> 2009-02-21-08:00
method: clock.offsetTime() -> 08:56:35.517-08:00
method: clock.offsetTimeToMinute() -> 08:56-08:00
method: clock.offsetTimeToSecond() -> 08:56:35-08:00
method: clock.offsetDateTime() -> 2009-02-21T08:56:35.518-08:00
method: clock.zonedDateTime() -> 2009-02-21T08:56:35.521-08:00 America/Los_Angeles
method: clock.zonedDateTimeToMinute() -> 2009-02-21T08:56-08:00 America/Los_Angeles
method: clock.zonedDateTimeToSecond() -> 2009-02-21T08:56:35-08:00 America/Los_Angeles
method: clock.year() -> Year=2009
method: clock.yearMonth() -> 2009-02

OffsetDate methods from offsetDate = clock.offsetDate() -> 2009-02-21-08:00
method: offsetDate.atMidnight() -> 2009-02-21T00:00-08:00
method: offsetDate.getDayOfMonth() -> 21
method: offsetDate.getDayOfWeek() -> DayOfWeek=SATURDAY
method: offsetDate.getDayOfYear() -> 52
method: offsetDate.getMonthOfYear() -> MonthOfYear=FEBRUARY
method: offsetDate.getYear() -> 2009
method: offsetDate.toDayOfMonth() -> DayOfMonth=21
method: offsetDate.toDayOfWeek() -> DayOfWeek=SATURDAY
method: offsetDate.toDayOfYear() -> DayOfYear=52
method: offsetDate.toLocalDate() -> 2009-02-21
method: offsetDate.toMonthOfYear() -> MonthOfYear=FEBRUARY
method: offsetDate.toYear() -> Year=2009
method: offsetDate.getDate() -> 2009-02-21

LocalDate methods from localDate = LocalDate.date(2019, 03, 01) -> 2019-03-01
method: localDate.atMidnight() -> 2019-03-01T00:00
method: localDate.getDayOfMonth() -> 1
method: localDate.getDayOfWeek() -> DayOfWeek=FRIDAY
method: localDate.getDayOfYear() -> 60
method: localDate.getMonthOfYear() -> MonthOfYear=MARCH
method: localDate.getYear() -> 2019
method: localDate.toDayOfMonth() -> DayOfMonth=1
method: localDate.toDayOfWeek() -> DayOfWeek=FRIDAY
method: localDate.toDayOfYear() -> DayOfYear=60
method: localDate.toLocalDate() -> 2019-03-01
method: localDate.toMonthOfYear() -> MonthOfYear=MARCH
method: localDate.toYear() -> Year=2019
method: localDate.getMonthDay() -> --03-01
method: localDate.getYearMonth() -> 2019-03
method: localDate.isLeapYear() -> false
method: localDate.toEpochDays() -> 17956
method: localDate.toModifiedJulianDays() -> 58543
method: localDate.toDateTimeFields() -> {ISO.Year=2019, ISO.MonthOfYear=3, ISO.DayOfMonth=1}

LocalTime methods, from localTime = clock.time() -> 08:56:35.549
method: localTime.getHourOfDay() -> 8
method: localTime.getMinuteOfHour() -> 56
method: localTime.getSecondOfMinute() -> 35
method: localTime.toDateTimeFields() -> {ISO.HourOfDay=8, ISO.MinuteOfHour=56, ISO.SecondOfMinute=35, ISO.NanoOfSecond=549000000}
method: localTime.toHourOfDay() -> HourOfDay=8
method: localTime.toLocalTime() -> 08:56:35.549
method: localTime.toMinuteOfHour() -> MinuteOfHour=56
method: localTime.toSecondOfMinute() -> SecondOfMinute=35
method: localTime.getNanoOfSecond() -> 549000000
method: localTime.toNanoOfSecond() -> NanoOfSecond=549000000
method: localTime.toNanoOfDay() -> 32195549000000
End basic tests...

Close inspection reveals that some methods in LocalDate are missing in OffsetDate. This isn't a huge problem, because OffsetDate can easily create a LocalDate object. But, it would be nice to have isLeapYear(), toEpochDays(), toModifiedJulianDays() and other methods in both classes. (This may already be in the works)...

Here is a simple way to access some of the missing methods:


offsetDate = clock.offsetDate()
offsetDate.toLocalDate().toEpochDays()
offsetDate.toLocalDate().toModifiedJulianDays()
offsetDate.toLocalDate().isLeapYear()
// or, even groovyer
offsetDate.toLocalDate().leapYear

Conclusion: What these tests really do is to demonstrate the basic functionality of the new javax.time classes. It's obvious that JSR-310 offers the horsepower missing from the current java implementations. And, it will be a very useful addition to the groovy toolkit as well.

Future posts will demonstrate the Adjusters, Matchers, Periods, Formatters and Parsers. Once the tests are cleaned up a bit, I'll post them on line.

Java Date/Time Objects with JSR-310

2009-02-14T17:31:00.000-08:00

I spent the day experimenting with the proposed javax.time API's from JSR-310. Modeled after the popular joda-time the new javax.time objects offer a rich set of date and time manipulation, basic math, durations, matching and time zone awareness.

At the top level, the concept of a TimeSource is introduced. This is a replacement for using System.currentTimeMillis() or System.timeNanos() or the standard java.util.Date and Calendar contructors. It can be spring loaded, so getting a source that has an offset is possible, which makes testing very easy.

The abstract Clock class is similar to TimeSource, but it is time zone aware. It's easy to get the local time (Clock.systemDefaultZone()) or to create a clock from a different time zone (Clock.system(TimeZone.UTC)).

Once the Clock object is created you can access today(), tomorrow(), yesterday(), the current DateTime and other methods for creating complex DateTime objects. You can also get the current instant which represents the current date/time as an instant since the epoch in nanoseconds.

The first step to using javax.time is to checkout the latest version from subversion. (I'm at build version 702).

Converting Date and Calendar to javax.time

A common way to construct both Date and Calendar from the java.util package is to use milliseconds. The Instant class from javax.time includes a method called toEpochMillis() that makes this easy: (groovy syntax)


utilDate = new java.util.Date()
instant = Instant.millisInstant( utilDate.time )
zone = ZoneOffset.zoneOffset( -8 )
dateTime = OffsetDateTime.dateTime( instant, zone )

Converting javax.time to Date/Calendar

To convert a javax.time object back to either Date or Calendar is just as simple:


ts = TimeSource.system()
dt = new Date( ts.instant().toEpochMillis() )

cal = Calendar.instance
cal.timeInMillis = ts.instant().toEpochMillis()

Date Math

Here is a quick example of what you can do with a DateTime object. In this case, I'll use an OffsetDateTime to express the full ISO8601 date. (again, groovy syntax)


clock = Clock.systemDefaultZone()
assert clock.today().toString() == "2009-02-14"
assert clock.tomorrow().toString() == "2009-02-14"

dt = clock.offsetDateTime()
assert dt.toString() == "2009-02-14T20:31:50.896-08:00"

dt.plusYears(3).toString() == "2012-02-14T20:31:50.896-08:00

assert dt.year == 2009
year = dt.toYear()  // get the object, not the int

assert year.lengthInDays() == 365
assert year.next().value == 2010
assert year.nextLeap().value == 2012
assert year.nextLeap().lengthInDays() == 366

yearMonth = YearMonth.yearMonth( dt )
assert yearMonth.toString() == "2009-02"
assert yearMonth.plusMonths(5).toString() == "2009-07"
assert yearMonth.minusMonths(5).toString() == "2008-09"

So you get the point. Many useful methods currently missing from the java JDK. So, hopefully this will be complete in time to make it into java 7. In the mean time, though not ready for production, it's available for bleeding-edge use.

A Groovy Cryptology Library

2008-12-22T16:23:00.000-08:00

Rain City's open-source groovy library has recently been enhanced to include a crypto package. Implementation includes symmetric and asymmetric algorithms, hmac, digest, guid, key pool and key store utilities. This post will explain the main classes and show a few examples.

Guid: The Guid class is a simple wrapper around java's UUID class. It makes it easy to create a new guid or read an existing guid string to create a guid object.


println new Guid().toString() -> e79a8462-32ad-48db-9905-3d53e1471de3

SaltedSecureRandom: SaltedSecureRandom provides a salt-seeded SecureRandom object using hardware dependent /dev/urandom. This works for all non-Microsoft-windows platforms. On windows, a simple secure random object is returned. (If you are serious about security, don't use windows).

Digest: This class is a thin wrapper around the JCE digest algorithms, including SHA-512. The class defaults to SHA-256 to maintain compatibility with other languages and platforms. MD5 is also available. The default provider is Sun. The class also provides a 'main' to enable checking digests from the command line.

ByteContainer: The ByteContainer class has utilities for converting strings to by streams, base64 bytes and hex strings. Invaluable for exchanging encrypted data.

Hmac: The Hmac class supports SHA-256 and MD5 and provides methods to set text and key to compute and render keyed digests. It also includes a key generator that defaults to SHA-256 algorithm.


def bc = new ByteContainer()
def k = Hmac.generateKey()

bc.bytes = k.encoded
println bc -> 85a749fd3818fccf0dc6997b6c9f25129babbf6b98606f8b51a14e68b5551fa6

Crypto: This is the primary class to use when you need asymmetric encryption using well known algorithms from various providers. The default algorithm is AES using the Bouncy Castle provider with PKCS7 padding. Other supported algorithms are Blowfish, DES, Tripple DES, and all the others supported by the Sun provider.


def crypto = new Crypto()  // requires bouncy castle

assert Security.getProvider("BC")

def pw = "my password".bytes
def key = crypto.createKeySpec( pw )
assert 'AES' == key.algorithm

def text = 'this is a test of plain text'

crypto.generateIV()
println crypto.ivSpec.getIV()

def enc = crypto.encrypt( text, pw )
def s64 = enc.encodeBase64().toString()
println s64 -> q7teBCCBK+n3qTSr518QIj04qa/cS/P2bhsUw/eg8a8=

def dec = crypto.decrypt( enc, pw )
def s = new String( dec )
println s -> this is a test of plain text

crypto.clearBytes pw
println "${pw}" -> [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

RsaKeyPool/RsaKeyStore: 2048 bit public/private keys use very large random prime numbers. So generating secure key pairs takes lots of CPU cycles and a fair amount of time. To alleviate any performance bottlenecks, the best approach is to create a key pool to serve up single use key pairs. The pool actually gets keys from a key store that can be built off-line to any desired size. This approach means that a server won't be bogged down initializing the pool when it first comes up (or is bounced).

The actual generation of the key pairs is through a JCE KeyPairGenerator using the Bouncy Castle provider. It's initialized to the key size (defaulting to 2048 bits) and uses the salted secure random class mentioned earlier.

Rsa: The Rsa class provides encryption and decryption of data using public/private keys. For the uninitiated, the main thing to remember is that the public key is used to encrypt and the private key to decrypt. The Rsa class makes this a bit easier by providing encrypt() and decrypt() methods. A simple use case example goes like this:

Alice wants to send an encrypted document to Bob. She begins the process by requesting a public key from Bob. This is what will be used to encrypt the document.
Bob creates a key pair (or gets a pair from the pool) and sends the public key back to Alice.
Alice creates a Rsa object and sets the public key to Bob's key. The document is encrypted then sent to Bob.
Bob receives the encrypted document and decrypts with Rsa using his private key.

Now lets take this simple use case and implement some of the parts. For our purposes, we will use two classes Alice, the sender and Bob the receiver.


import org.raincity.glib.crypto.*

class Alice {
def encrypted
def send = { bob, text ->
def key = bob.getKey()
def rsa = Rsa.instance
rsa.pubKey = key
encrypted = rsa.encrypt( text.bytes )
bob.receive( encrypted )
}
}

class Bob {
def text
def keyPair

def getKey() {
keyPair = RsaKeyPool.instance.keyPair
return keyPair.publicKey
}

def receive = { encrypted ->
def rsa = Rsa.instance
rsa.privKey = keyPair.privateKey
text = new String( rsa.decrypt( encrypted ) )
}
}

alice = new Alice()
bob = new Bob()
alice.send( bob, "This is a super secret message" )
println "encrypted=${new BigInteger(alice.encrypted).toString(16)}"
println bob.text

And when you run the script you see...


encrypted=496bc13c9ba194a2becc525839bab8d9c40a7c126a1ac32f0aa21...
decrypted=This is a super secret message

Future posts will discuss how these classes are used to support Adobe Flex to Grails encrypted exchange sessions. If you are interested you can download the library and examples in a single jar file here.

Groovy Excel Report Library

2008-08-01T16:00:00.000-07:00

I just finished a series of reports for one of my customers using grails as the backend and Flex for the user interface. As with many of my reports for this customer, they requested the output be delivered as CSV files. Easy, and very boring.

So, with the help of JExcel and the jxl APIs from Andy Khan, I created an alternate output of an excel spreadsheet rather than CSV flat file. This has the additional benefit of sending multi-dimensional results, e.g., summary and details on two separate sheets, output statistics and charting on a third. This is much more useful for the end-user.

To make life easy, I created a wrapper around the jxl libraries to make it possible to use an alternate excel API such as Apache POI. At the same time, I created a set of conventions that use a template spreadsheet to easily create all the output without having to program any of the "view" components (i.e., classic MVC). The controller class is Workbook and the Model data is configured in Worksheet objects. All the formatting for dates, numbers, layout, etc. is in configured in templates (xsl files) easily configured and modified by non-programmers.

Since the target for this is inside war-deployed web applications I had to configure an external templates folder to enable report format changes without having to re-war/deploy or bounce the server. The templates folder is part of a more extensive Exchange class that coordinates remote data exchanges to a known area outside the web-space.

Conventions: Below is a list of convetions used to build reports...

xls template contains all formatting and sheets for the report
key words embedded into the template provides cell locations where data is to be written. values stored in the Worksheet object are used by the Workbook controller to update the spreadsheet output.
a special type of keyword called "ReportData" is used to insert multilple rows into the spreadsheet, based on query result sets (or any list of data).

The Worksheet models also have a place holder for "options" that change/enhance the controller's behavior as it builds output. Examples include "greenbar" to create green-bar reports, "showItemNumbers" to write out the actual row number to the left of the ReportData.

Working Example: To demonstrate the report library I created a "Customer Invoice" report. The template has three sheets, includes graphic images, and shows two of the three "ReportData" sets in green bar (and blue bar) with automatic item numbering.

The Code: The full report can be built and run with just a few lines of code. Here is a typical example that prints the invoice, details, and expenses for a single period.

The three sheets are defined as details, expenses and invoice. Invoice details and expense rows are fetched for a specific period with GORM in the typical way. Report fields are extracted to match the template report data layout. The main invoice sheet has a few extra named values (line 28-31) and constructs two lines of data (lines 33 & 34).

Report Output: Here are the three sheets created by the code above.

Dynamic Grails Application Configuration

2008-07-26T12:03:00.000-07:00

If you deploy your grails applications with war files you may find it necessary implement an application configuration mechanism that lives outside the web-space. My first though was to use and external file, probably xml and read it in when needed. But, I decided that persisting configurations to the database provided a much more universal solution. It also avails itself to adding a UI or acting as a remote web-service.

The Domain Object, AppConfig is very simple. It has a name, parameter, value and status:


class AppConfig {
 String name
 String parameter
 String value
 String status
}

The "name" and "parameter" columns uniquely identify the configuration setting and the "value" is a string representation of the parameter's value. The status can be anything, but I find it useful to set the target environment, e.g., 'dev' or 'test'. I usually use the class name as the configuration name, so looping through a set of parameters is as easy as this:


AppConfig.findAllByName( this.class.name ).each { p ->
 switch(p.parameter) {
   case 'enabled': enabled = (p.value == 'true' ? true : false)
   case 'other': other = p.value
   case 'anInt: myInt = Integer.parseInt( p.value )
 }
}

I use a standard dataset loader to read in and insert settings on application startup taking care not to clobber existing values. Configurations are read in during development, test and production.

A typical use is to set parameters for Quartz Jobs. This way, the parameters are read each time the job starts to dynamically configure actions. The most common is to enable or disable the job. This comes in handy if a job goes bad but you can't bring down the web server.

Groovy Polymorphics with Closures

2008-07-16T21:30:00.000-07:00

Let me begin by stating the problem: I need an object that can locate files in a specific folder, parse the selected files, then execute database updates against the parsed results. The twist is that the update process might be inside a grails application using GORM, or, it might be a standalone jdbc process that has it's own data access mechanism. Here is the proposed object:


class Process {
 def findFiles() { /* return a list of files */ }
 def parseFile(file) { /* parse the file, return the data */ }

 // the update function (closure)
 def update

 // the runner
 def run() {
   def list = findFiles()
   records = []
   list.each { records << parseFile( it ) }    
   update( records )  
 }
}

Forget about the empty methods for now, let's just assume they work. Notice that run() finds the files, parses them, then updates them. Here is how the target object is used:

def process = new Process()
process.update = (isGorm ? gormUpdater : jdbcUpdater)
process.run()

The first thing you might notice is that 'update' in the Process class has no body--it's simply a tag for the closure name. When the class becomes an object, update is assigned the closure method based on the runtime environment. Here is how a very simple GORM closure would be defined:

myGormUpdater = { records ->
records.each { row ->
  row.save()
}
}

Simple. Loop through the records and save each one. The polymorphic magic is implemented with a closure, not a sub class. This eliminates class bloat by using portable stand-alone methods assigned to normal variables of a class--like named function pointers. An added benefit is that the closure can be used on similar classes that require the same functionality.

Real World Scenario: Lets say I have a small group of software engineers. Two of my engineers are data-feed experts, another three are database domain experts. Me, well, I'm the manager.

So, since I have the big picture, I design the base class and create test fixture data. Then, I tell my feed experts to give me methods for parsing a set of files. They don't have to worry about where the files are, just how to parse and create datasets. At the same time I assign the database ops to the database group. They don't have to concern themselves with where the data comes from, just how to convert datasets into database updates for all our target databases.

Even though I'm a manager, I contribute in a real way to the code base. And, to keep my team productive, I create the test data for the parser and database groups. The experts are assigned tasks that they handle best.

Combine this with daily scrum communications and, for the hands-on CTO, this is very Groovalicious!

Flex on Grails

2008-07-15T09:00:00.000-07:00

Borrowing from Flex/Rails libraries created about a year ago, I have created a Flex/Grails application using similar HTTP/XML protocols. The application requests report data via HTTP, the server creates the report, persisting it to a file, then the client pulls the data back with a file request. This was the only way I could get Flex/Flash to open a "file-save" dialog to let the user write to their local hard-drive.

I used Flex's HTTPService object to do the initial report request, then FileReference with additional URLVariables parameters to retrieve the file. Grails/GORM made the data retrieval easy and the controller/service packages were simple implementations. This was comparable to Flex/Rails implementation that I worked on a year ago.

The server side runs in a war distribution so the output file had to exist outside the webspace. The Flex application requests the file with an HTTP post using URLRequest with specific parameters (URLVariables) to retrieve the file. Because the server is deployed with only war files, the request is made to a specific controller with a "getfile" action that retrieves the file from someplace on the server's drive (obviously not in the war file) and is consistent with where the file was originally written. Once the file is returned to the client it is removed, so it can only be read back a single time.

This first iteration returns a CSV file, but creating a full xls spreadsheet or a PDF document would be easy to implement. There is also an xml document that can be returned to enable showing the report in a local viewer, e.g., a data grid.

Replacing GORM with Groovy Sql for Lightwieght Jobs

2008-07-10T10:00:00.000-07:00

GORM, the Grails Object-Relational Mapping persistence framework provides a rational way to use Hibernate based on pojo (or groovy pogo) domain classes and basic conventions. Create a domain class, define some constraints, start up grails and viola! tables, columns, indexes are created without coding a line of SQL/DDL or creating annoying XML mappings.

But, what if you have a simple standalone process that requires single or multiple database access? Currently GORM has trouble working outside of grails, and creating a new grails project just to support a simple, no-client script is overkill. For that, I use groovy's database access classes in groovy.sql and a small custom library to easily access multiple data sources and provide simple query tools optimized for batch operations.

Here's an example: Tracking multiple UPS shipments. The application must access existing shipment requests from a local database, then query the UPS tracking remote service to discover shipment status. When a shipment is picked up or delivered, another local database is updated and a third system is updated via XML feed.

To summarize, the requirements are:

provide simultaneous access to multiple databases
query UPS tracking via HTTP requests
create XML responses and write to a messaging system

Cron triggers this process multiple times during the working day. It runs on the production machine (1MB slice at slicehost), so should be kind to existing web and database applications serving the user base.

Design Implementation:
As always, I start with the tests. Using groovy test case, the database queries, inserts, and updates tests were created, then the target classes and methods. Queries are always the easiest, and updates are straight forward as well. Inserts with auto generated IDs are a bit tricker, but here is where groovy sql comes to the rescue.

Creating new tables presented a new problem: should I create SQL/DDL scripts? No! For this, it was easy enough to access the current grails application and create the new domain classes, run grails to have Hibernate create the new tables.

The groovy Sql class provides a method called executeInsert(). The magic in this method is that it returns an array of all auto-generated keys. So, if I'm working in MySql or Oracle or whatever, inserts all look the same--no database specific code required.

Database access defaults to the production environment but database tests must operate on the test database (similar to grails). This is easy to accomplish by simply overriding the default behavior in the test scripts.

Conclusions:

Use groovy Sql for small, non-web applications that require single or multiple database access.
Use Grails/GORM for all database/table creation and maintenance (no DDL scripts). This also has the additional benefit of enabling access to new tables through grails if required.
Use executeInsert() to return auto-generated keys when inserting new rows.

MySQL InnoDB Dialect

2008-05-09T15:39:00.000-07:00

I missed a configuration detail in conf/DataSource.groovy prior to going to production on a recent grails project. I didn't notice the damage until I looked closely at the backup script created by mysqldump. The re-build script called out ENGINE=MyISAM rather than InnoDB. So, no transactions, cascading, or other basic features.

The Fix: To correct the problem, I inserted the following into the hibernate section:

dialect='org.hibernate.dialect.MySQL5InnoDBDialect'

When I ran my tests, there were a couple of changes that need to be made mainly because transactions were now working correctly. The next step was to repair the production database.

The Repair: The sql rebuild script needed to be modified to replace MyISAM with InnoDB--a perfect job for groovy. Here is the script:

#!/usr/bin/env groovy
file = new File( 'production.sql' )

new File('fix.sql').withPrintWriter { writer ->
file.eachLine { line ->
  writer.println(line.replace('MyISAM', 'InnoDB'))
}
}

Simple to implement. Very groovy, and now with transactions!

Alternate Logging in Grails

2008-05-08T18:58:00.000-07:00

For most cases, the Grails logging system, based on log4j, works fine, especially for development. But when in production, if you deploy using a single jar, it gets a bit trickier when trying to enable and disable logging. So, to that end, I have created a quick and dirty logging system using groovy's advanced file features.

Groovy Files: In java, log files are simple to configure, but you end up writing the same old try/catch/finally code--very old style. In groovy, all you need is a file name and you are good to go. Here is a groovy example:

def file = new File("test.log")
file.write("this")

No try/catch. No worrying about opening or closing. Just name it and write to it. Very groovy.

Applications Log Files: Each application has it's own specific configuration. So, in the grails-app/conf folder is the perfect place an for ApplicationConfig class to hold the application's configurations. Inside this class are a few methods that define the external logging system. The methods include getLogDirectory(), getLogProperties(), getLogfile(name), etc. For our purposes, getLogfile() is the only method we need to invoke. Again, an example:

def logger = new ApplicationConfig().getLogfile('my-service.log')
logger?.append("my message")

And that's it. So, logger? only appends the message if a log file exists. The log file is external to the web container, and easy to create (touch my-service.log) or remove. If it exists, the it gets logged to. If not, then no logging occurs. The sys-admin, armed with a list of available log file targets, can create or remove logs on the fly without touching the web server. Again, very groovy.

Grails Application Deployed to Slicehost

2008-03-06T07:00:00.000-08:00

We deployed our beta application on slicehost virtual server. The slice is 1GB using fedora 7, mysql 5.1, jetty 6.1.6 and resin 3.2. Groovy version 1.5.4 is installed on the slice and we used Grails 1.0.1 for development, but it's not installed on the slice.

The application at this point is minimal, but is scheduled to go live later this month. I can't go into to much detail, but application provides a data service between United Parsel Service and a major customer to schedule shipments of large and small freight.

Groovy DateTime Math

2008-03-04T11:58:00.000-08:00

DateTime is capable of simple increment (++), decrement (--), and all the add/roll methods provided by Calendar. Here is an example of some of the ops:

refDate = new Date()
dt = new DateTime(date:refDate)
tomorrow = dt + 1
yesterday = dt - 1

assert refDate == dt // insure that the original did not change
assert refDate + 1 == tomorrow
assert refDate - 1 == yesterday
assert refDate + 7 == nextWeek

Groovy Equality: Groovy evaluates '==' as equality, not identity. So it was necessary to override equals() to compare to the millisecond and return true or false. The override makes it possible to return true when the underlying milliseconds match whether from Date, Calendar, Long, or DateTime.

Problems with Milliseconds: When working through the plus and minus methods I made the mistake of using the raw milliseconds to add days. This worked fine on my linux machine, but mysteriously died on OSX. What I neglected to take into consideration was the switch between standard and daylight time (luckily my development cycle is close to the change or I would have missed it). So, a quick fix was to rely on java.util.Date to add days and return a new DateTime copy. A better approach may be to use groovy's Duration classes...

Durations: The groovy library includes a set of duration classes in the groovy.time package. The duration classes are containers for years, months, days, hours, minutes, seconds and millis. They support basic math functions to add and subtract durations--very groovy. DateTime uses the duration classes for basic math with the following methods:

now = new DateTime()
current = now.asDuration()
twodays = new Duration(2, 0, 0, 0, 0)
dt = DateTime.fromDuration( current + todays )

Combined with groovy's TimeCategory, you can do this:

use(TimeCategory) {
  future = new DateTime(date:2.weeks.from.now)
  oneWeek = 1.week  // this is a duration

  // schedule something once each week for the next 5 weeks...
  dates = []
  5.times { dates << it.weeks.from.now }
}

Additional utility methods and parsing to come next...

Groovy DateTime Comparisons

2008-03-04T10:30:00.001-08:00

Those of you following my posts know that I'm working on enhanced date/time support for groovy. In a previous post, I discussed a new DateTime class that leverages the power of java's Calendar and updates the interface in a groovy way. Which brings me to after(), before(), and compareTo() methods.

Calendar implements compareTo() that accepts only a calendar object. If you pass it a Date and exception is thrown. How restrictively lame.

DateTime's compareTo() accepts multiple types that can be evaluated to the millisecond level. So, DateTime, Date, Calendar, even long types are accepted. This approach enables this to work:


now = [ new Date(), Calendar.getInstance(), System.currentTimeMillis, new DateTime() ]

yesterday = new DateTime() - 1
tomorrow = new DateTime() + 1
later = new DateTime(hours:23, minutes:59)

now.each {
 assert yesterday.before( it )
 assert tomorrow.after( it )
 assert laster.after( it )
}

DateTime. How groovy is that!

Groovy Temporal Support Continued

2008-03-04T09:33:00.000-08:00

One area where ruby outshines groovy is in date and date time support. Ruby's date time support isn't more capable, just easier to use. Java, and groovy have Calendar--very complete, and very clunky. There is also joda time. Very complete, but a bit of an overkill for my tastes.

So here are the objectives:

ability to add/subtract dates with integers representing days, months, etc.
ability to subtract two dates to yield intervals representing days, hours, etc.
ability to compare two dates
ability to parse any type of date input, and most easily parse the standards
ability to format dates in a wide variety of ways
ability to create dates with integers and words like 10.days.ago or nextweek

Java's calendar class supports the first three requirements, parsing is available through SimpleDateFormat class. Groovy's TimeCategory enables some of the most common integer enhancements to create dates based on common temporal works. So, the pieces are in place, the next step is to wrap this in a groovy way.

DateTime, the Groovy Calendar: The first step is to create an easy (easier) to use, full featured DateTime class. One approach is to simply extend GregorianCalanedar and add the new methods, mostly getters and setters. That's were I started but soon discovered a better solution was to contain the calendar object, and use invokeMethod() to pass methods and arguments. Then, rather than define properties of the DateTime object, define getters and setters in bean style making the variables appear as if they were members. For example, I define these two methods:

int getYear() { return calendar.get(Calendar.YEAR) }
void setYear(int yr) { calendar.set(Calendar.YEAR, yr) }

Now, I can use short hand to access "year" like this:

def dateTime = new DateTime(year:2008)
assert dateTime.year == 2008
dateTime.year = 2020
assert dateTime.year == 2020
dateTime.year++
assert dateTime.year == 2021

Groovy, right? And adding other methods like setSeconds(), clearTime(), isWednesday() were easy one-liners to implement. I also took the liberty of shifting the month to 1..12 rather than 0..11, the calendar default.

Constructors: This was also very groovy. After I added the getter/setters, constructing with the virtual member vars was easy. So the DateTime class can be constructed like this:

dt = new DateTime(year:2020, month:4, day:1)
dt = new DateTime(hours:0, minutes:0, seconds:0)
dt = new DateTime().clearTime()
dt = new DateTime() + 5

Date/Time Formatting: The Rails group enhanced ruby's DateTime.to_s() method by adding formatting like this: to_s(format). This makes a lot of sense. To implement this I use a format hash that stores formatting strings by name. The map is static to enable use application wide (I may regret this later) and there is a default format member variable that controls how toString() is formatted. I also added toString(format) to display a date in multiple formats like this:

// default format is a modified ISO-8601 called 'db'
dt = new DateTime(year:2010, day:25, month:4, hours:15, minutes:35, seconds:29)
assert dt.toString() == '2010-04-25 15:35:29'
assert dt.toString(8601) == '2010-04-25T15:35:29-0800'
assert dt.toString('mdy') == '04/25/2010'
assert dt.toString('dmy') == '25.04.2010'
assert dt.toString('dMony') == '25-Apr-2010'

Other formats are available in the formats hash. The actual formatting is done using SimpleDateFormat that supplies a wide range of date formats and is compatible with Java 1.4, important in the groovy community.

A Groovy Date Parser: I've seen many approaches to this, usually making use of java's SimpleDateFormat class with a boat load of formats. I think there may be a better way. First, extracting the date parts, usually the numbers and constructing date time objects based on the numeric values. I'll discuss this implementation later...

Groovy Temporal Support Classes

2008-03-02T11:08:00.000-08:00

Groovy adds a rich set of features to java including closures, ranges, dynamic metaClass, access to existing java code base, etc. What is missing is full featured date processing. One approach to fixing this is to use existing libraries of java date extensions, such as joda time. Today I decided to implement some basic date/time features without a third party dependency. I may end up using joda as the project unfolds, but for now I'm trying to resist the temptation.

HourMinuteSecond: This class acts as a container and formatter of hours, minutes and seconds similar to groovy's TimeDuration class. In native form it's completely mutable, so not thread safe. But it can be made immutable by invoking asImmutable(). I'll explain how groovy makes this easy to implement later on.

The class API is as follows:

static HourMinuteSecond now()
static HourMinuteSecond fromDate(date)

def parse('hh:mm:ss')
def set(hours, minutes, seconds)
def set(Date dt)
def next()  // enables ++
def previous()  // enables --
def add(hours, minutes, seconds)
def sub(hours, minutes, seconds)
boolean isZero()
String toString() // formatted as hh:mm:ss
void addZeroEventListener(closure)
def asImmutable()

The object can be used as a simple up/down timer measured in seconds. When attached to a one second ticker thread, only the next() method needs to be invoked to tally up the seconds. Used as a count down timer, an event is triggered when zero is reached.

The methods are actually closures, so creating the immutable clone was as simple as pointing all the mutators to a single closure that throws an UnsupportedOperationException when a mutating method is called. This mimics how groovy's List.asImmutable() method works.

I will post more as the temporal project unfolds, and supply a link once the library is ready for distribution.

Grails Dataset Loaders, Part 2

2008-02-07T21:02:00.000-08:00

About a month ago I posted a few thoughts on Dataset Loaders for Grails. Now that one of my Grails projects is going into beta, I thought it would be a good time to revisit the issue.

Rails History: With rails I used fixtures for test data and had to hand code data loaders for production. I put the loaders in lib/tasks. They were rake tasks that did the initial loading and/or refreshing of data for a specified environment. So data loading for a production or development was a manual task. Not a huge burden, but just one more thing to remember.

Now that I'm in the Grails realm I have created automatic loaders that are environment aware. A DatasetBootStrap class loads data for development, test, and production whenever the server starts. This is in line with what hibernate does when it starts up--verify that the domain models match the current data source, and make changes where necessary. In rails this is done with migrations--a good idea, but yet another manual step.

DatasetBootStrap: The DatasetBootStrap class first determines the current environment, then loads the appropriate dataset classes and invokes the "load" method (or closure if you will). A list of datasets is configured manually for each environment. This approach makes it possible to share loaders for all three environments (or four if you use staging). Some data is just for test. Other data, like a list of US States, is loaded for all three environments. The main point is nothing is repeated, keeping my code as DRY as possible.

Datasets: I currently keep my datasets in src/groovy/datasets, but I think that it's time to move this folder into grails-app/conf/datasets. I also use a package declaration (package datasets) to insure that dataset classes don't conflict with other classes (I've had problems with this in the test structure when trying to create unit tests that have a name conflict with integration tests).

Simulating Rails-Like Test Fixtures: This was the easy part. In each test that requires a specific dataset, the setUp() method is used to load as many datasets as necessary to create a solid test environment. Loading all the data is as easy as calling the DatasetBootStrap's init() method.

Grails Integration: My goal for Datasets is to propose it's use for all grails projects by integrating it into the grails core. I have a bit of refactoring to do before this is practical, but would welcome any comments or suggestions from other grails (or rails) users as to what features a Dataset loader module should include.

UPS Web Services Interface

2008-02-06T19:57:00.001-08:00

One of my customers has a requirement to use United Parcel Service to ship large and small packages. UPS has a great set of documentation, but they handle large and small packages with different technologies. Small packages are simple XML request/response transactions that provide methods for requesting a shipment, accepting shipments, and voiding shipments. Large packages allow you to request a shipment, but the request/response is all in SOAP, something of an overkill in my opinion.

UPS requires lots of data concerning who, what and where to establish their pricing. Data is sent to a URL/end point and a response is returned. The response includes pricing and tracking numbers, as well as image data for printing barcoded shipping labels. So, it is mainly a data transfer operation-- a single service.

SOAP is great if you have lots of defined services, like weather in Seattle or Paris or zip code 94705. Or conversion of units from/to. But for simple data transfer, REST, or just plain old XML request/response is much more efficient.

Here is a diagram of the complete data transfer...

...

As you can see, there are lots of communications. The first question is, why is there a server in the middle? Well, here's the deal--the SunReturns client is just that, a client. The data really resides at the DataServer--the client simply persists it's own private session. The UPS server is at their site. All communications are via SOA/services--some simple request/response (ala REST) and others use SOAP.

The technology used in the middle is grails/groovy. The SunReturns server is currently a mixture of JSP (legacy) and groovy/GSP. The groovy components are XML only and include controllers, services, domain objects, etc--similar to grails but without the Hibernate layer. It talks to legacy databases, both Oracle and MySQL--not impossible with grails, but definitely outside the conventions.

Conclusions: The natural loose coupling of SOA provides a new vehicle to solve business problems. At the same time, distributing business processes across diverse servers provides a workable migration path to newer technologies. Groovy, grails, and simple XML transport provide a way to implement SOA with a minimum of effort.

Business Service Components

2008-01-16T17:36:00.000-08:00

Every business application has a set of commonalities that lend themselves to standardization. In the rails, and to some extent grails world, this translates into plugins. The restful authorization plugin being a good example.

But, what if rather than a plugin, you had access to a software service that provide the same functionality? Here is a comparison:

Using a business related plugin:

download and install the plugin into your project
build out all the required tables
hookup your internal logic to insure that the plugin loads correctly
use the plugin

Using a Software Business Service:

use the service

The choice should be obvious. Rather than dealing with database migrations, version control, broken or non-existent tests, integration into your pristine project, you simply use the service. A good example is the captcha service called Captchator from Andreas Schwartz. This service replaces logic that traditionally comes in the form of a plugin with a callable service that does what a captcha component needs to do. No downloads required. No migration of new tables. Just use the service.

Now imagine a full suite of services for business processes. Authentication, company and contact information, time tracking, invoicing, fulfillment--all discrete services that a VAR or software developer can simple use as components in a vertical solution. Wouldn't that be cleaner than installing plugins?

Git for distributed backups

2008-01-05T09:12:00.001-08:00

I usually use svn for code and document backups but today I decided to use git to automate my database backups. Git seems to run much faster and offers a bit more security. It also functions as a good transport for multiple distribution to remote sites.

The backups are in ~/backups so I created a git repository and moved it to my remote slicehost site. Here are the steps:

cd backups
git init
git commit -a -m 'initial repo'
cd ~
git clone --bare ~/backups backups.git
scp -r backups.git dpw@raincity.slice.com:

The next step was to move the backups.git folder to /public/git and chown -R git:git. Then to read the repo back:

mv backups /tmp
git clone ssh://git@raincity.slice.com/public/git/backups.git

At this point I have local and remote repos. I went to other machines and cloned the repo to insure that I have multiple locations on and off site. The next step was to hook git up to my standard backup from the master database. At the tail end of the SQL backup script I do this:

git add .
git commit -m "automated backup on `date`..."
git push

And there you go. This provides a local and remote backup. Configuring cron on other machines is as simple as doing a 'cd ~/backups ; git pull'. Life is easy...

Survey Project--A Grails Implementation

2008-01-05T06:45:00.000-08:00

I started small grails application today to enable creating surveys and questions and tracking questions and results. The domain classes are:

Survey: the primary table that defines a survey project
SurveyQuestion: object to hold a specific question mapped to a survey
SurveyResponder: the person that is answering the survey questions
SurveyResponse: the recorded answers that the reponder enters

And here is the proposed ER diagram, created using dbDesigner.

As the diagram shows, surveys have many questions, questions have many responses. Surveys also have many responders and responders have many responses. You can also see that this is a very simple survey with no branches.

Grails/GORM Implementation:
I began by generating the Survey and Survey question domain classes and basic validation tests. Next was to create the SurveyReponder and SurveyResponse classes. The implementation was straight forward, but I had to fiddle with the domain class definitions to get hibernate to understand my intent. Just a matter of specifically defining class references in dependent models rather than depending on the 'belongs_to' and 'has_many' declarations.

Datasets and Loaders (Fixtures):
Unlike Rails, Grails doesn't use fixtures for test data. Lucky for me I have a Grails enhancement the solves this based on a sandbox proposal from the Grails guys. So I generated the dataset loader classes using: grails create-dataset [domain]. This creates the following:

a DataLoaderBootStrap class in grails-app/conf/ (unless it exists)
a DomainDataset class in src/groovy/datasets
a set of random data to be loaded

An example of what gets created is here:

> grails create-dataset survey
> ...
> file grails-app/conf/DataLoaderBootStrap.groovy exists...
> created file src/groovy/datasets/SurveyDataset.groovy
>

The SurveyDataset class looks like this:


class SurveyDataset {
dataset = {
def set = [
[ name:'879',description:'510',email:'680' ],
]

return set
}

load() {
dataset().each { data ->
obj = new Survey( data )
obj.save()
}
}
}

As you can see, the data is contrived but all of the fields are present and other than the actual data, the script is good to go. I'll show how this data can be loaded from the console, command line and within test scripts in a later post.

Grails Dataset Loaders, aka Test Fixtures

2008-01-04T18:30:00.000-08:00

One thing I've been missing since working in Grails is test fixtures ala Rails. The first steps I follow when developing data models is to create test data, which is fairly easy in Rails. For Rails I had to do some extra work to get this data to load into the development db and I also had to augment the yaml files with ERB to generate random data.

The Grails sandbox area has a thread that proposes creating test fixtures that begin in late 2006. As the conversation unfolded, there was a decision to use the term Dataset rather then Fixture--a welcomed change. But the thread kind of died as of October of 2006, so I guess I need to implement this on my own. Here's my attempt...

Datasets and Loaders (Fixtures):
I followed the sandbox proposal and created Dataset classes rather than yaml files. Closures were an obvious choice to create datasets, but I also had some other requirements. My objectives were:

datasets are usually a single set, but may be separated by dev, test, and production
datasets are implemented as closures to enable generating random data
dataset loaders may be separated into development, test, and production
they can be loaded from within the application on start up
they can be loaded on demand from test classes (similar to calling out fixtures)
they can be loaded through scripts (ant or grails)
they can be loaded from grails shell and console

For all environments a DataLoaderBootStrap class controls loading when the application starts (grails run-app). The bootstrap is environment aware, and loads specified datasets from individual Dataset classes, e.g. UserDataset, CountryDataset, etc. based on the current environment.

The datasets are classes that have a dataset closure that defines hash maps of data (not model classes) and a load() method to do the actual database inserts (or updates). The classes are placed in the src/groovy/datasets folder. Their is a load() method that creates objects from the hash set then invokes "save" to either insert or update depending on the state of the database. This has the benefit of creating predictable data without having to continually drop or trucate tables (ala rails).

When testing in the console, the loaders can be run manually. Test scripts can invoke loaders at any time to insure that there is predictable data. A command line script can be invoked to load all data for the given environment using "grails [env] load-data".

I thought of creating a plugin for this but it doesn't really fit the plugin framework. So, for now I plan to simply keep it as a platform enhancement.

Web Containers on SliceHost

2008-01-02T09:14:00.000-08:00

I have a small virtual server at slice host (256M). I originally thought it would be appropriate for rails/mongrel and apache, but probably too small for a java web container. But, after installing resin and running it through some basic tests, it looks like even the small slice is fine for testing and semi-live (e.g. demo) applications.

My next step is to test jetty and glassfish. I don't think there will be problems with jetty, but glassfish may be too much of a memory hog to work correctly.

Running 'top' shows that resin actually uses about 60M when it starts up and goes up to 90M under basic use. This will undoubtedly change when spring/hibernate and real database access kicks in, but it looks good so far. My goal is to install a grails application to see if slice host can be used for basic demo purposes.

10 Reasons to Switch from Rails to Grails

2007-12-30T18:30:00.000-08:00

After spending a few years really enjoying Rails it was difficult to bring myself to even try groovy and grails. But my latest contract forced me to look for alternatives, and I'm glad I did. Here are some reasons that you may want to switch...

GORM with hibernate/spring and jpa is much better than ActiveRecord
No distribution problems; runs in many production ready containers
Internationalization out of the box (not specifically ignored as DHH does)
Transactions actually work, and they include save-points.
Not only dynamic finders and counters, but dynamic listOrderBy
No green threads (although this may be fixed in Ruby 2.0, around 2010?)
Ability to use pessimistic locking out of the box
Real-live prepared statements and .withCriteria method
Production level test reporting with built in Mocking and Stubbing
Search operations are based on Lucene (with a plugin)

All of these don't make sense for a non-java coder. And my startup time for grails would have be much longer without my prior experience with Rails and Ruby.

One thing I thought I might be giving up was the endless list of valuable gems that make developing in Rails a real pleasure. But after re-reviewing the list of open source products from apache, java source, spring, hibernate, and even Sun (glassfish), I can't think of any gems that I will miss.

Grails Application Development Notes

2007-12-30T16:10:00.000-08:00

I started a grails application for Sun using the latest version of groovy (1.5) and grails (1.0-RC3). Things went fairly smooth but I was glad to be under 'git' control to rollback branches as the project developed. My first problem with with mixed case characters in domain and controllers...

I originally created an IBBUser domain model. Ok, I should probably re-name it IbbUser but old habits die hard. Anyway, the model generation went fine, but when I went to create the controller using generate-all, that's when things fell apart.

I'm not sure why (yet) but 'generate-all' created my views in an IBBUser folder. This was a problem when I ran the app, so I scrapped the project and re-generated with the domain name of 'user'. I probably could have used 'ibbuser' in lower case, but oh well...

Working with generated code in grails or rails can get out of hand quickly--so many files and folders to track. So I set up branches--one for domains, the other for controllers. Before generating code, I quickly go to the appropriate branch, generate and test the code, do multiple commits then merge it to the master branch when I'm sure things are working as they should. If not, it's easy to revert or even delete the entire branch and start over.

Database Configuration: Out of the box grails configures the application for hypersonic. To configure MySQL requires three steps: 1) create the development, test, and production databases and login permissions, 2) modify the conf/DataSource.groovy configuration, and 3) drop the MySQL connector/j jar into the project's lib folder. When this is complete simply do a 'grails run-app' and you will see the new tables appear in the database.

The project at this point has only a single domain class, User so a single table is created called 'user'. All columns were created including an 'id' and 'version' column. Without any constraints, all columns default to 'not null'.

I prefer the rails convention of plurals and lock_version, but I'll leave that for now.

Constraints: Adding constraints was very straight forward. I watched the server restart and the database re-build with each change. Lots of magic going on in the background. I tested the constraints through the UI. The next step is to create unit and integration tests. More on that in a later post...

Upgrading to Groovy 1.5

2007-12-11T11:40:00.000-08:00

I upgraded to the latest 1.5.0 version of groovy today. I was a bit of a pain, but after writing an install script (sandbox/groovy/install-groovy.sh) I was able to update all of my servers. Some of the new features include:

enums
annotations
generics
groovy interactive shell
updates to the groovy console

You can read about additional upgrade specifics and view code snippet examples here. Quoting from the article...

Groovy is Java, and Groovy makes Java groovier. Compared with other languages, Groovy is certainly the language that provides the flattest learning curve to Java developers, thanks to a very similar syntax.

I would add that ex-java programmers that have moved to ruby will love groovy. Closures, interactive shell, access to all your current java libraries, built in ORM, and lots of other goodies.

Here is a sample of how I use groovy to read job logs. Jobs run nightly on Sun production servers...

import groovy.sql.Sql
import database.*

oracle = new Oracle()
today = oracle.timestamp(new Date() - 1)
db = oracle.getStarsProd()

sql = "select * from job_log where start_time > ? order by job_id" db.eachRow(sql, [ today ]) {job ->
print "$job.job_id $job.start_time $job.end_time"
println "\t$job.row_count\t$job.error_count\t$job.job_name"
}

This simple script dumps jobs run in the current day. The database package is a collection of groovy scripts that contain oracle specific connections and utilities. More on the specifics of what's in that package in another post...

Git for Flex Projects

2007-12-10T11:23:00.000-08:00

I have been using subversion for Flex2 projects since it was released in 2006. I use the full version because I had problems getting the eclipse plugin to work. But I had other problems with the full Flex2 IDE as well. One problem was there is no facility to exclude files from the build path. So, all of my .svn files get copied from the src folder to bin. Permission problems also don't allow me to do a "clean" through the Flex2 IDE. Bad. Very bad.

So, I have converted all my Flex2 and action script 3 projects to use git rather than subversion. Simple to do, I just exported my adobe subversion branch and ran git init. I added a single .gitignore at the project root to ignore files in all the bin folders. Then I used Flex2 to recompile all the projects and ran git status to verify the ignores. It's all good...

Note: this would have been a bit easier to just use git-svn clone but my mac is missing the perl bindings required for git-svn.

The Git Server: Next was to create a --bare copy and transfer it to my git server at slicehost. First the clone.

git clone --bare ~/work/adobe adobe.git
touch adobe.git/git-daemon-export-ok
scp -r adobe.git dpw@raincity.slice.com:

Then on the server, simply move this to the git directory and fix the permissions.

cd /public/git/
sudo mv ~/adobe.git .
sudo chown -R git:git adobe.git

And test on the client side with the following and the new project is loaded to the client.

git clone ssh://git@raincity.slice.com/public/git/adobe.git adobe

After getting used to git I plan to move all projects there. If I can find the perl hooks, I'll probably use git-svn at least the the older (Sun) projects.

Rails 2.0

2007-12-08T09:00:00.000-08:00

I spent some time today to convert all my on-going projects to Rails 2.0. Not a huge effort, but here are my notes.

My first problem was getting "gem install rails" to upgrade smoothly. It appears that the rails team noticed the problem and quickly put up version 2.0.1, but it also failed. A quick google search pointed me to a Polish site that had what looked like a solution. A few modifications and I was ready to go. The highlights are...

svn co http://dev.rubyonrails.org/svn/rails/tags/rel_2-0-1/ rails-2.0.1

cd rails-2.0.1

for f in *
do
 [ -d $f/pkg ] && { cd $f ; rake package ; cd .. }
done

for f in */pkg/*.gem

do
 sudo gem install $f --local
done

With the six gems now freshly compiled I put them into a tarball and moved them to a distribution point. Then I wrote a short script to untar and install locally (sandbox/rails/gem-install.sh). That made it easy to update all of my machines, including slicehost. (I did have a problem on one of my older machines. I had to modify the script to load the gems one at a time).

Next were the tests. I was able to get all projects to test without errors. I created a small demo project to make sure that was working as well.

Happy rails 2.0.1!

Replacing Subversion with Git for version control

2007-12-07T08:07:00.000-08:00

I downloaded git on one of my fedora machines (fedora 7) to get a feeling for how it works vs subversion. I used yum to get the latest version (1.5.3.3) with this command:

sudo yum install git

Before getting started with git, first thing is to establish some global configurations. The standard configs are saved in ~/.gitconfig (currently in config/common). There is also a ~/.gitk config file that should be modified for the mac (put in config/osx/sedona) to set the font to Monaco 12 rather than Helvetica 9.

Using Git: There are tutorials for using git on the main site and one for subversion users. After the installation, I used the tutorial to establish tracking to some basic accounting files, time sheets, check register, etc. It started like this.

cd accounting
git init
git add .
git commit

Now the accounting directory is under source control. Another great resource for git is from peepcode.

Remote Repository: The next step is to get a remote repository established. I played around with the git socket protocol on slice.host, but found it a bit too confusing. So after reviewing the peepcode git session, I decided to use ssh and a restricted shell for the remote repository. I followed these steps:

login to the remote machine (raincity.slice.com)
create a new 'git' user: useradd --shell /usr/bin/git-shell git
copy to the server: scp -r accounting dpw@raincity.slice.com:accounting
create a /public/git folder and move the accounting to it
set owner to git:git

So it wasn't too difficult. After running clone, I did a few push and pulls from different machines and watched the changes on the remote repository. Works good...

Sprouts for Flex Application Generation

2007-12-05T10:00:00.000-08:00

A second generation refactoring of AsProject for creating Flex projects is being created by Luke Bayes and Ali Mills at assertTrue. The project has great promise, and since it is based on ruby, gems, yaml and rake it has instant credibility with me. The project is in alpha stage, so there is lots of work to be completed, but here are some thoughts about my first experiences with sprouts...

First, I installed the gem on both OSX (10.4) and Fedora (7) with:

sudo gem install sprout

Then I created my first project on both machines.

sprout -s mxml MyProject

Since this is the first application in this project, there are a bunch of dependency files that need to be downloaded. Sprout gives a good visual indication of what it's doing, so it was not a problem. Once the basic projects were created, it was time for the initial test.

cd MyProject/project ; rake

If you don't have a mm.cfg at your home directory you'll be prompted (pestered?) to create one. This is more of a flex thing rather than sprouts, and I have lots of issues with how flex handles the debug thing, but for now, I just answered yes...

My results on linux were as expected. The project compiled and ran just fine. No so on my mac. It compiled ok, but the runner failed. I was able to access the compiled swf on my browser so I'm assuming that my mac's flex installation is messing up the runner. (After perusing the project issues I see that this is a known problem. I also looked at a post from LeakyMmembrain to help smooth out some issues (but rake still hangs)...)

So now that I have a project framework, next is to see how it works with the Flex/Eclipse IDE (flex 2 on my mac only). This wasn't as I hoped. The generated structure doesn't play nice with Flex's vision of projects, i.e., there is a single level project where src, test, assets, css, and lib reside. but, on the same level are script, rakefile.rb and bin. What flex tries to do is grab all the files and copy them to the bin folder--not good.

Sprout Wish List:

Here is a short list of things I would like to see in sprouts:

a rake plugin similar to 'lib/tasks' in rails
rake file named Rakefile (ok, that's a nit)
better Flex IDE integration, at least for mxml projects

So, I'm hoping that future versions, or possibly a custom template configuration will solve the Flex IDE problem. In any case I'm looking forward to following the progress of the sprouts.

Programming for the Haxe Compiler

2007-12-05T09:30:00.000-08:00

Haxe is a programming environment and compiler for javascript and action script like files. It will compile to flash swf files so it's great for web application development.

I downloaded and installed the linux version (fedora 7) and tried out a few of the examples. The first was a classic hello world app. Simple, fast compiler, but with Flex/Flash now open source, I don't think there is a compelling reason to use haxe.

But, I left the files in my sandbox. You never know...

Geo Coding with Google KML

2007-12-03T08:00:00.000-08:00

The google map API is easy to use either from javascript or directly through curl, or ruby. A recent post by assay depot on replacing or foregoing the heavy weight Cartographer plugin for a lighter approach has inspired me to produce a stand-alone ruby library to grab coordinates and address info using google, or any other service (geocoder.us comes to mind). So, here's my attempt.

First off is to get registered with google to get an access key. Second was to place this in a secure location so that it can be used, but not seen by others. So I created a simple google.rb script to house this and other login creds. It's stored in the home directory of what ever server I'm currently on, so easy to get to but out of view from the web space.

The next step was to create a simple value object that holds physical address information. I tried to make the object as generic as possible by cross referencing google KML (keyhole markeup language) schema and the commercial geocoder.us attributes as well as the TIGER/Line data provided by the US Census Bureau . Lots of attributes, but what I need is basic address and coordinates so the attributes are:

street (or thoroughfare)
city
state code
postal code
country
latitude
longitude
accuracy

Next was to create a query request object. The query request is used to launch queries to any geo source, parse the results and return a populated value object. A typical query would be:

include Geocode
source = GoogleMapSource.new(Google.access_key)
request = Query.new(source)      
info = request.search('1211 13th St, Boulder')

The result value object (info) returned from the search would show:

street => 1211 13th St  
city => Boulder
state => CO
postal_code => 80302
country => USA
latitude => 40.008704 longitude => -105.276221
accuracy => 8

The search results from google is in XML format, so populating the address object with the correct values requires parsing the results. For this I created a KmlParser class. The KML schema supports multiple addresses, county, altitude, and other attributes. Here is the google output response to the previous query.

The QueryObject returns the first address found, but subsequent addresses are available (query.addresses). Also the status code for the query is available along with a boolean status_ok method, so a safer call would be:

request.search('1211 13th St, Boulder')
info = request.addresses.first if request.status_ok

You can download the classes including address info, parser, google access, etc here. (Note: You will have to provide your own access code to use.)

OpenID with Flex and Rails

2007-12-01T16:24:00.000-08:00

Standard Applications...

We actively support OpenID for our open source products that require login authentication. Most applications don't require rigorous authentication, so presenting a simple dialog with only open id uri works fine.

The dialog above allows the user to access an open id application or register for a new open id account. The registration process takes the user to an open id provider, such as MyOpenID. The backend, rails in this case uses the access uri to discover the provider and confirm the authentication, i.e., log the user in.

Additional Security...

Using open id for your applications means that you don't have to store, or protect the user's password. It also lets the user access many applications with a single password. But sometimes an application, or a specific process within an application requires further authentication. That's where the authentication dialog comes in.

The dialog above presents an input and set of images. The input accepts the user's challenge code and the images are part of a group configured by the user. Only one of the images is the correct one, so if the user enters the correct challenge code and clicks on the correct image, access is granted. Look ma, no passwords!

Challenge Code...

So where does this challenge code come from? Well, during the registration process the user selects the method of delivery for the challenge code. The options are sms, email, pager, or a discrete application. The challenge code itself is a random set of characters with a minimum of ten characters (all numeric for pagers). For email a complex uuid is used so copy/paste is usually required.

You can access a non-working demo here. We hope to have a working demo soon...

Time Tracker Database Schema

2007-11-30T12:43:00.000-08:00

I'm working on a new project that was intended as a large scale demo for the flex2rails library but has grown into it's own beast. This article discusses the thinking behind the database design for time tracker.

The image to the right is the full schema for time tracker as of version 2007.11.29. I'll start with the basic business requirements to explain the entities.

Business Requirements:

register users that belong to a company
register companies (customers) that have contacts
associate mailing address, email, phone, sms with contacts
support projects that have discrete tasks
associate users with projects
support time sheet entries made by users
support ad-hoc user attributes such as skill level, job title, etc.

And here are my ground rules for creating entities in rails projects:

all tables have a discrete, autogenerated ID column as primary key
all tables include lock_version, created_at, updated_at columns (exceptions are for :through associations)
all tables include a bit_flags column that contains multiple boolean flags held in a single integer. the mappings are defined in the rails model using the has_flags plugin
table names are plural
most columns are not-null

There are two basic entity groups: user, contact and company related and project related. Lets start with the users.

Users: The users table is configured for open ID. it contains and open id URI, and email and optional name. There is also an access_key column that will be used to continue login sessions across multiple applications, similar to a session id, but more flexible. Bit flags, defined in the model, include active, admin, banned, etc. An associated table is user_attributes. This is used to store ad-hoc values such as job title, rolls, etc.

You will notice that there is no password or password_hash column. We use open id so don't require storing passwords in the user table. But, if we did store passwords, the last place to do it is right along side the access/user code, or even in the same database. So, if we ever have a need to actually store passwords, it would be in a separate database similar to a unix shadow file.

Contacts: Entries into to contacts table may or may not be associated with users, typically they are. Examples include project members, coders, etc. or customer contacts, managers, etc. But, some customers are not registered users so they are not associated with the user table. Other contacts may be virtual, such as "bill to" or "ship to".

Contact methods define ways to communicate with a contact such as phone numbers, email addresses, sms, chat handles, etc. Contacts may also be associated with zero or more mailing addresses that include physical coordinates.

Companies: Company rows may exist on their own but more typically they are associated with (and owners of) projects, have contacts, addresses, etc. Each project must be associated with a company, typically the client/customer.

Projects: Projects are associated with a companies and can have project tasks and project members. Projects can also themselves be associated with other projects as a sub-project. Project members are registered users. Projects typically have a start date.

Project Tasks: Project tasks belong to projects and users and have a single task type. Tasks have a start date and due date and optionally number of hours bid.

Time Sheets: Time sheet entries are created by users to log time spent on project tasks. Logged time includes a start time and number of hours. There is also a single activity type associated with each time sheet. Activity types are user defined and typically include design, coding, debugging, testing, etc.

So this is a quick tour of the Time Tracker database schema. The complete migration scripts are contained in the open source project and will be made available for download when the project goes to beta (projected for Q1-2008).

Non Intrusive Flex/Rails Remoting

2007-11-29T14:44:00.000-08:00

Objective: create an open source remoting framework to communicate between Flex UI and Rails server without modifying any rails code.

Solution: Rails restful scaffold resource generation.

Flex has some great xml processing tools. Rails can easily generate xml out of the box. So, match up the 7 rails restful methods with a Flex/AS3 framework and life is easy...

This library includes connection configuration, index, show, create, update, destroy classes that operate against a remote Rails site. The rails code can be generated using resource scaffolding and remain unmodified.

Here is a live demo. It's a simple list/detail application that uses the remoting framework.

Download the swc here or the source code here. Or, you can check out the the latest by running:

svn co http://svn.raincitysoftware.com/adobe/org-raincity-rails/src/

Source code for the demo application is also available here.

Using the framework is quite easy. Simply create a controller and register the rails host and services in your main application. Then, create handlers for the discrete service methods including query (list), fetch (show), create, update and delete. Custom methods can be created by extending RailsService and implementing the send() method.

I'm currently updating the documentation with a step by step tutorial.

PureMVC - an Application Framework

2007-11-29T11:33:00.000-08:00

I was introduced to PureMVC a few weeks ago through assertTrue's first rate video cast covering various Flex application frameworks including Cairngorm, MVCS, and others including my favorite, none of the above. But, PureMVC has many intriguing features, including...

great documentation--the best I've seen short of having to buy a bulky book from amazon
great use of standard patterns
good use of publish/subscribe event model for loose coupling
promoted as lightweight, but feels like a java framework

So as a java engineer it does feel lightweight, especially after working with spring, hibernate and j2EE--but after working with rails for the past two years, I still feel like it's a bit heavy. But for someone looking for the best framework for Flex applications, PureMVC gets my vote.

So why do I prefer none of the above? Well I'm not completely convinced, but I have a feeling that when a project gets to a usable size, the amount of registration for each command and value object would be overwelming. I also am more familiar with logic inside the controller as opposed to inside discrete mediators.