Grails Dataset Loaders, Part 2

About a month ago I posted a few thoughts on Dataset Loaders for Grails. Now that one of my Grails projects is going into beta, I thought it would be a good time to revisit the issue.

Rails History: With rails I used fixtures for test data and had to hand code data loaders for production. I put the loaders in lib/tasks. They were rake tasks that did the initial loading and/or refreshing of data for a specified environment. So data loading for a production or development was a manual task. Not a huge burden, but just one more thing to remember.

Now that I'm in the Grails realm I have created automatic loaders that are environment aware. A DatasetBootStrap class loads data for development, test, and production whenever the server starts. This is in line with what hibernate does when it starts up--verify that the domain models match the current data source, and make changes where necessary. In rails this is done with migrations--a good idea, but yet another manual step.

DatasetBootStrap: The DatasetBootStrap class first determines the current environment, then loads the appropriate dataset classes and invokes the "load" method (or closure if you will). A list of datasets is configured manually for each environment. This approach makes it possible to share loaders for all three environments (or four if you use staging). Some data is just for test. Other data, like a list of US States, is loaded for all three environments. The main point is nothing is repeated, keeping my code as DRY as possible.

Datasets: I currently keep my datasets in src/groovy/datasets, but I think that it's time to move this folder into grails-app/conf/datasets. I also use a package declaration (package datasets) to insure that dataset classes don't conflict with other classes (I've had problems with this in the test structure when trying to create unit tests that have a name conflict with integration tests).

Simulating Rails-Like Test Fixtures: This was the easy part. In each test that requires a specific dataset, the setUp() method is used to load as many datasets as necessary to create a solid test environment. Loading all the data is as easy as calling the DatasetBootStrap's init() method.

Grails Integration: My goal for Datasets is to propose it's use for all grails projects by integrating it into the grails core. I have a bit of refactoring to do before this is practical, but would welcome any comments or suggestions from other grails (or rails) users as to what features a Dataset loader module should include.

UPS Web Services Interface

One of my customers has a requirement to use United Parcel Service to ship large and small packages. UPS has a great set of documentation, but they handle large and small packages with different technologies. Small packages are simple XML request/response transactions that provide methods for requesting a shipment, accepting shipments, and voiding shipments. Large packages allow you to request a shipment, but the request/response is all in SOAP, something of an overkill in my opinion.

UPS requires lots of data concerning who, what and where to establish their pricing. Data is sent to a URL/end point and a response is returned. The response includes pricing and tracking numbers, as well as image data for printing barcoded shipping labels. So, it is mainly a data transfer operation-- a single service.

SOAP is great if you have lots of defined services, like weather in Seattle or Paris or zip code 94705. Or conversion of units from/to. But for simple data transfer, REST, or just plain old XML request/response is much more efficient.

Here is a diagram of the complete data transfer...


...

As you can see, there are lots of communications. The first question is, why is there a server in the middle? Well, here's the deal--the SunReturns client is just that, a client. The data really resides at the DataServer--the client simply persists it's own private session. The UPS server is at their site. All communications are via SOA/services--some simple request/response (ala REST) and others use SOAP.

The technology used in the middle is grails/groovy. The SunReturns server is currently a mixture of JSP (legacy) and groovy/GSP. The groovy components are XML only and include controllers, services, domain objects, etc--similar to grails but without the Hibernate layer. It talks to legacy databases, both Oracle and MySQL--not impossible with grails, but definitely outside the conventions.

Conclusions: The natural loose coupling of SOA provides a new vehicle to solve business problems. At the same time, distributing business processes across diverse servers provides a workable migration path to newer technologies. Groovy, grails, and simple XML transport provide a way to implement SOA with a minimum of effort.

Business Service Components

Every business application has a set of commonalities that lend themselves to standardization. In the rails, and to some extent grails world, this translates into plugins. The restful authorization plugin being a good example.

But, what if rather than a plugin, you had access to a software service that provide the same functionality? Here is a comparison:

Using a business related plugin:

1. download and install the plugin into your project
2. build out all the required tables
3. hookup your internal logic to insure that the plugin loads correctly
4. use the plugin
   

Using a Software Business Service:

1. use the service

The choice should be obvious. Rather than dealing with database migrations, version control, broken or non-existent tests, integration into your pristine project, you simply use the service. A good example is the captcha service called Captchator from Andreas Schwartz. This service replaces logic that traditionally comes in the form of a plugin with a callable service that does what a captcha component needs to do. No downloads required. No migration of new tables. Just use the service.

Now imagine a full suite of services for business processes. Authentication, company and contact information, time tracking, invoicing, fulfillment--all discrete services that a VAR or software developer can simple use as components in a vertical solution. Wouldn't that be cleaner than installing plugins?

Git for distributed backups

I usually use svn for code and document backups but today I decided to use git to automate my database backups. Git seems to run much faster and offers a bit more security. It also functions as a good transport for multiple distribution to remote sites.

The backups are in ~/backups so I created a git repository and moved it to my remote slicehost site. Here are the steps:

• cd backups
• git init
• git commit -a -m 'initial repo'
• cd ~
• git clone --bare ~/backups backups.git
  
• scp -r backups.git dpw@raincity.slice.com:

The next step was to move the backups.git folder to /public/git and chown -R git:git. Then to read the repo back:

• mv backups /tmp
• git clone ssh://git@raincity.slice.com/public/git/backups.git

At this point I have local and remote repos. I went to other machines and cloned the repo to insure that I have multiple locations on and off site. The next step was to hook git up to my standard backup from the master database. At the tail end of the SQL backup script I do this:

• git add .
• git commit -m "automated backup on `date`..."
• git push

And there you go. This provides a local and remote backup. Configuring cron on other machines is as simple as doing a 'cd ~/backups ; git pull'. Life is easy...

Survey Project--A Grails Implementation

I started small grails application today to enable creating surveys and questions and tracking questions and results. The domain classes are:

• Survey: the primary table that defines a survey project
• SurveyQuestion: object to hold a specific question mapped to a survey
• SurveyResponder: the person that is answering the survey questions
• SurveyResponse: the recorded answers that the reponder enters

And here is the proposed ER diagram, created using dbDesigner.


As the diagram shows, surveys have many questions, questions have many responses. Surveys also have many responders and responders have many responses. You can also see that this is a very simple survey with no branches.

Grails/GORM Implementation:
I began by generating the Survey and Survey question domain classes and basic validation tests. Next was to create the SurveyReponder and SurveyResponse classes. The implementation was straight forward, but I had to fiddle with the domain class definitions to get hibernate to understand my intent. Just a matter of specifically defining class references in dependent models rather than depending on the 'belongs_to' and 'has_many' declarations.

Datasets and Loaders (Fixtures):
Unlike Rails, Grails doesn't use fixtures for test data. Lucky for me I have a Grails enhancement the solves this based on a sandbox proposal from the Grails guys. So I generated the dataset loader classes using: grails create-dataset [domain]. This creates the following:

• a DataLoaderBootStrap class in grails-app/conf/ (unless it exists)
• a DomainDataset class in src/groovy/datasets
• a set of random data to be loaded

An example of what gets created is here:

> grails create-dataset survey
> ...
> file grails-app/conf/DataLoaderBootStrap.groovy exists...
> created file src/groovy/datasets/SurveyDataset.groovy
>

The SurveyDataset class looks like this:

class SurveyDataset {
dataset = {
def set = [
[ name:'879',description:'510',email:'680' ],
]

return set
}

load() {
dataset().each { data ->
obj = new Survey( data )
obj.save()
}
}
}

As you can see, the data is contrived but all of the fields are present and other than the actual data, the script is good to go. I'll show how this data can be loaded from the console, command line and within test scripts in a later post.

Grails Dataset Loaders, aka Test Fixtures

One thing I've been missing since working in Grails is test fixtures ala Rails. The first steps I follow when developing data models is to create test data, which is fairly easy in Rails. For Rails I had to do some extra work to get this data to load into the development db and I also had to augment the yaml files with ERB to generate random data.

The Grails sandbox area has a thread that proposes creating test fixtures that begin in late 2006. As the conversation unfolded, there was a decision to use the term Dataset rather then Fixture--a welcomed change. But the thread kind of died as of October of 2006, so I guess I need to implement this on my own. Here's my attempt...

Datasets and Loaders (Fixtures):
I followed the sandbox proposal and created Dataset classes rather than yaml files. Closures were an obvious choice to create datasets, but I also had some other requirements. My objectives were:

• datasets are usually a single set, but may be separated by dev, test, and production
• datasets are implemented as closures to enable generating random data
  
• dataset loaders may be separated into development, test, and production
• they can be loaded from within the application on start up
• they can be loaded on demand from test classes (similar to calling out fixtures)
• they can be loaded through scripts (ant or grails)
• they can be loaded from grails shell and console
  

For all environments a DataLoaderBootStrap class controls loading when the application starts (grails run-app). The bootstrap is environment aware, and loads specified datasets from individual Dataset classes, e.g. UserDataset, CountryDataset, etc. based on the current environment.

The datasets are classes that have a dataset closure that defines hash maps of data (not model classes) and a load() method to do the actual database inserts (or updates). The classes are placed in the src/groovy/datasets folder. Their is a load() method that creates objects from the hash set then invokes "save" to either insert or update depending on the state of the database. This has the benefit of creating predictable data without having to continually drop or trucate tables (ala rails).

When testing in the console, the loaders can be run manually. Test scripts can invoke loaders at any time to insure that there is predictable data. A command line script can be invoked to load all data for the given environment using "grails [env] load-data".

I thought of creating a plugin for this but it doesn't really fit the plugin framework. So, for now I plan to simply keep it as a platform enhancement.

Web Containers on SliceHost

I have a small virtual server at slice host (256M). I originally thought it would be appropriate for rails/mongrel and apache, but probably too small for a java web container. But, after installing resin and running it through some basic tests, it looks like even the small slice is fine for testing and semi-live (e.g. demo) applications.

My next step is to test jetty and glassfish. I don't think there will be problems with jetty, but glassfish may be too much of a memory hog to work correctly.

Running 'top' shows that resin actually uses about 60M when it starts up and goes up to 90M under basic use. This will undoubtedly change when spring/hibernate and real database access kicks in, but it looks good so far. My goal is to install a grails application to see if slice host can be used for basic demo purposes.