Grails Dataset Loaders, Part 2

About a month ago I posted a few thoughts on Dataset Loaders for Grails. Now that one of my Grails projects is going into beta, I thought it would be a good time to revisit the issue.

Rails History: With rails I used fixtures for test data and had to hand code data loaders for production. I put the loaders in lib/tasks. They were rake tasks that did the initial loading and/or refreshing of data for a specified environment. So data loading for a production or development was a manual task. Not a huge burden, but just one more thing to remember.

Now that I'm in the Grails realm I have created automatic loaders that are environment aware. A DatasetBootStrap class loads data for development, test, and production whenever the server starts. This is in line with what hibernate does when it starts up--verify that the domain models match the current data source, and make changes where necessary. In rails this is done with migrations--a good idea, but yet another manual step.

DatasetBootStrap: The DatasetBootStrap class first determines the current environment, then loads the appropriate dataset classes and invokes the "load" method (or closure if you will). A list of datasets is configured manually for each environment. This approach makes it possible to share loaders for all three environments (or four if you use staging). Some data is just for test. Other data, like a list of US States, is loaded for all three environments. The main point is nothing is repeated, keeping my code as DRY as possible.

Datasets: I currently keep my datasets in src/groovy/datasets, but I think that it's time to move this folder into grails-app/conf/datasets. I also use a package declaration (package datasets) to insure that dataset classes don't conflict with other classes (I've had problems with this in the test structure when trying to create unit tests that have a name conflict with integration tests).

Simulating Rails-Like Test Fixtures: This was the easy part. In each test that requires a specific dataset, the setUp() method is used to load as many datasets as necessary to create a solid test environment. Loading all the data is as easy as calling the DatasetBootStrap's init() method.

Grails Integration: My goal for Datasets is to propose it's use for all grails projects by integrating it into the grails core. I have a bit of refactoring to do before this is practical, but would welcome any comments or suggestions from other grails (or rails) users as to what features a Dataset loader module should include.

UPS Web Services Interface

One of my customers has a requirement to use United Parcel Service to ship large and small packages. UPS has a great set of documentation, but they handle large and small packages with different technologies. Small packages are simple XML request/response transactions that provide methods for requesting a shipment, accepting shipments, and voiding shipments. Large packages allow you to request a shipment, but the request/response is all in SOAP, something of an overkill in my opinion.

UPS requires lots of data concerning who, what and where to establish their pricing. Data is sent to a URL/end point and a response is returned. The response includes pricing and tracking numbers, as well as image data for printing barcoded shipping labels. So, it is mainly a data transfer operation-- a single service.

SOAP is great if you have lots of defined services, like weather in Seattle or Paris or zip code 94705. Or conversion of units from/to. But for simple data transfer, REST, or just plain old XML request/response is much more efficient.

Here is a diagram of the complete data transfer...


...

As you can see, there are lots of communications. The first question is, why is there a server in the middle? Well, here's the deal--the SunReturns client is just that, a client. The data really resides at the DataServer--the client simply persists it's own private session. The UPS server is at their site. All communications are via SOA/services--some simple request/response (ala REST) and others use SOAP.

The technology used in the middle is grails/groovy. The SunReturns server is currently a mixture of JSP (legacy) and groovy/GSP. The groovy components are XML only and include controllers, services, domain objects, etc--similar to grails but without the Hibernate layer. It talks to legacy databases, both Oracle and MySQL--not impossible with grails, but definitely outside the conventions.

Conclusions: The natural loose coupling of SOA provides a new vehicle to solve business problems. At the same time, distributing business processes across diverse servers provides a workable migration path to newer technologies. Groovy, grails, and simple XML transport provide a way to implement SOA with a minimum of effort.