Grails 1.3.4, Groovy 7 and NetBeans 6.8

After about a year of doing nothing but Adobe Action Script projects, I'm finally back to a java/groovy/grails project. So, I downloaded the latest grails and fired up my version of NetBeans (6.8) and generated some code. Here's what I discovered...

First, NetBeans is very groovy/grails friendly. I was able to generate the project inside the IDE. I then dropped to the command line to generate domain classes and tests. I was able to run the full test suite inside the IDE, but I still prefer the command line.

I noticed that the generated code doesn't include an ant script, but just as well. I'm so sick of the XML crap, and the grails command line does everything you need, so...

Next step is to generate controllers, and maybe a few views, but I usually use grails as a JSON or XML server with a Flex/Action Script UI. I'm also creating all the unit tests as I go and running them from inside the IDE and from the command line.

Next post will discuss the controllers and any problems related to getting the application into production. So far everything works as advertised--better than previous versions, and very productive. My new best friend...

A Groovy Cryptology Library

Rain City's open-source groovy library has recently been enhanced to include a crypto package. Implementation includes symmetric and asymmetric algorithms, hmac, digest, guid, key pool and key store utilities. This post will explain the main classes and show a few examples.

Guid: The Guid class is a simple wrapper around java's UUID class. It makes it easy to create a new guid or read an existing guid string to create a guid object.

println new Guid().toString() -> e79a8462-32ad-48db-9905-3d53e1471de3

SaltedSecureRandom: SaltedSecureRandom provides a salt-seeded SecureRandom object using hardware dependent /dev/urandom. This works for all non-Microsoft-windows platforms. On windows, a simple secure random object is returned. (If you are serious about security, don't use windows).

Digest: This class is a thin wrapper around the JCE digest algorithms, including SHA-512. The class defaults to SHA-256 to maintain compatibility with other languages and platforms. MD5 is also available. The default provider is Sun. The class also provides a 'main' to enable checking digests from the command line.

ByteContainer: The ByteContainer class has utilities for converting strings to by streams, base64 bytes and hex strings. Invaluable for exchanging encrypted data.

Hmac: The Hmac class supports SHA-256 and MD5 and provides methods to set text and key to compute and render keyed digests. It also includes a key generator that defaults to SHA-256 algorithm.

def bc = new ByteContainer()
def k = Hmac.generateKey()

bc.bytes = k.encoded
println bc -> 85a749fd3818fccf0dc6997b6c9f25129babbf6b98606f8b51a14e68b5551fa6

Crypto: This is the primary class to use when you need asymmetric encryption using well known algorithms from various providers. The default algorithm is AES using the Bouncy Castle provider with PKCS7 padding. Other supported algorithms are Blowfish, DES, Tripple DES, and all the others supported by the Sun provider.

def crypto = new Crypto()  // requires bouncy castle

assert Security.getProvider("BC")

def pw = "my password".bytes
def key = crypto.createKeySpec( pw )
assert 'AES' == key.algorithm

def text = 'this is a test of plain text'

crypto.generateIV()
println crypto.ivSpec.getIV()

def enc = crypto.encrypt( text, pw )
def s64 = enc.encodeBase64().toString()
println s64 -> q7teBCCBK+n3qTSr518QIj04qa/cS/P2bhsUw/eg8a8=

def dec = crypto.decrypt( enc, pw )
def s = new String( dec )
println s -> this is a test of plain text

crypto.clearBytes pw
println "${pw}" -> [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

RsaKeyPool/RsaKeyStore: 2048 bit public/private keys use very large random prime numbers. So generating secure key pairs takes lots of CPU cycles and a fair amount of time. To alleviate any performance bottlenecks, the best approach is to create a key pool to serve up single use key pairs. The pool actually gets keys from a key store that can be built off-line to any desired size. This approach means that a server won't be bogged down initializing the pool when it first comes up (or is bounced).

The actual generation of the key pairs is through a JCE KeyPairGenerator using the Bouncy Castle provider. It's initialized to the key size (defaulting to 2048 bits) and uses the salted secure random class mentioned earlier.

Rsa: The Rsa class provides encryption and decryption of data using public/private keys. For the uninitiated, the main thing to remember is that the public key is used to encrypt and the private key to decrypt. The Rsa class makes this a bit easier by providing encrypt() and decrypt() methods. A simple use case example goes like this:

• Alice wants to send an encrypted document to Bob. She begins the process by requesting a public key from Bob. This is what will be used to encrypt the document.
• Bob creates a key pair (or gets a pair from the pool) and sends the public key back to Alice.
• Alice creates a Rsa object and sets the public key to Bob's key. The document is encrypted then sent to Bob.
• Bob receives the encrypted document and decrypts with Rsa using his private key.

Now lets take this simple use case and implement some of the parts. For our purposes, we will use two classes Alice, the sender and Bob the receiver.

import org.raincity.glib.crypto.*

class Alice {
def encrypted
def send = { bob, text ->
def key = bob.getKey()
def rsa = Rsa.instance
rsa.pubKey = key
encrypted = rsa.encrypt( text.bytes )
bob.receive( encrypted )
}
}

class Bob {
def text
def keyPair

def getKey() {
keyPair = RsaKeyPool.instance.keyPair
return keyPair.publicKey
}

def receive = { encrypted ->
def rsa = Rsa.instance
rsa.privKey = keyPair.privateKey
text = new String( rsa.decrypt( encrypted ) )
}
}

alice = new Alice()
bob = new Bob()
alice.send( bob, "This is a super secret message" )
println "encrypted=${new BigInteger(alice.encrypted).toString(16)}"
println bob.text

And when you run the script you see...

encrypted=496bc13c9ba194a2becc525839bab8d9c40a7c126a1ac32f0aa21...
decrypted=This is a super secret message

Future posts will discuss how these classes are used to support Adobe Flex to Grails encrypted exchange sessions. If you are interested you can download the library and examples in a single jar file here.

Groovy Excel Report Library

I just finished a series of reports for one of my customers using grails as the backend and Flex for the user interface. As with many of my reports for this customer, they requested the output be delivered as CSV files. Easy, and very boring.

So, with the help of JExcel and the jxl APIs from Andy Khan, I created an alternate output of an excel spreadsheet rather than CSV flat file. This has the additional benefit of sending multi-dimensional results, e.g., summary and details on two separate sheets, output statistics and charting on a third. This is much more useful for the end-user.

To make life easy, I created a wrapper around the jxl libraries to make it possible to use an alternate excel API such as Apache POI. At the same time, I created a set of conventions that use a template spreadsheet to easily create all the output without having to program any of the "view" components (i.e., classic MVC). The controller class is Workbook and the Model data is configured in Worksheet objects. All the formatting for dates, numbers, layout, etc. is in configured in templates (xsl files) easily configured and modified by non-programmers.

Since the target for this is inside war-deployed web applications I had to configure an external templates folder to enable report format changes without having to re-war/deploy or bounce the server. The templates folder is part of a more extensive Exchange class that coordinates remote data exchanges to a known area outside the web-space.

Conventions: Below is a list of convetions used to build reports...

• xls template contains all formatting and sheets for the report
• key words embedded into the template provides cell locations where data is to be written. values stored in the Worksheet object are used by the Workbook controller to update the spreadsheet output.
  
• a special type of keyword called "ReportData" is used to insert multilple rows into the spreadsheet, based on query result sets (or any list of data).

The Worksheet models also have a place holder for "options" that change/enhance the controller's behavior as it builds output. Examples include "greenbar" to create green-bar reports, "showItemNumbers" to write out the actual row number to the left of the ReportData.

Working Example: To demonstrate the report library I created a "Customer Invoice" report. The template has three sheets, includes graphic images, and shows two of the three "ReportData" sets in green bar (and blue bar) with automatic item numbering.

The Code: The full report can be built and run with just a few lines of code. Here is a typical example that prints the invoice, details, and expenses for a single period.























The three sheets are defined as details, expenses and invoice. Invoice details and expense rows are fetched for a specific period with GORM in the typical way. Report fields are extracted to match the template report data layout. The main invoice sheet has a few extra named values (line 28-31) and constructs two lines of data (lines 33 & 34).

Report Output: Here are the three sheets created by the code above.

Flex on Grails

Borrowing from Flex/Rails libraries created about a year ago, I have created a Flex/Grails application using similar HTTP/XML protocols. The application requests report data via HTTP, the server creates the report, persisting it to a file, then the client pulls the data back with a file request. This was the only way I could get Flex/Flash to open a "file-save" dialog to let the user write to their local hard-drive.

I used Flex's HTTPService object to do the initial report request, then FileReference with additional URLVariables parameters to retrieve the file. Grails/GORM made the data retrieval easy and the controller/service packages were simple implementations. This was comparable to Flex/Rails implementation that I worked on a year ago.

The server side runs in a war distribution so the output file had to exist outside the webspace. The Flex application requests the file with an HTTP post using URLRequest with specific parameters (URLVariables) to retrieve the file. Because the server is deployed with only war files, the request is made to a specific controller with a "getfile" action that retrieves the file from someplace on the server's drive (obviously not in the war file) and is consistent with where the file was originally written. Once the file is returned to the client it is removed, so it can only be read back a single time.

This first iteration returns a CSV file, but creating a full xls spreadsheet or a PDF document would be easy to implement. There is also an xml document that can be returned to enable showing the report in a local viewer, e.g., a data grid.

Replacing GORM with Groovy Sql for Lightwieght Jobs

GORM, the Grails Object-Relational Mapping persistence framework provides a rational way to use Hibernate based on pojo (or groovy pogo) domain classes and basic conventions. Create a domain class, define some constraints, start up grails and viola! tables, columns, indexes are created without coding a line of SQL/DDL or creating annoying XML mappings.

But, what if you have a simple standalone process that requires single or multiple database access? Currently GORM has trouble working outside of grails, and creating a new grails project just to support a simple, no-client script is overkill. For that, I use groovy's database access classes in groovy.sql and a small custom library to easily access multiple data sources and provide simple query tools optimized for batch operations.

Here's an example: Tracking multiple UPS shipments. The application must access existing shipment requests from a local database, then query the UPS tracking remote service to discover shipment status. When a shipment is picked up or delivered, another local database is updated and a third system is updated via XML feed.

To summarize, the requirements are:

• provide simultaneous access to multiple databases
• query UPS tracking via HTTP requests
• create XML responses and write to a messaging system
  

Cron triggers this process multiple times during the working day. It runs on the production machine (1MB slice at slicehost), so should be kind to existing web and database applications serving the user base.

Design Implementation:
As always, I start with the tests. Using groovy test case, the database queries, inserts, and updates tests were created, then the target classes and methods. Queries are always the easiest, and updates are straight forward as well. Inserts with auto generated IDs are a bit tricker, but here is where groovy sql comes to the rescue.

Creating new tables presented a new problem: should I create SQL/DDL scripts? No! For this, it was easy enough to access the current grails application and create the new domain classes, run grails to have Hibernate create the new tables.

The groovy Sql class provides a method called executeInsert(). The magic in this method is that it returns an array of all auto-generated keys. So, if I'm working in MySql or Oracle or whatever, inserts all look the same--no database specific code required.

Database access defaults to the production environment but database tests must operate on the test database (similar to grails). This is easy to accomplish by simply overriding the default behavior in the test scripts.

Conclusions:

• Use groovy Sql for small, non-web applications that require single or multiple database access.
• Use Grails/GORM for all database/table creation and maintenance (no DDL scripts). This also has the additional benefit of enabling access to new tables through grails if required.
  
• Use executeInsert() to return auto-generated keys when inserting new rows.
  

Grails Application Deployed to Slicehost

We deployed our beta application on slicehost virtual server. The slice is 1GB using fedora 7, mysql 5.1, jetty 6.1.6 and resin 3.2. Groovy version 1.5.4 is installed on the slice and we used Grails 1.0.1 for development, but it's not installed on the slice.

The application at this point is minimal, but is scheduled to go live later this month. I can't go into to much detail, but application provides a data service between United Parsel Service and a major customer to schedule shipments of large and small freight.

Grails Dataset Loaders, Part 2

About a month ago I posted a few thoughts on Dataset Loaders for Grails. Now that one of my Grails projects is going into beta, I thought it would be a good time to revisit the issue.

Rails History: With rails I used fixtures for test data and had to hand code data loaders for production. I put the loaders in lib/tasks. They were rake tasks that did the initial loading and/or refreshing of data for a specified environment. So data loading for a production or development was a manual task. Not a huge burden, but just one more thing to remember.

Now that I'm in the Grails realm I have created automatic loaders that are environment aware. A DatasetBootStrap class loads data for development, test, and production whenever the server starts. This is in line with what hibernate does when it starts up--verify that the domain models match the current data source, and make changes where necessary. In rails this is done with migrations--a good idea, but yet another manual step.

DatasetBootStrap: The DatasetBootStrap class first determines the current environment, then loads the appropriate dataset classes and invokes the "load" method (or closure if you will). A list of datasets is configured manually for each environment. This approach makes it possible to share loaders for all three environments (or four if you use staging). Some data is just for test. Other data, like a list of US States, is loaded for all three environments. The main point is nothing is repeated, keeping my code as DRY as possible.

Datasets: I currently keep my datasets in src/groovy/datasets, but I think that it's time to move this folder into grails-app/conf/datasets. I also use a package declaration (package datasets) to insure that dataset classes don't conflict with other classes (I've had problems with this in the test structure when trying to create unit tests that have a name conflict with integration tests).

Simulating Rails-Like Test Fixtures: This was the easy part. In each test that requires a specific dataset, the setUp() method is used to load as many datasets as necessary to create a solid test environment. Loading all the data is as easy as calling the DatasetBootStrap's init() method.

Grails Integration: My goal for Datasets is to propose it's use for all grails projects by integrating it into the grails core. I have a bit of refactoring to do before this is practical, but would welcome any comments or suggestions from other grails (or rails) users as to what features a Dataset loader module should include.

UPS Web Services Interface

One of my customers has a requirement to use United Parcel Service to ship large and small packages. UPS has a great set of documentation, but they handle large and small packages with different technologies. Small packages are simple XML request/response transactions that provide methods for requesting a shipment, accepting shipments, and voiding shipments. Large packages allow you to request a shipment, but the request/response is all in SOAP, something of an overkill in my opinion.

UPS requires lots of data concerning who, what and where to establish their pricing. Data is sent to a URL/end point and a response is returned. The response includes pricing and tracking numbers, as well as image data for printing barcoded shipping labels. So, it is mainly a data transfer operation-- a single service.

SOAP is great if you have lots of defined services, like weather in Seattle or Paris or zip code 94705. Or conversion of units from/to. But for simple data transfer, REST, or just plain old XML request/response is much more efficient.

Here is a diagram of the complete data transfer...


...

As you can see, there are lots of communications. The first question is, why is there a server in the middle? Well, here's the deal--the SunReturns client is just that, a client. The data really resides at the DataServer--the client simply persists it's own private session. The UPS server is at their site. All communications are via SOA/services--some simple request/response (ala REST) and others use SOAP.

The technology used in the middle is grails/groovy. The SunReturns server is currently a mixture of JSP (legacy) and groovy/GSP. The groovy components are XML only and include controllers, services, domain objects, etc--similar to grails but without the Hibernate layer. It talks to legacy databases, both Oracle and MySQL--not impossible with grails, but definitely outside the conventions.

Conclusions: The natural loose coupling of SOA provides a new vehicle to solve business problems. At the same time, distributing business processes across diverse servers provides a workable migration path to newer technologies. Groovy, grails, and simple XML transport provide a way to implement SOA with a minimum of effort.

Survey Project--A Grails Implementation

I started small grails application today to enable creating surveys and questions and tracking questions and results. The domain classes are:

• Survey: the primary table that defines a survey project
• SurveyQuestion: object to hold a specific question mapped to a survey
• SurveyResponder: the person that is answering the survey questions
• SurveyResponse: the recorded answers that the reponder enters

And here is the proposed ER diagram, created using dbDesigner.


As the diagram shows, surveys have many questions, questions have many responses. Surveys also have many responders and responders have many responses. You can also see that this is a very simple survey with no branches.

Grails/GORM Implementation:
I began by generating the Survey and Survey question domain classes and basic validation tests. Next was to create the SurveyReponder and SurveyResponse classes. The implementation was straight forward, but I had to fiddle with the domain class definitions to get hibernate to understand my intent. Just a matter of specifically defining class references in dependent models rather than depending on the 'belongs_to' and 'has_many' declarations.

Datasets and Loaders (Fixtures):
Unlike Rails, Grails doesn't use fixtures for test data. Lucky for me I have a Grails enhancement the solves this based on a sandbox proposal from the Grails guys. So I generated the dataset loader classes using: grails create-dataset [domain]. This creates the following:

• a DataLoaderBootStrap class in grails-app/conf/ (unless it exists)
• a DomainDataset class in src/groovy/datasets
• a set of random data to be loaded

An example of what gets created is here:

> grails create-dataset survey
> ...
> file grails-app/conf/DataLoaderBootStrap.groovy exists...
> created file src/groovy/datasets/SurveyDataset.groovy
>

The SurveyDataset class looks like this:

class SurveyDataset {
dataset = {
def set = [
[ name:'879',description:'510',email:'680' ],
]

return set
}

load() {
dataset().each { data ->
obj = new Survey( data )
obj.save()
}
}
}

As you can see, the data is contrived but all of the fields are present and other than the actual data, the script is good to go. I'll show how this data can be loaded from the console, command line and within test scripts in a later post.

10 Reasons to Switch from Rails to Grails

After spending a few years really enjoying Rails it was difficult to bring myself to even try groovy and grails. But my latest contract forced me to look for alternatives, and I'm glad I did. Here are some reasons that you may want to switch...

1. GORM with hibernate/spring and jpa is much better than ActiveRecord
2. No distribution problems; runs in many production ready containers
3. Internationalization out of the box (not specifically ignored as DHH does)
4. Transactions actually work, and they include save-points.
   
5. Not only dynamic finders and counters, but dynamic listOrderBy
   
6. No green threads (although this may be fixed in Ruby 2.0, around 2010?)
   
7. Ability to use pessimistic locking out of the box
   
8. Real-live prepared statements and .withCriteria method
9. Production level test reporting with built in Mocking and Stubbing
10. Search operations are based on Lucene (with a plugin)

All of these don't make sense for a non-java coder. And my startup time for grails would have be much longer without my prior experience with Rails and Ruby.

One thing I thought I might be giving up was the endless list of valuable gems that make developing in Rails a real pleasure. But after re-reviewing the list of open source products from apache, java source, spring, hibernate, and even Sun (glassfish), I can't think of any gems that I will miss.