Git for distributed backups

I usually use svn for code and document backups but today I decided to use git to automate my database backups. Git seems to run much faster and offers a bit more security. It also functions as a good transport for multiple distribution to remote sites.

The backups are in ~/backups so I created a git repository and moved it to my remote slicehost site. Here are the steps:

• cd backups
• git init
• git commit -a -m 'initial repo'
• cd ~
• git clone --bare ~/backups backups.git
  
• scp -r backups.git dpw@raincity.slice.com:

The next step was to move the backups.git folder to /public/git and chown -R git:git. Then to read the repo back:

• mv backups /tmp
• git clone ssh://git@raincity.slice.com/public/git/backups.git

At this point I have local and remote repos. I went to other machines and cloned the repo to insure that I have multiple locations on and off site. The next step was to hook git up to my standard backup from the master database. At the tail end of the SQL backup script I do this:

• git add .
• git commit -m "automated backup on `date`..."
• git push

And there you go. This provides a local and remote backup. Configuring cron on other machines is as simple as doing a 'cd ~/backups ; git pull'. Life is easy...

Survey Project--A Grails Implementation

I started small grails application today to enable creating surveys and questions and tracking questions and results. The domain classes are:

• Survey: the primary table that defines a survey project
• SurveyQuestion: object to hold a specific question mapped to a survey
• SurveyResponder: the person that is answering the survey questions
• SurveyResponse: the recorded answers that the reponder enters

And here is the proposed ER diagram, created using dbDesigner.


As the diagram shows, surveys have many questions, questions have many responses. Surveys also have many responders and responders have many responses. You can also see that this is a very simple survey with no branches.

Grails/GORM Implementation:
I began by generating the Survey and Survey question domain classes and basic validation tests. Next was to create the SurveyReponder and SurveyResponse classes. The implementation was straight forward, but I had to fiddle with the domain class definitions to get hibernate to understand my intent. Just a matter of specifically defining class references in dependent models rather than depending on the 'belongs_to' and 'has_many' declarations.

Datasets and Loaders (Fixtures):
Unlike Rails, Grails doesn't use fixtures for test data. Lucky for me I have a Grails enhancement the solves this based on a sandbox proposal from the Grails guys. So I generated the dataset loader classes using: grails create-dataset [domain]. This creates the following:

• a DataLoaderBootStrap class in grails-app/conf/ (unless it exists)
• a DomainDataset class in src/groovy/datasets
• a set of random data to be loaded

An example of what gets created is here:

> grails create-dataset survey
> ...
> file grails-app/conf/DataLoaderBootStrap.groovy exists...
> created file src/groovy/datasets/SurveyDataset.groovy
>

The SurveyDataset class looks like this:

class SurveyDataset {
dataset = {
def set = [
[ name:'879',description:'510',email:'680' ],
]

return set
}

load() {
dataset().each { data ->
obj = new Survey( data )
obj.save()
}
}
}

As you can see, the data is contrived but all of the fields are present and other than the actual data, the script is good to go. I'll show how this data can be loaded from the console, command line and within test scripts in a later post.

Grails Dataset Loaders, aka Test Fixtures

One thing I've been missing since working in Grails is test fixtures ala Rails. The first steps I follow when developing data models is to create test data, which is fairly easy in Rails. For Rails I had to do some extra work to get this data to load into the development db and I also had to augment the yaml files with ERB to generate random data.

The Grails sandbox area has a thread that proposes creating test fixtures that begin in late 2006. As the conversation unfolded, there was a decision to use the term Dataset rather then Fixture--a welcomed change. But the thread kind of died as of October of 2006, so I guess I need to implement this on my own. Here's my attempt...

Datasets and Loaders (Fixtures):
I followed the sandbox proposal and created Dataset classes rather than yaml files. Closures were an obvious choice to create datasets, but I also had some other requirements. My objectives were:

• datasets are usually a single set, but may be separated by dev, test, and production
• datasets are implemented as closures to enable generating random data
  
• dataset loaders may be separated into development, test, and production
• they can be loaded from within the application on start up
• they can be loaded on demand from test classes (similar to calling out fixtures)
• they can be loaded through scripts (ant or grails)
• they can be loaded from grails shell and console
  

For all environments a DataLoaderBootStrap class controls loading when the application starts (grails run-app). The bootstrap is environment aware, and loads specified datasets from individual Dataset classes, e.g. UserDataset, CountryDataset, etc. based on the current environment.

The datasets are classes that have a dataset closure that defines hash maps of data (not model classes) and a load() method to do the actual database inserts (or updates). The classes are placed in the src/groovy/datasets folder. Their is a load() method that creates objects from the hash set then invokes "save" to either insert or update depending on the state of the database. This has the benefit of creating predictable data without having to continually drop or trucate tables (ala rails).

When testing in the console, the loaders can be run manually. Test scripts can invoke loaders at any time to insure that there is predictable data. A command line script can be invoked to load all data for the given environment using "grails [env] load-data".

I thought of creating a plugin for this but it doesn't really fit the plugin framework. So, for now I plan to simply keep it as a platform enhancement.

Web Containers on SliceHost

I have a small virtual server at slice host (256M). I originally thought it would be appropriate for rails/mongrel and apache, but probably too small for a java web container. But, after installing resin and running it through some basic tests, it looks like even the small slice is fine for testing and semi-live (e.g. demo) applications.

My next step is to test jetty and glassfish. I don't think there will be problems with jetty, but glassfish may be too much of a memory hog to work correctly.

Running 'top' shows that resin actually uses about 60M when it starts up and goes up to 90M under basic use. This will undoubtedly change when spring/hibernate and real database access kicks in, but it looks good so far. My goal is to install a grails application to see if slice host can be used for basic demo purposes.

10 Reasons to Switch from Rails to Grails

After spending a few years really enjoying Rails it was difficult to bring myself to even try groovy and grails. But my latest contract forced me to look for alternatives, and I'm glad I did. Here are some reasons that you may want to switch...

1. GORM with hibernate/spring and jpa is much better than ActiveRecord
2. No distribution problems; runs in many production ready containers
3. Internationalization out of the box (not specifically ignored as DHH does)
4. Transactions actually work, and they include save-points.
   
5. Not only dynamic finders and counters, but dynamic listOrderBy
   
6. No green threads (although this may be fixed in Ruby 2.0, around 2010?)
   
7. Ability to use pessimistic locking out of the box
   
8. Real-live prepared statements and .withCriteria method
9. Production level test reporting with built in Mocking and Stubbing
10. Search operations are based on Lucene (with a plugin)

All of these don't make sense for a non-java coder. And my startup time for grails would have be much longer without my prior experience with Rails and Ruby.

One thing I thought I might be giving up was the endless list of valuable gems that make developing in Rails a real pleasure. But after re-reviewing the list of open source products from apache, java source, spring, hibernate, and even Sun (glassfish), I can't think of any gems that I will miss.

Grails Application Development Notes

I started a grails application for Sun using the latest version of groovy (1.5) and grails (1.0-RC3). Things went fairly smooth but I was glad to be under 'git' control to rollback branches as the project developed. My first problem with with mixed case characters in domain and controllers...

I originally created an IBBUser domain model. Ok, I should probably re-name it IbbUser but old habits die hard. Anyway, the model generation went fine, but when I went to create the controller using generate-all, that's when things fell apart.

I'm not sure why (yet) but 'generate-all' created my views in an IBBUser folder. This was a problem when I ran the app, so I scrapped the project and re-generated with the domain name of 'user'. I probably could have used 'ibbuser' in lower case, but oh well...

Working with generated code in grails or rails can get out of hand quickly--so many files and folders to track. So I set up branches--one for domains, the other for controllers. Before generating code, I quickly go to the appropriate branch, generate and test the code, do multiple commits then merge it to the master branch when I'm sure things are working as they should. If not, it's easy to revert or even delete the entire branch and start over.

Database Configuration: Out of the box grails configures the application for hypersonic. To configure MySQL requires three steps: 1) create the development, test, and production databases and login permissions, 2) modify the conf/DataSource.groovy configuration, and 3) drop the MySQL connector/j jar into the project's lib folder. When this is complete simply do a 'grails run-app' and you will see the new tables appear in the database.

The project at this point has only a single domain class, User so a single table is created called 'user'. All columns were created including an 'id' and 'version' column. Without any constraints, all columns default to 'not null'.

I prefer the rails convention of plurals and lock_version, but I'll leave that for now.

Constraints: Adding constraints was very straight forward. I watched the server restart and the database re-build with each change. Lots of magic going on in the background. I tested the constraints through the UI. The next step is to create unit and integration tests. More on that in a later post...

Upgrading to Groovy 1.5

I upgraded to the latest 1.5.0 version of groovy today. I was a bit of a pain, but after writing an install script (sandbox/groovy/install-groovy.sh) I was able to update all of my servers. Some of the new features include:

• enums
• annotations
• generics
• groovy interactive shell
  
• updates to the groovy console

You can read about additional upgrade specifics and view code snippet examples here. Quoting from the article...

Groovy is Java, and Groovy makes Java groovier. Compared with other languages, Groovy is certainly the language that provides the flattest learning curve to Java developers, thanks to a very similar syntax.

I would add that ex-java programmers that have moved to ruby will love groovy. Closures, interactive shell, access to all your current java libraries, built in ORM, and lots of other goodies.

Here is a sample of how I use groovy to read job logs. Jobs run nightly on Sun production servers...

import groovy.sql.Sql
import database.*

oracle = new Oracle()
today = oracle.timestamp(new Date() - 1)
db = oracle.getStarsProd()

sql = "select * from job_log where start_time > ? order by job_id" db.eachRow(sql, [ today ]) {job ->
print "$job.job_id $job.start_time $job.end_time"
println "\t$job.row_count\t$job.error_count\t$job.job_name"
}

This simple script dumps jobs run in the current day. The database package is a collection of groovy scripts that contain oracle specific connections and utilities. More on the specifics of what's in that package in another post...