Node Clean Up Scripts
As you work out your cluster configuration, you will find it useful to wipe out the data persisted on nodes and start over clean. This can be easily accomplished with a short script that removes certain files. Below is an example of such a script.
/etc/init.d/tomcat7 stop sleep 20 killall java > /dev/null sleep 5 rm --preserve-root -f /var/lib/tomcat7/logs/* rm --preserve-root -rf /data/fcrepo/* rm --preserve-root -rf /var/lib/tomcat7/work/Catalina/*
You will need to adjust the folders to match your own configuration. However, the key ingredients are stopping the Fedora process, then wiping out the persistent data folder and your webapp's temp folder. This script also cleans out old logs, making it easier to find logs for the current configuration.
Simultaneous Terminal Sessions
A program like terminator (Gnome) can help with establishing ssh sessions to all your nodes quickly. Such programs will also allow you to group those sessions and then broadcast keystrokes to all of them at once. This makes quick configuration tests a breeze. You just have to sometimes wait and make sure that all sessions are keeping up with your commands.
Terminator in particular will let you create a "layout" that starts up your terminals in a desired window arrangement and runs the ssh commands.
Please add more terminal tools to this list, if you have a favorite.
Puppet is a powerful way to manage a large cluster and can make it very easy to deploy new hosts. A puppet module is under development that will provide configuration templates for new fedora hosts.