IntroductionUsing the multicast journal transport, it is possible to achieve replication or read-only mirroring of Fedora servers via journaling. Journal entries are created on a read/write Fedora server (the leader), and sent to other Fedora servers (followers) operating in journal recover mode. ComponentsTo implement replication between Fedora servers, at present three components are necessary:
Leader (Master) Fedora RepositoryThe leader operates in full read/write mode, and creates journals for any write operations requested of it. This leader must be configured to use the multicast journal writer, and all follower RMI receivers must be listed in the journal configuration in The multicast journal writer is able to designate any transport as 'crucial' (i.e. Fedora will no longer service write requests if it fails), and at least one transport must be designated as crucial. Common practice is to have a local file transport writing to the leader's file system that is designated as crucial, with all remote RMI transports as not crucial. RMI Journal ReceiversRMI receivers listen for journal events from the leader, and write journal files into a directory containing journal entries. The size, duration, and name of each file is determine entirely by the leader server, since the server is responsible for sending 'open file' and 'close file' events. In the end, the journal files produced by a receiver are identical to that those created locally by the leader server or any other receiver. The receiver application is completely separate from Fedora and merely writes files to a directory that follower Fedora instances read at their convenience. Just as is the case for a Fedora instance logging journals to the local filesystem, journals that are open and still being written to will be prefixed with an _underscore. The RMI receiver application is distributed as a separate download in the Fedora distribution, or may be built from source by the java -jar RmiJournalReceiver.jar /path/to/journals The RMI receiver will run continuously, and will print any logging error messages to the standard output console. In production, it is recommended to run this as a daemon or service, and direct the output to a log file. Journal events are pushed from the server to the follower. As such, the machines on which the RMI receiver is deployed must have two ports available: 1099 and 1100. Firewalls on follower machines must allow connections to these ports. Follower (Slave) Fedora RepositoriesFollower repositories are simply Fedora instances that are configured to run in recover mode, and use a following journal reader. In their While the RMI receiver must run continuously, the follower Fedora server can be stopped and started at will. The consequence of shutting down a follower Fedora is merely that journal files will pile up in the specified directory. When started, the follower Fedora will quickly play back journals and empty the directory until caught up. The follower Fedora server is always in read-only mode. Query and read-only API operations are available at all times. Because journal files are read only when they are closed, any data written to the journal file will not appear in the follower until its file is closed, which depends on the leader configuration. Small journal file size limits and shorter lifetimes will result in the followers being closer to the leader state at any point in time. Also note that the follower Fedora, when playing back journals, behaves exactly as the leader server when it initially received each request. This has implications for JMS. For example, if a follower has messaging enabled, each journal operation _will_ result in a JMS message. If the follower has a Gsearch index, it will update from the follower exactly as one for the leader instance would. Thus, it is possible to replicate entire stacks of services around each follower. ScenariosThe replication scheme described here is useful in a number of situations: BackupsBecause data in Fedora is distributed between the filesystem and multiple databases (object registry database, resource index, etc), it is nearly impossible to obtain a completely consistent snapshot/dump of the all Fedora content from a live system. Thus, it is helpful to deploy a follower that can be shut down as necessary to allow for database and filesystem dumps to be assembled and archived - allowing the leader to run uninterrupted. Upgrades and MaintenanceFollowers may be manually turned into leaders by swapping Load BalancingFor high-volume mostly read-only sites, followers may be used to provide data as part of a load-balancing scheme. The caveat here is that the followers will be slightly behind the leader as the journal files are written, closed, and processed. Small journal sizes and short lifetimes will mitigate this issue slightly. |