Background
Panel |
---|
Born-digital collections present a challenge to traditional ways of describing content and making it discoverable. Disks, drives, directories, etc. may contain many thousands of files which would be difficult to describe in detail in an EAD. The approach described here starts with an EAD that contains only a single reference to the born-digital portion of a collection, expressed as a specific Series within the larger collection. The Forensic Toolkit (FTK) software is used to produce both disk images and related analysis files, plus detailed technical information about individual files on the media. The output of this process is augmented, transformed, and eventually translated into a set of digital objects in DOR (Stanford's Fedora-based Digital Object Registry) and shared objects for Hypatia. A separate process is used for converting the Collection EAD into metadata objects representing the full context of the born-digital and other materials, with links made between the containers and the detailed objects. |
Section |
---|
Column |
---|
| The Stanford Stephen J. Gould collection will be used to describe the initial implementation of this process. It is believed to represent a template for similar work with other born digital collections going forward. The directory output for the Gould collection contains |
|
...
the EAD and the content and metadata files for both Media and file objects (irrelevant files not shown): - M1437 Gould
- Computer Media Photo
- Disk Image
- CM001.001[.dd]
- CM001.001.csv
- CM001.001.txt
- (etc)
- Display Derivatives
- EAD
- FTK xml
- files
- Report_transformed.xml
- Report.fo
- Report.xml
|
|
...
Sample of transformed output:
Note that the first 2 directories map to objects describing the physical media and will be the source of creating the "unprocessed" collection, while Display Derivatives and FTK files map to individual file content & description and will be used to create the "processed" collection. |
Column |
---|
| The Import/conversion process will produce this arrangement of media and file objects in DOR. These are shown in bold; we are not concerned here how the Collection and Series nodes themselves get created, nor how much of the rest of the EAD is also represented as objects: - Collection object
- Series set -- Series 1 ..."
- :
- Series set -- "Series 6: Born Digital Materials"
- Media object 1
- File object 1
- File object 2
:
- Media object 2
- :
Any intellectual arrangement of this information -- categorization into genres (correspondence, novels, etc) for instance, or tracing iterations of a work across devices, etc -- will be a separate process of augmenting the descriptive metadata for these objects. |
|
Each <file> segment will be a the basis of a separate DOR or Hypatia Digital Object. The differences are:
- Stanford DOR (Digital Object Registry) objects are metadata-only, with content externally managed. Objects at Stanford will not have "content" datastreams.
- Stanford objects have an identityMetadata datastream that may or may not be present in Hypatia demo objects. Regardless, it is not a standard part of Hydra-compliant objects.
Collection and Series objects
The Collection and Born-Digital Series objects themselves are created first, ahead of FTK processing. All FTK processed materials for a collection are processed together and are members of the Born-Digial Series set. Media objects must be linked to the appropriate series via an isMemberOf relationship.
Media (e.g. Disk Image) objects
The FTK processing must first create a set of media objects representing the physical media (hard drive, diskette, etc) on which the files were found. This has been described as a view of the "unprocessed" collection, meaning it has not been processed down to the individual units of content, the separate files.
Note that a Media object has characteristics of
- an "item" -- it represents a unit of meaning and has content "parts" as separate objects
- a "set" -- it has object related to it as members ... should we consider a specialized relationship for this?
Sample of the starting lines of the .txt file describing the media object.
Panel |
---|
Created By AccessData® FTK® Imager 3.0.1.1467 110406 Case Information: Acquired using: ADI3.0.1.1467 Case Number: M1437 Evidence Number: CM004 Unique Description: Examiner: Peter Chan Notes: 5.25 inch Floppy Disk |
Note that some of this information is repeated as header information in the file-level FTK report below.
From: Disk Image // CMnnn.001.txt | maps to
| notes |
---|
Evidence Number: CM004
| descMetadata <mods:title>
| Would correspond to EAD <c><unittitle>
|
Evidence Number: CM004 | descMetadata <mods:identifier type="???>> | Would correspond to EAD <c><unitid> |
Case Number: M1437 | DC <dc:identifier>hypatia:M1437</dc:identifier> descMetadata <mods:identifier type="local">M1437</mods:identifier> | |
Notes: 5.25 inch Floppy Disks
| descMetadata <mods:physicalDescription> <mods:extent>
| Would correspond to EAD <physdesc> in a node describing the media.
|
(implied)
| RELS-EXT isMemberOfCollection | A link to the Collection object
|
(implied)
| RELS-EXT isMemberOf
| A link to the Series object |
identityMetadata – label = colleciton context...
File objects
File objects are the node objects representing individual files. The atomistic model would have these objects constructed as a parent (metadata) object and a child (content) object. For simplicity, we will create these File objects as a single object, combining the Hydra commonMetadata and genericContent models.
Sample of transformed FTK file (though the converison modules will not use this as input, but rather work from the same mapping rules that express this output).
Panel |
---|
<ftk_report |
Panel |
---|
Wiki Markup | <?xml version="1.0" encoding="UTF-8"?>
<ftk_report xmlns:fo="http://www.w3.org/1999/XSL/Format">
<Series>Series 6</Series>
<file>
<filename>BU3A5</filename>
<Item_Number>1004</Item_Number>
<series>Series 6: Born Digital Materials</series> <collection_title>Stephen Jay Gould papers</collection_title> <callnumber>M1437</callnumber> <file> <filename>BU3A5</filename> <item_number>1004</item_number> <filepath>CM006.001/NONAME \ [FAT12\]/\[root\]/BU3A5</filepath>
<filesize>35654</filesize>
<disk_image_no>CM006</disk_image_no> <filesize>35654</filesize> <filesize_unit>B</filesize_unit> <file_creation_date>n/a</file_creation_date>
<file_accessed_date>n/a</file_accessed_date>
<file_modified_date>12/8/1988 6:48:48 AM (1988-12-08 14:48:48 UTC)</file_modified_date>
<MD5_Hash>976EDB782AE48FE0A84761BB608B1880</MD5_Hash>
<Restricted>False</Restricted>
<Label>\[Medium\] 5.25 inch Floppy Disks</Label>
<Label>\[Type\] Books</Label>
<Label>\[Title\]The Burgess Shale and the Nature of History</Label>
<filetype>WordPerfect 4.2</filetype>
<Duplicate_File> </Duplicate_File>
<Export_path>files\BU3A5.wp</Export_path>
</file>
<file>
<filename>BUR3-1</filename>
<Item_Number>3005</Item_Number>
<filepath>CM005.001/NONAME \[FAT12\]/\[root\]/BUR3-1</filepath>
<filesize>92745</filesize>
<file_creation_date>n/a</file_creation_date>
<file_accessed_date>n/a</file_accessed_date>
<file_modified_date>12/8/1988 6:35:06 AM (1988-12-08 14:35:06 UTC)</file_modified_date>
<MD5_Hash>D3EB7E35856697B8F697193DB2CB4D83</MD5_Hash>
<Restricted>False</Restricted>
<Label>\[Medium\] 5.25 inch Floppy Disks</Label>
<Label>\[Type\] Books</Label>
<Label>\[Title\]The Burgess Shale and the Nature of History</Label>
<filetype>WordPerfect 4.2</filetype>
<Duplicate_File> </Duplicate_File>
<Export_path>files\BUR3-1.wp</Export_path>
</file>
|
Each <file> segment will be a the basis of a separate Hypatia Digital Object. This is a clean example where the atomistic model adds overhead (separate metadata and content objects) and an integrated object combining commonMetadata and genericContent should be considered.
Information from: Disk Image // CMnnn.001.txt | |
---|
<series>Series 6: Born Digital Materials
| Matches EAD series <unittitle> Part of <location> (1)
|
<collection_title>Stephen J. Gould Papers
| Matches EAD <archdesc <title> Part of <location> (1)
|
<note>5.25 inch Floppy Disks</note>
| from Disk Image // CM001.001.txt -- FTK header Value will be the same as "Medium" at the file level.
|
<callnumber>M143</callnumber> | |
<md5_hash>976EDB782AE48FE0A84761BB608B1880</md5_hash> <restricted>False</restricted> <access_rights>Public</access_rights> <title>The Burgess Shale and the Nature of History </title> <filetype>WordPerfect 4.2</filetype> <duplicate_File> </duplicate_File> <export_path>files\BU3A5</export_path> </file>
|
Information from: FTK xml // Report_transformed.xml | maps to (within item objects)
| notes
|
---|
<filename>BU3A5</filename>
| descMetadata <mods:identifier type="filename">BU3A5</mods:identifier>
| this is the original file name as it appeared on the original media. |
Information from: FTK xml // Report_transformed.xml | |
---|
<filename>BU3A5</filename>
|
|
<Item_Number>1004</Item_Number>
| descMetadata <ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="cfd513f6-71e5-4859-ae1f-1ea51c1a085e"><ac:plain-text-body><![CDATA[<mods:identifier type="ftk_id">1004</mods:identifier>
| internal FTK reference only, to disambiguate references in the FTK report
|
<filepath>CM006.001/NONAME [FAT12]/[root]/BU3A5</filepath> original file in FTK xml // files ]]></ac:plain-text-body></ac:structured-macro> display derivatives in Display Derivaties named using <item_number> <ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="01d901bd-467f-40b0-a331-ab4adb109033"><ac:plain-text-body><![CDATA[tale fully qualified filename from portion after [root]
| ]]></ac:plain-text-body></ac:structured-macro> |
<disk_image_no>CM006</disk_image_no>
| |
<filesize>35654</filesize>
| |
| descMetadata <mods:location> <mods:physicalLocation type="filepath">CM006.001/NONAME [FAT12]/[root]/BU3A5</mods:physicalLocation> </mods:location>
| location of file on original media
everything after [root] can be taken as the fully qualified filename
|
<disk_image_no>CM006</disk_image_no>
| Naomi thinks this should be handled with a link to the media (disk image) object
RELS-EXT <isMemberOf xmlns="info:fedora/fedora-system:def/relations-external#" rdf:resource="info:fedora/hypatia:(id for media object CM006)"/>
descMetadata <mods:location> (1)
| This token, taken from the head of the <filepath>, is the only data link between the FTK output for a file object and the corresponding media object. We want a data link in descriptive metadata as well as an RDF link to the corresponding object.
|
<filesize>35654</filesize>
| descMetadata - human friendly <mods:physicalDescription> <mods:extent>35654</mods:extent> </mods:physicalDescription>
contentMetadata - for machine <contentMetadata> <resource> <file size="35654" | This should be a human friendly version of the file size. The machine friendly version is in contentMetadata.
Could be used by conversion to compare against the file size as computed locally, a quick check prior to checksum validation?
|
<filesize_unit>B</filesize_unit> | use to determine filesize in bytes (convert to bytes if nec)
| Needed to correctly interpret <filesize>, if used | <filesize_unit>B</filesize_unit> |
|
<file_creation_date>n/a</file_creation_date>
| descMetadata <mods:originInfo> <mods:dateCreated>n/a</mods:dateCreated> </mods:originInfo> | |
<file_accessed_date>n/a</file_accessed_date>
| descMetadata <mods:originInfo> <mods:dateOther>n/a</mods:dateOther> </mods:originInfo>
| |
<file_modified_date>12/8/1988 6:48:48 AM (1988-12-08 14:48:48 UTC)</file_modified_date>
| descMetadata <mods:originInfo> <mods:dateModified>12/8/1988 6:48:48 AM (1988-12-08 14:48:48 UTC)<\mods:dateModified> </mods:originInfo>
| |
<MD5_Hash>976EDB782AE48FE0A84761BB608B1880</MD5_Hash>
| contentMetadata <file> <checksum type="md5">976EDB782AE48FE0A84761BB608B1880</checksum> </file>
| Used for checksum validation of a file during processing. This value will eventually be part of contentMetadata, but probably not as a value transferred from here.
|
<restricted>False</Restricted>
| | true=visible staff only, not discoverable .... Hypatia only
|
<label name="Medium">5.25 inch Floppy Disks</label>
| Part of <location> (1) |
<label name="Type">Books</label>
| tag
|
<type>Books</type>
| Naomi sez: mods doc says this is controlled vocab, so this won't work ... [http://www.loc.gov/standards/mods/mods-outline.html#typeOfResource ] descMetadata <mods:typeOfResource>Books</mods:typeOfResource>
|
|
<title>The Burgess Shale and the Nature of History</title>
| descMetadata <mods:relatedItem displayLabel="Appears in" type="host"> <mods:titleInfo> <mods:title>The <label name="Title">The Burgess Shale and the Nature of History</label>
| | /mods:title> </mods:titleInfo> <mods:relatedItem>
| This is not the title of the file or the file content directly, but the author's title to which the file relates.
|
<filetype>WordPerfect 4.2</filetype>
| descMetadata <mods:note displayLabel="filetype"> | |
<Duplicate_File> </Duplicate_File>
| | * blank, null value or empty string - original file , not a duplicate "Primary" - possibly indicates Primary file to keep/store "Secondaryis unique in collection, no duplicates * "M" - The main file in a duplicate relationship. Neither better nor worse than the duplicate file, but simply the file examined first. * "D" - indicates a duplicate file to be ignored --> ignore for now .
Note that this is content duplication based on having the same checksum (name conflicts are different and handled another way). The two files may or may not have the same name. It is desirable to have a note and/or relationship in each record indicating the presence of a duplicate file in the collection. Details tbd.
|
<export <Export_path>files\BU3A5.wp</Exportexport_path> | | The file as saved by FTK for further processing.
|
(implied)
| RELS-EXT isMemberOf
(see also <disk_image_no>CM006</disk_image_no> ) | A link to the Media object |
(1) Location/container information -- for every file object created, create a <mods:location> description that places the resource in the context of the collection by combining collection name, intermediate series/group/etc name(s), and the ID+description of the media on which the file resides:
<location>collection-title - series-title - media-title (media-description)</location
e.g.,
<location>Stephen J. Gould Papers - Series 6: Born Digital Materials - CM006 (5.25 inch Floppy Disks)
Use this concept for objectLabel?
September 17 Update