tag:blogger.com,1999:blog-63809974209319430412024-03-19T07:53:47.540+00:00Wellcome Digital LibraryChristy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.comBlogger28125tag:blogger.com,1999:blog-6380997420931943041.post-72235719037161382642012-10-19T15:27:00.000+01:002012-10-19T15:42:32.095+01:00Digesting Ingest<br />
<div style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<a href="http://images.wellcome.ac.uk/indexplus/obf_images/cc/74/5fd547137110e7a2c4776a064264.jpg" target="_blank"><img alt="Alcohol metabolism" height="254" src="http://images.wellcome.ac.uk/indexplus/obf_images/cc/74/5fd547137110e7a2c4776a064264.jpg" width="320" /></a></div>
<i>Harkiran Dhindsa, and Rioghnach Ahern, Digital Ingest Officers working on the Wellcome Digital Library, describe their experiences using Goobi on a daily basis, and some of the lessons learnt as we scaled up into full production over this past summer:</i><br />
<br />
<a href="http://www.digiverso.com/en/products/goobi" target="_blank">Goobi </a>is a workflow-based management system that allows us to track and manage the <a href="http://wellcomedigitallibrary.blogspot.co.uk/2012/08/half-million-milestone.html" target="_blank">workflows</a> for various digitisation projects, be that archives, books, film or audio files. Many steps are fully automated, including the conversion of TIFF to JPEG2000 and the ingest of content into our repository <a href="http://wellcomedigitallibrary.blogspot.co.uk/2011/07/preserving-our-digital-assets-1-sdb4.html" target="_blank">Safety Deposit Box</a>.<br />
<br />
We found the user interface of Goobi to be intuitive. Training in basic ingest processes was quick. A number of the team are using this system. With regular usage, we were working efficiently and became familiar with the functionality. <a href="http://www.loc.gov/standards/mets/" target="_blank">METS </a>editing is facilitated through a web form which allows JPEG images of individual pages to be viewed.
Using such a system eliminates the need to keep separate spread-sheets. Because of the way Goobi tracks the workflow by registering each step, it means different staff can continue with tasks at any open step. At any point if an error is noticed - for example a missing image in a book - a correction message can be sent back along the workflow to the appropriate person.<br />
<br />
Goobi produces METS files, which describe objects including their access and license status. Although Goobi writes the METS files, the structure of an object is created manually, depending on the project. Much of our time is spent working on METS editing, particularly in adding restrictions for material which contains sensitive data. Goobi can handle a number of projects at the same time, so we can easily switch between working on archives and books. It can handle different tasks simultaneously. For example, an ingest officer can let an image upload task run for, let’s say, an archive collection, while continuing to edit METS data on books.<br />
<br />
<h3>
Lessons learnt</h3>
As daily users of Goobi, here are some of the lessons we have learnt:<br />
<br />
Prior to import into Goobi, catalogued items are photographed and then the digitised images are checked for <a href="http://wellcomelibrary.blogspot.co.uk/2012/06/sensitivity-assessment-finer-points-of.html" target="_blank">data sensitivity</a>. In the early stages of the project, areas that could be improved for a more accurate and efficient workflow became obvious. Amongst one of the first archival collections to be digitised, some of the images that were available as a backlog, and were due to be uploaded into Goobi, did not reflect the archive catalogue (CALM). This was because changes had been made to the catalogue after photography. The lesson learnt from this experience is that <b>photography should only be carried out after cataloguing has been truly completed</b> so that the arrangement of material is firmly established.<br />
<br />
To upload images into Goobi, they are first copied from a working network directory to a temporary drive created by Goobi for the user who has accepted the upload task. This process can be terminated by other activities if the network to the local PC is running at full capacity. When this happens we have to redo the transfer, taking extra time to complete the task. <b>Thankfully, this image upload task will be automated in the future</b>, bypassing the local PCs completely. However, the running of several tasks simultaneously will still be limited by server capacity when uploading large files.<br />
<br />
After METS editing was completed on one of the archive collections, we were given further sensitivity data. To add these new sensitivity restrictions, we had to “roll back” processes that had already been ingested, thereby re-running part of the workflow. It is very easy to prompt a second ingest into the digital asset management system in the process, resulting in duplicated sets of files, as the roll-back process is less intuitive and not intended for regular use. Again, we have learned an important lesson. <b>It will always be necessary to edit METS files</b>. Changes to the workflow steps in Goobi to make this more straightforward would be useful, but it would be even better to finalise sensitivity lists before METS editing is completed in order to minimise duplication of effort.<br />
<br />
A workflow system such as Goobi becomes imperative when ingesting mass collections of archives and books. Images that have gone through the complete ingestion process in Goobi will be accessed online via the <a href="http://wellcomedigitallibrary.blogspot.co.uk/2012/05/developing-player-for-wellcome-digital.html" target="_blank">Player</a>. Seeing the images in an attractive interface is a satisfactory part of this work as this is where all of the different tasks come to fruition: the digitised archives and books available to the public to view in a user-friendly form — soon to be publicly available!
<br />
<br />
<i>Authors: Harkiran Dhindsa and Rioghnach Ahern</i>Christy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.com13tag:blogger.com,1999:blog-6380997420931943041.post-48622346421670796392012-08-13T16:29:00.000+01:002012-08-13T16:29:14.989+01:00Half a million milestoneWe have recently completed processing half a million images in our workflow system (Goobi) ahead of making our new website available to the public later on this year. These images are the product of on-site digitisation of our archives and genetics book collections for the WDL pilot programme.<br />
<br />
In previous posts on this blog we have talked about our <a href="http://wellcomedigitallibrary.blogspot.co.uk/2011/09/preserving-our-digital-assets-2-worm.html">storage system</a>, <a href="http://wellcomedigitallibrary.blogspot.co.uk/2012/05/serving-servers-technical.html">server environment</a>, our <a href="http://wellcomedigitallibrary.blogspot.co.uk/2011/07/preserving-our-digital-assets-1-sdb4.html">digital asset management system</a>, and our image "<a href="http://wellcomedigitallibrary.blogspot.co.uk/2012/05/developing-player-for-wellcome-digital.html">player</a>," but central to all of this is our workflow system, an enhanced and customised version of the open-source software <a href="http://www.digiverso.com/products/goobi?lang=en">Goobi </a>developed originally at the University of G<em style="background-color: white; font-family: arial, sans-serif; font-size: small; font-style: normal; font-weight: bold; line-height: 16px;">ö</em>ttingen, and supported by Intranda.<br />
<br />
Goobi is an extremely flexible database system that allows us to create and modify workflows (series of tasks both manual and automatic) for specific projects. These tasks can be recorded in Goobi (such as "Image capture"), done <i>by</i> Goobi (such as "Image conversion JPEG" for viewing images in the Goobi system itself), or can be initiated by Goobi (such as "SDB ingest" which triggers our digital asset management system - SDB - to ingest content).<br />
<br />
We are currently working on ingesting images for three different workflows, described and simplified below.<br />
<br />
<b>Archive backlog</b><br />
This workflow allows us to ingest all the images we created from archives since the project began in December 2009 up to May 2012 - around 320,000 of them - from eight collections. So far, we have finished processing around 250,000 images from this set.<br />
<br />
The steps for each item to be ingested are as follows (automatic steps are in italics):<br />
<br />
<ol>
<li>Export MARC XML as a batch from the Library catalogue (per collection)</li>
<li>Import metadata as a batch into Goobi</li>
<li>Import JPEG2000 images one folder at a time from temporary storage to Goobi-managed directories</li>
<li><i>Goobi converts JPEGs for viewing in the Goobi interface</i></li>
<li>Check that images are correctly associated with the metadata</li>
<li>Add access control "code" to items with sensitive material (restricted or closed)</li>
<li><i>Trigger SDB ingest, passing along key information to enable this</i></li>
<li><i>Import administrative/technical metadata from SDB after ingest</i></li>
<li><i>Export METS files to be used by the "player"</i></li>
</ol>
<div>
<b>Archives digitisation</b></div>
<div>
This workflow deals with the current digitisation by tracking and supporting activities from the very beginning of the digitisation process. So far, we have imported very few images for this project, having just started using this workflow in earnest. The steps include:</div>
<div>
<ol>
<li>Export MARC XML as a batch from the Library catalogue (per collection)</li>
<li>Import metadata as a batch into Goobi</li>
<li>Group metadata into "batches" in Goobi for each archive box (usually 5 - 10 folders or "items")</li>
<li>Track the preparation status at the box level (in process/completed) and record information for the next stage (Image capture)</li>
<li>Track photography status at the box level and record information for the next stage (QA)</li>
<li>Track QA at the box level and return any items to photography if re-work is required</li>
<li>Import TIFF images via Lightroom (which converts RAW files and exportsTIFF files directly into Goobi)</li>
<li><i>Convert TIFFs to JPEGs</i></li>
<li>Check that images are correctly associated with the metadata </li>
<li>Add access control "code" to items with sensitive material (restricted or closed)</li>
<li><i>Convert TIFFs to JPEG2000</i></li>
<li><i>See 7-9 above</i></li>
</ol>
<div>
<b>Genetics books</b></div>
<div>
The other half our ingest effort has focused on the Genetics books. We have imported into Goobi over 250,000 images from this collection since digitisation began in February of this year. This workflow is very similar to Archives digitisation - containing steps related to the entire end-to-end process. The main differences being that as the digisation is being done by an on-site contractor, images are delivered to us as TIFFs, and while there are no sensitivity issues, there is metadata editing to add structure to aid navigation, and a range of "conditions of use" codes depending on the restrictions copyright holders request us to make.</div>
<div>
<br /></div>
<div>
<ol>
<li>Export MARC XML as a batch from the Library catalogue (whole collection)</li>
<li>Import metadata as a batch into Goobi</li>
<li>Track preparation status at the book level and record information for next stage (Image capture)</li>
<li>Track image capture status at the book level</li>
<li>Track QA status (QA is done on the TIFFs supplied by the contractor)</li>
<li>Import TIFF images one folder at a time from temporary storage to Goobi-managed directories</li>
<li><i>Convert TIFFs to JPEGs</i></li>
<li>Check that images are correctly associated with the metadata </li>
<li>Associate images with structural metadata (covers, titlepage, table of contents) thereby enabling navigation to these elements in the "player"</li>
<li>Add page numbering </li>
<li>Add licence code to books that have use restrictions (such as no full download allowed) as per requests by copyright holders</li>
<li><i>As above</i></li>
</ol>
</div>
<div>
<b>We haven't got it all figured out yet</b></div>
<div>
Other workflows we have not yet put into production, include born digital materials, A/V materials, items with multiple copies/volumes or "parts" (such as a video and its transcript), and manuscripts. We are looking at implementing new or different functionality in Goobi in the near future as well, including JPEG2000 validation using the <a href="https://github.com/openplanets/jpylyzer/">Jpylyzer </a>script, automated import of images, and configuring existing functionality in Goobi to support OCR and METS-ALTO files to name a few. These changes are aimed at minimising manual interaction with the material to save on time and improve accuracy. </div>
</div>
<div>
<br /></div>
Christy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.com3tag:blogger.com,1999:blog-6380997420931943041.post-45657819075052392772012-07-04T09:06:00.003+01:002012-07-04T09:06:47.294+01:00OCRing typescript: A benchmarking test with PRImADuring our pilot phase, we plan to add over 1 million images of archives to the Wellcome Library website, all created during the 20th century. Much of this is comprised of typescript: letters, drafts of papers and articles, research notes, memos, and so on. Key to improving discoverability of these papers is OCR (optical character recognition), which allows us to encode the words as text and include them in a full-text index. We set out to test whether typescript material would provide accurate OCR results that could be included in our index.<br />
<div>
<br /></div>
<div>
<h3>
OCR technology</h3>
OCR works by segmenting a block of text to the individual character level and then comparing the patterns to a known set of characters in a wide range of typefaces. The accuracy of character recognition relies on the source information having clear-cut, and common, letter forms. The accuracy of word recognition relies on both character recognition, and the availability of comprehensive dictionaries for comparison. OCR software can compare these words to dictionaries to enhance word recognition, and also to estimate accuracy rates (or levels of confidence).<br />
<br />
When it comes to good quality, clearly printed text, OCR can be extremely accurate even without any human intervention - with rates of 99% or higher for modern printed content (less than one word out of one hundred words having at least one inaccurate character), reducing as you recede in time to 95% for 1900 - 1950 printed material, and lower for 19th century material (see <a href="http://www.dlib.org/dlib/july09/munoz/07munoz.html" target="_blank">Tanner, Munoz and Ros, 2009</a>). For some formats, OCR is worse still - as the document mentioned above shows, 19th century newspapers may only reach 70% significant word accuracy (words not including "stop" words such as definite/indefinite articles, and other non-search terms).<br />
<br />
<h3>
Typescript testing</h3>
Regarding our archival collections, there is a wide range of content that is theoretically OCR'able. Some will OCR very well, such as professionally printed matter. But much non-handwritten content is by the nature of the age of this material in a typescript form. We had no idea how well this type of content would OCR. To find out we commissioned Apostolos Antonacopoulos and Stefan Pletschacher based at the University of Salford and members of <a href="http://www.primaresearch.org/" target="_blank">PRImA </a>(Pattern Recognition and Image Analysis Research) to do a benchmarking exercise from which we could determine whether we could rely on raw OCR outputs, should not OCR this type of material at all, or to test various methods to improve OCR'ability (such as post-processing of particular images).<br />
<span style="background-color: white;"><br /></span><br />
<span style="background-color: white;">Apostolos and Stefan chose a selection of 20 documents from a larger sample we provided originating from our Mourant and Crick digitised collections. These 20 documents where manually transcribed using the <a href="http://www.primaresearch.org/tools.php" target="_blank">Aletheia </a>groundtruthing tool for comparison to the output of three OCR engines, Abbyy <a href="http://www.abbyy.com/ocr_sdk/?gclid=CI2rpZ_z_bACFVMTfAodAXXqNw" target="_blank">FineReader </a>Engines 9 and 10 and <a href="http://code.google.com/p/tesseract-ocr/" target="_blank">Tesseract</a>, open source OCR software.</span><br />
<br />
The results of the OCR benchmarking test show that original, good quality typescript content can reach up to <b>97% significant word accuracy </b>with Abbyy Fine Reader Engine 10 (such as this example below):<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOF_hyWpYORTvCPzhx1j29TYhABAH_KVbLEoTNzO_faaxR-kKQ4T8761f4BVCry_okXdTtWw3dFPGhD42_nMoZXw_4Hm7rPcMV7wveiikxQKAnkkLydZZ1aV1zo68R73wfLbvCK7a2QmsB/s1600/Archive-OCR-1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="166" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOF_hyWpYORTvCPzhx1j29TYhABAH_KVbLEoTNzO_faaxR-kKQ4T8761f4BVCry_okXdTtWw3dFPGhD42_nMoZXw_4Hm7rPcMV7wveiikxQKAnkkLydZZ1aV1zo68R73wfLbvCK7a2QmsB/s400/Archive-OCR-1.jpg" width="400" /></a></div>
<br />
<span style="background-color: white;">At the bottom end, carbon copies with fuzzy ink can result in <b>virtually 0% accuracy</b> in any OCR software:</span><br />
<span style="background-color: white;"><br /></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTeKf4Vg5RzCSRvKz7AXgNascYBUzZKjCPtIQSOD2ptYQdnSnvcMoMwmDCgwWYacUVRAuMS_CXf-OxsQVVE-ajgO7-jYpeGEtHYK-ZA7-u7xBOBYMzcOYQF0LsfxBdSIL3yDm5swXSMAX4/s1600/Archive-OCR-2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="178" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTeKf4Vg5RzCSRvKz7AXgNascYBUzZKjCPtIQSOD2ptYQdnSnvcMoMwmDCgwWYacUVRAuMS_CXf-OxsQVVE-ajgO7-jYpeGEtHYK-ZA7-u7xBOBYMzcOYQF0LsfxBdSIL3yDm5swXSMAX4/s400/Archive-OCR-2.jpg" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
What was pleasantly surprising was how the average-quality and poorer content fared. On the better end of the scale we have <b>93% accuracy</b> despite some broken characters:</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbyD3BhEVnnMfkq9dJLST8wBY_UaJwYzERz5IJVjWro2RnYzBbw3S8N2pY8u1U-5mPS8djMi2O6tKnwS-QXkFwOlQwyP0Z5gb8PglWMD1wgalvTZbkQABKyM_R6XKH36hU6Yi9Baq_UIFL/s1600/Archive-OCR-3.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="155" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbyD3BhEVnnMfkq9dJLST8wBY_UaJwYzERz5IJVjWro2RnYzBbw3S8N2pY8u1U-5mPS8djMi2O6tKnwS-QXkFwOlQwyP0Z5gb8PglWMD1wgalvTZbkQABKyM_R6XKH36hU6Yi9Baq_UIFL/s400/Archive-OCR-3.jpg" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
And here we have poorer quality typescript producing <b>72% accuracy </b>with many faint and broken characters:</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifKHvmo4_xFlJdbrv1AkKfhHmFGyhx6a0Vux2Vuk_ksswB0d7FUuwp1XCg1sv8kBSlrkbvlapGAD34WFxHVfP9MAm2cbhFn7JW5KxOuOBQO8FY9CMGpVhFrcLECtNrhXfAznFdjpAObQkT/s1600/Archives-OCR-4.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="141" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifKHvmo4_xFlJdbrv1AkKfhHmFGyhx6a0Vux2Vuk_ksswB0d7FUuwp1XCg1sv8kBSlrkbvlapGAD34WFxHVfP9MAm2cbhFn7JW5KxOuOBQO8FY9CMGpVhFrcLECtNrhXfAznFdjpAObQkT/s400/Archives-OCR-4.jpg" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<b>The average rate for 16 images of good to poorer quality typescript is 83% significant word accuracy (excluding the carbon copies)</b>. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Accuracy levels are reported here according to the results from Abbyy FineReader Engine 10 on the "typescript" setting. The reported accuracy rates covers all the visible text on the page including letterheads, pre-printed text such as contact details, text overwritten by manual annotations and so on. Naturally, errors are more likely to occur in these areas, which (except in the case of text overwritten by manual annotations) are not of much significance in terms of indexing and discoverability. Further tests would be required to determine what the accuracy rate is with these areas excluded. For example, it may be possible to digitally remove the annotations in this draft version of Francis Crick and James Watson's "<a href="http://bit.ly/MTdzvq">A Structure for DNA</a>" to raise the overall accuracy rate (currently only 45%):</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgg5kWFrlhWib-NKgq19m6xD4SSTaOel41AXExjyf6BcsgTS8gVRny8Rey-g0jOOlGFej5YyCPh9WQHJonFz-sWSdxp6vv5Yq1kGw4Axmatlzl_6KnyIDaH_XnKwyaASuIGvO0QuTyYzZif/s1600/Archive-OCR-5.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgg5kWFrlhWib-NKgq19m6xD4SSTaOel41AXExjyf6BcsgTS8gVRny8Rey-g0jOOlGFej5YyCPh9WQHJonFz-sWSdxp6vv5Yq1kGw4Axmatlzl_6KnyIDaH_XnKwyaASuIGvO0QuTyYzZif/s640/Archive-OCR-5.jpg" width="514" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
There is some variation between FineReader 9 and 10 where one or the other may have a small advantage with a few cases showing as much as a 30% difference. Overall, there is only 1% difference when looking at averages between 9 and 10. Tesseract, on the other hand, was far less accurate especially for the poorer quality typescript (roughly half as accurate overall).</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
There are a few things we could do to improve accuracy: </div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<ol>
<li>Incorporate medical dictionaries to improve recognition and confidence of scientific terms</li>
<li>Enhance images to "remove" any annotations prior to OCR'ing</li>
<li>Develop a workflow that would divert images down different paths depending on content ("typescript" path, FineReader 9 or 10, enhancements to be applied or not applied, etc.)</li>
</ol>
<span style="background-color: white;">We may find that an average of 83% word accuracy overall is perfectly adequate for our needs in terms of indexing terms and allowing people to discover content </span><span style="background-color: white;">efficiently</span><span style="background-color: white;">. Further investigation is required, but this report has given us a good foundation from which to press on with our OCR'ing plans.</span><br />
<span style="background-color: white;"><br /></span><br />
<span style="background-color: white;">These digital collections are not yet available online, but will be accessible from autumn 2012.</span></div>Christy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.com19tag:blogger.com,1999:blog-6380997420931943041.post-28604178883400912932012-05-15T17:53:00.000+01:002012-05-15T17:53:56.603+01:00Serving servers: a technical infrastructure plan<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4vCGYTtFJRnk2vK4f8aDrFm-vPqcBs5MgXHSwmjvXgE-DDeQsLdvYcbEKjUvm4dpTr8kppHpfEVlwkseCblBJZULLQFeMhIkVGSbdrrAFh67OFCRBsleoYHT3Sm667iEH7pKKfViX5eL3/s1600/SustainableEnergy.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="" border="0" height="186" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4vCGYTtFJRnk2vK4f8aDrFm-vPqcBs5MgXHSwmjvXgE-DDeQsLdvYcbEKjUvm4dpTr8kppHpfEVlwkseCblBJZULLQFeMhIkVGSbdrrAFh67OFCRBsleoYHT3Sm667iEH7pKKfViX5eL3/s400/SustainableEnergy.jpg" title="Sources of sustainable energy, Digital artwork by Rowena Dugdale, from Wellcome Images" width="400" /></a></div>
<br />
As we aim to provide a fast, efficient and robust technical
architecture for the Wellcome Digital Library, the Wellcome Trust IT department
has been working closely with our software suppliers to specify a suitable
server architecture. This work is still in progress, but we now have the
skeleton idea of how many servers we are likely to need and for what purposes.
The scale of the architecture requirements shows that setting up and delivering
digital content is a significant undertaking.<br />
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
In order to serve up millions of images, plus thousands of A/V
files, born digital content and the web applications that make them accessible,
we believe we’ll need around 17 (virtual) servers for the production
environment (the “live” services), and a further 10 servers for our staging and
development environments. In the production environment, nearly every server is
duplicated to ensure redundancy and a smooth delivery service, which is why the
numbers are so high. The content management system coupled with its SQL
database requires four servers, for example. The image delivery environment
needs six servers for data delivery, on-the-fly image conversion and tile
creation, and media proxy servers creating digital content URLs that divorce
the user-request mechanism from the actual content held on our servers for
security reasons.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Most of the servers run on Windows 2008, although our image server
(IIPImage) will run on Linux Ubuntu. The virtual servers share CPUs, but the
number of CPUs available mean that each server gets the equivalent of either 2
or 4 CPUs, leading to a total 48 CPU requirement (288 cores as each CPU has 6
cores) . RAM varies from 2GB to 8Gb depending on the anticipated usage of a
particular application on that server. The total RAM requirement for the
production architecture is estimated at 124Gb. These specifications are
currently our best guess, and will be tested in the weeks to come as we start
to deploy the hardware.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
The staging environment allows system upgrades, patches or new
development work to be applied and tested
separately from the live production environment. This means that any
changes can be tested thoroughly before changes are made publicly visible
and/or usable. Actual development work is carried out in the development
environment, before deployment for final testing on the staging servers. This
means that applications such as the web content management system and the
delivery system applications must be replicated in these two additional
environments, along with their server requirements.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<i><span style="font-family: Calibri, sans-serif;">With thanks to David Martin,
IT Project Manager, as the source of my information.</span></i>Christy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.com13tag:blogger.com,1999:blog-6380997420931943041.post-82807547018478938282012-05-11T15:37:00.000+01:002012-05-11T15:37:26.647+01:00Developing a player for the Wellcome Digital LibraryPrevious posts here have covered the digitisation of books and archives and the storage of the resulting files (mostly <a href="http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&sqi=2&ved=0CHYQFjAA&url=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FJPEG_2000&ei=hyOtT--xHsmX8QPr0YXnCg&usg=AFQjCNF--8QZEIp_jrHe5VCYibPV_i0FrA&sig2=cRdoEcEYm1jwVPH7cHl4GA" target="_blank">JPEG2000</a> images, but some video and audio too). Now it’s time to figure out how visitors to the Wellcome Library site actually view these materials via a web browser.<br />
<br />
The digitisation workflow ends with various files being saved to different Library back-end systems:<br />
<br />
<ul>
<li>The <a href="http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&sqi=2&ved=0CGIQFjAA&url=http%3A%2F%2Fwww.loc.gov%2Fstandards%2Fmets%2F&ei=LiOtT-HADoXb8gOlt-XWCg&usg=AFQjCNG06NGYoR5agDhZxOj5UoG4tXYkPA&sig2=NnpDaj4vL5x0DqIfiQPTXw" target="_blank">METS </a>file is a single XML document that describes the structure of the book or archive, providing metadata such as title and access conditions. </li>
<li>Each page of the book (or image of an archive) is stored as a JPEG2000 file in the Library’s asset management system, <a href="http://wellcomedigitallibrary.blogspot.co.uk/2011/07/preserving-our-digital-assets-1-sdb4.html" target="_blank">Safety Deposit Box</a> (SDB). Each image file in SDB has a unique filename (in fact a GUID), and this is referenced in the METS file. So given the METS file and access to the asset management system, we could retrieve the correct JPEG 2000 images in the correct order. </li>
<li>Additional files might be created, such as METS-ALTO files containing information about the positions of individual words on a digitised page; we’ll want to use this information to highlight search results within the text. </li>
</ul>
So how do we use these files to allow a site visitor to read a book?<br />
<br />
<h3>
Rendering JPEG 2000 files</h3>
Our first problem is that we can’t just serve up a JPEG2000 image to a web browser – the format is not supported. And even if it was, the archival JPEG2000 files are large: several megabytes each. The solution to this problem is familiar from services like Google Maps – we break the raw image up into web-friendly tiles and use them at different resolutions (zoom levels). When you use Google Maps, you can keep dragging the map around to explore pretty much anywhere on Earth – but your browser didn’t load one single enormous map of the world. Instead, the map is delivered to you as 256x256 pixel image files called tiles, and your browser only makes requests for those tiles that are needed to show the area of the map visible in your browser’s viewport. Each tile is quite small and hence very quick to download – here’s a Google map tile that shows the Wellcome Library:<br />
<br />
<a href="http://mt1.google.com/vt/lyrs=m@176000000&hl=en&src=app&x=65487&s=&y=43573&z=17&s=Ga">http://mt1.google.com/vt/lyrs=m@176000000&hl=en&src=app&x=65487&s=&y=43573&z=17&s=Ga</a><br />
<br />
Google Maps is a complex JavaScript application that causes your browser to load the right tiles at the right time (and in the right place). This keeps the user experience slick. We need that kind of user experience to view the pages of books.<br />
<br />
There are several JavaScript libraries available that solve the difficult problem of handling the viewport and generating the correct tile requests in response to user pan and zoom activity. We’ve settled on <a href="http://gallery.expression.microsoft.com/SeadragonAjax" target="_blank">Seadragon</a>, because we really like the way it zooms smoothly (via alpha blending as you move from one zoom level’s tiles to another). A very nice existing example of this is at the Cambridge Digital Library’s Newton Papers project:<br />
<br />
<a href="http://cudl.lib.cam.ac.uk/view/PR-ADV-B-00039-00001/">http://cudl.lib.cam.ac.uk/view/PR-ADV-B-00039-00001/</a><br />
<br />
This site uses a viewer built around Seadragon; an individual tile looks like this:<br />
<br />
<a href="http://cudl.lib.cam.ac.uk/content/images/PR-ADV-B-00039-00001-000-00105_files/11/3_2.jpg">http://cudl.lib.cam.ac.uk/content/images/PR-ADV-B-00039-00001-000-00105_files/11/3_2.jpg</a><br />
<br />
The numbers on the end indicate that this jpeg tile is for zoom level 11, column 3, row 2. As you explore the image, your browser makes dozens, even hundreds of individual tile requests like this. It feels fast because each individual tile is tiny and downloads in no time.<br />
<br />
For more about tiled zoomable images, these blog posts are an excellent introduction:<br />
<br />
<ul>
<li><a href="http://www.gasi.ch/blog/inside-deep-zoom-1/">http://www.gasi.ch/blog/inside-deep-zoom-1/</a></li>
<li><a href="http://www.gasi.ch/blog/inside-deep-zoom-2/">http://www.gasi.ch/blog/inside-deep-zoom-2/</a> </li>
</ul>
So how do we get from a single JPEG2000 image to hundreds (or even thousands) of JPG tiles? It’s possible to prepare your image tiles in advance, so that you process the source image once and store folders of prepared tiles on your web server. For small collections of images this is a simple way to go and doesn’t require anything special on the server. But for the Library, it’s not practical – we don’t want to prepare tiles as part of the digitisation workflow. They are not “archival”, and they take up a lot of extra storage space. We need something that can generate tiles on the fly from the source image, given the tile requests coming from the browser.<br />
<br />
For this we need an <a href="http://en.wikipedia.org/wiki/Image_server" target="_blank">Image Server</a>, and we’ve chosen <a href="http://iipimage.sourceforge.net/" target="_blank">IIPImage </a>for its performance and native Seadragon (Deep Zoom) support. The Image Server generates browser-friendly JPEG images from regions of the source image at particular zoom levels. When your browser makes a request to the image server for a particular tile, the image server extracts the required region from the source JPEG 2000 file and serves it up to you an ordinary JPEG.<br />
<br />
<h3>
Viewer or Player? Or Reader? </h3>
The next piece of the puzzle is the browser application that makes the requests to the server. A book or archive is a sequence of images along with a lot of other metadata. And it’s not just books – the Library also has video and audio content. All of these are described in detail by METS files produced during the digitisation/ingest workflow. In the world of tile-based imaging, the term “viewer” is often used to describe the browser component of the system, but we seem to have fallen naturally to using the term “Player” to describe it – it plays books, videos and audio, so “Player” it is. Our player needs to be given quite a lot of data to know what to play.<br />
<br />
We could just expose the METS file directly, but it is large and complex and much of it is not required in the Player. So we’re developing an intermediate data format, which effectively acts as the public API of the Library. Given a Library catalogue number, the player requests a chunk of data from the server; this tells it everything it needs to know to play the work, in a much simpler format than the METS file. In the future other systems could make use of this API (at the moment it’s exposed as JSON).<br />
<br />
<h3>
The user experience </h3>
The user won’t just be viewing a sequence of images, like a slide show. It should be a pleasant experience to read a book from cover to cover. Many users will be using a tablet, reading pages in portrait aspect ratio. We aim to make this a good e-reading experience too, augmented by search and navigation tools.<br />
<br />
The user experience might start with a search result from the Library’s main search tool. For books that have been digitised, the results page will provide an additional link directly to the player “playing” the digitised book. The URL of the book is an important part of the user experience, and we want to keep it simple. In future, library.wellcome.ac.uk/player/b123456 would be the URL of the work with catalogue refrence number b123456; that URL would take you straight to the player.<br />
<br />
We want to be able to link directly to a particular page of a particular book, just as a printed citation could. This deeper URL would be /player/b123456#/35. But we can do better than that; our URL structure should extend to describe the precise region of a page, so that one reader could line up a particular section of text on a page, or a picture, and send the URL to another reader; the second reader would see the work open at the same page, and zoomed in on the same detail.<br />
<br />
<h3>
Access Control </h3>
Much of the material being made available is still subject to copyright. Those works that are cleared for online publication by the Trust’s<a href="http://wellcomedigitallibrary.blogspot.co.uk/2012/04/clearing-copyright-for-books.html" target="_blank"> copyright clearance strategy</a> still need some degree of access control applied to them; typically the user will be required to register before viewing them. This represents a significant architectural challenge, because we need to enforce access restrictions down to the level of individual tile requests. We don’t want anyone “scraping” protected content by making requests for the tiles directly, bypassing the player.<br />
<br />
<h3>
Performance and Scale </h3>
As well as the technical challenges involved in building the Player, we need to ensure that content is served to the player quickly. Ultimately the system will need to scale to serve millions of different book pages. Between the player and the back end files is a significant middle tier: the Digital Delivery System, of which the Player is a client. This layer is the Library’s API for Digital Delivery. The browser-based player interacts with it to retrieve data to display a book, highlight search results, generate navigation and so on. The Image Server is a key component of this system.<br />
<br />
<i>This post was written by Tom Crane, Lead Developer at Digirati, working with his colleagues on developing digital library solutions for the Wellcome Digital Library.</i>Christy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.com1tag:blogger.com,1999:blog-6380997420931943041.post-58302400544168764242012-04-30T10:46:00.001+01:002012-04-30T13:33:11.326+01:00Will more data lead to different histories being told?New technology is making information more widely available and, when it launches later this year, the WDL will make it easier to access historical evidence about the foundations of modern genetics. Will this democratize our understanding of the history of genetics and lead to different versions of the history being told?<br />
<br />
There is an African proverb which says that history is written by the hunter not the lion. History inevitably simplifies the past and the selection process can be subjective. When it launches the WDL will start to put 21 archive collections and around 2,000 books on-line. The project is to digitise as much as we can rather than cherry pick the highlights. This means that the building blocks used by historians to piece together the past will be made freely available to a wider audience. A lot of this material may seem like mundane workaday stuff. Users will have to wade through a lot of material to reach the bits they are interested in but this is probably a more accurate reflection of the scientific research process.<br />
<br />
Flashes of genius are essential but they do not happen in isolation. Thomas Edison’s phrase about invention being 1% inspiration and 99% perspiration applies to scientific research too. The discovery process needs both.<br />
<br />
Watson and Crick were extremely clever to work out the helical structure of DNA but they did not get there simply because they were lone geniuses. Before they made their discovery a lot of people had spent years experimenting, writing and thinking about DNA. There had even been flashes of insight which ended up being wrong.I recently read a letter from Gerald Oster sent to Aaron Klug after Rosalind Franklin’s death, in which he recalled his time working in London. He reflects that even though he had much of the relevant information by early 1950 he lacked the insight to work out the structure of DNA. This letter (FRKN/06/07/001-2) is held by the <a href="http://www.chu.cam.ac.uk/archives/" target="_blank">Churchill Archives Centre</a> in Cambridge and a digitised version will become part of the WDL.<br />
<br />
I am rather hoping that the WDL might help us to recognise that while flashes of inspiration are part of scientific discovery they are only possible because a team of other people paved the way.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6380997420931943041.post-40476671180074456942012-04-19T09:57:00.000+01:002012-04-19T09:57:43.530+01:00Clearing copyright for books: preliminary ARROW resultsAs part of the genetics books project, we are tackling issues of copyright clearance and due diligence head on. Up to 90% of this collection is in copyright, or is likely to be in copyright, so developing a copyright clearance strategy was one of our earliest considerations. This turned into a useful project to test-run the EC-funded <a href="http://www.arrow-net.eu/" target="_blank">ARROW </a>system on a large scale. ARROW provides a workflow for libraries and other content repositories to determine whether books are in-commerce, in copyright, and whether the copyright holders can be identified and traced. This system has undergone small tests throughout Europe, including the UK (using collections and metadata from the British Library), but in order to determine whether ARROW is feasible on a large scale, a realistic large-scale project was needed.<br />
<br />
The Wellcome's genetics books project provided this opportunity, and the challenge was taken up by the ALCS and the PLS jointly, as announced previously on our <a href="http://wellcomelibrary.blogspot.co.uk/2011/12/alcs-and-pls-to-provide-rights.html" target="_blank">Library Blog</a>. Results from ARROW, combined with the responses from contacted rights holders, determine whether the Wellcome Library will publish a work online.<br />
<br />
The collection of (roughly) 1,700 potentially in-copyright books is not enormous, but it is diverse, and has already thrown up some interesting wrinkles in the copyright clearance workflow.<br />
<br />
For example, according to the AARC2 standard used to catalogue these books, only up to three authors are included in the metadata record (followed by <i>et al</i>). Works with more than three authors, and collected works such as conference proceedings, had to be manually consulted in order to identify all the named contributors. This inflated the known number of contributors to nearly 7,000 (4 authors on average per book).<br />
<br />
Embedded below is a presentation I gave at the London Book Fair earlier this week, which provides an overview of the process, and preliminary statistics from the first 500 books to complete the ARROW workflow.<br />
<br />
<div id="__ss_12599390" style="width: 425px;">
<span style="display: block; margin: 12px 0pt 4px;"></span><b style="display: block; margin: 12px 0pt 4px;"> </b><b style="display: block; margin: 12px 0pt 4px;"><a href="http://www.slideshare.net/Wellcome/lbf-2012-v2" target="_blank" title="Copyright Clearance for Genetics Books, A pilot project at the Wellcome Library">Copyright Clearance for Genetics Books, A pilot project at the Wellcome Library</a></b> <iframe frameborder="0" height="355" marginheight="0" marginwidth="0" scrolling="no" src="http://www.slideshare.net/slideshow/embed_code/12599390" width="425"></iframe> <br />
<div style="padding: 5px 0 12px;">
View more <a href="http://www.slideshare.net/thecroaker/death-by-powerpoint" target="_blank">PowerPoint</a> from <a href="http://www.slideshare.net/Wellcome" target="_blank">Wellcome</a> </div>
</div>Christy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.com1tag:blogger.com,1999:blog-6380997420931943041.post-74724739841946555372012-04-02T12:25:00.002+01:002012-04-02T12:26:23.729+01:00Learning lessons on the Genetics Books digitisation project<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-9RIU7SkEofW8K3AVz7WOIGvymD3kGcK_RtmN0IS_6iZH92gadHsMXjHFNApVgKMNgAB4zScsJyg6MQC6zYn8XtYL3tAnhlX6ms0Er08wO3SSquLvTqt8xHdZp8swLOrhOskgC93T9X_E/s1600/Books.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-9RIU7SkEofW8K3AVz7WOIGvymD3kGcK_RtmN0IS_6iZH92gadHsMXjHFNApVgKMNgAB4zScsJyg6MQC6zYn8XtYL3tAnhlX6ms0Er08wO3SSquLvTqt8xHdZp8swLOrhOskgC93T9X_E/s320/Books.jpg" width="211" /></a></div>
A key component of the theme of our digitisation pilot programme - "Foundations of Modern Genetics" - is a set of printed textbooks and secondary sources published between 1850 and 1990 that shed light on the development of genetic and genomic research. The total collection identified is around 2,000 books. The goal is to digitise these texts in full, and make them freely available online via the Wellcome Digital Library (we are of course dealing with <a href="http://wellcomelibrary.blogspot.co.uk/2011/12/alcs-and-pls-to-provide-rights.html" target="_blank">copyright clearance</a>).<br />
<br />
Digitisation of books often looks and sounds straightforward. It is not always straightforward of course - but the new book scanners on the market these days do make it quick. There are standard ways of book scanning - you put the book on a cradle, and either turn the pages (by hand), or use a "robotic" contraption that turns the pages automatically. You can use scanning technology, or one-shot dSLR cameras; panes of glass to hold the pages down, or small grips on the outer margins of the pages. The choice depends on the physical nature of the books and how quickly you want to digitise. Even when outsourcing it is useful to understand how book scanning really works. Our Genetics Books digitisation project - a pilot project - is giving us this opportunity.<br />
<br />
We commissioned local digitisation company <a href="http://www.bespokearchivedigitisation.com/" target="_blank">Bespoke Archive Digitisation</a> to carry out the digitisation work for this pilot project. As the digitisation is carried out on site, we have been involved to some extent in all aspects of the digitisation, including the setup and use of new types of equipment, the QA process involved in book digitisation, and the workflow of image conversion and delivery. As we have never carried out high-throughput book digitisation at the Wellcome Library before, this has been a huge learning curve for us, allowing us to gain knowledge that will come in very useful in the future with new (and hopefully larger) projects.<br />
<br />
<br />
Bespoke Archive Digitisation uses a robotic book scanner and a manual book scanning unit (for books that are not robust enough for the robotic scanner, are outsized, etc.). Both of these "scanners" use Canon 5D Mark II cameras, two per unit to capture each page of an opening simultaneously. The robotic book scanner is the latest version from Kirtas, the <a href="http://www.kirtas.com/kabisIII.php" target="_blank">Kabis III</a>. Richard Keenan, owner of Bespoke Archive Digitisation explains, "this unit has a number of time-saving features such as "fluffers," a “snubber,” and a self adjusting book cradle which moves to keep the book at the correct angle to be photographed. This is accomplished through various sensors and lasers, which monitor the book throughout imaging to keep it in the correct position, but must also be monitored by the operator."<br />
<br />
A key lesson, according to Richard, is that "although all robotic book scanners include a published throughput (2,890 pages per hour for this particular unit), it is important to understand that the published throughputs do NOT mean that you can do 2,890 pages per hour, hour after hour without stopping. Each book must be set up on the cradle, the cameras may need some adjustment/focusing, and page turning does require manual intervention, every time, to ensure the pages are flat, and to prevent page curvature and glare (especially on sealed paper).<br />
<br />
"Also, it is very important to remember that this is just the image capture stage, the pages then have to be batch processed, edited and rigorously quality assessed which can take the same, or more time than imaging.
Depending on the book's structure - page thickness, binding type, size of the book etc - you will find that speeds vary considerably, a realistic estimate of throughput over a significant period of time is approximately 1000 pages per hour, but this can be much lower with some books.<br />
<br />
"Although these figures differ by a large margin from those published, the Kabis III from Kirtas is still probably the fastest way to digitize books, and the important thing is that the quality of output produced is excellent if operated correctly.
The on board editing software 'Book Scan Editor' is very handy, offering the usual cropping, image adjustment and sharpening options, but also deskewing and xml conversion and even OCR.
I would say that another thing to bear in mind here, is that there is a large learning curve with this technology, so for anyone thinking of using one - particularly those who have no experience with robotic book scanners - plan plenty of time in the project for training and testing periods."Christy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.com0tag:blogger.com,1999:blog-6380997420931943041.post-13337429247462588562012-03-23T16:42:00.001+00:002012-03-23T10:37:12.414+00:00Three Geneticists from the University of GlasgowThe Wellcome Digital Library isn’t just about collections held in the Wellcome Library. We are working with a number of other organisations that hold material on the history of modern genetics. One of the contributing partners, the University of Glasgow Archives Service, have just started to digitise the archives of three men who worked at the University’s Department of Genetics - Guido Pontecorvo (1907-1999), James (Jim) Harrison Renwick (1926-1994) and Malcolm Ferguson-Smith (1931 – ).<br /><br />You can see photos of their brand new digitising suite <a href="http://universityofglasgowlibrary.wordpress.com/2012/03/02/puds-blog-hello/" target="_blank">here</a>.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6380997420931943041.post-62576216582862378852012-03-12T12:36:00.000+00:002012-03-12T14:54:56.229+00:00Wellcome Digital Library updateThe Wellcome Digital Library pilot has been underway for 18 months with 6 months to go before we launch the new Library website. This will provide access to a wide range of digital content related to the <a href="http://wellcomelibrary.blogspot.com/2010/08/wellcome-library-launches-major.html" target="_blank">Foundations of Genetics theme</a>. All of the work done so far has been behind the scenes: digitising content, procuring and developing our digital library systems, and designing a new website. We are looking forward to displaying the product of all this work to the public - but we're not quite there yet!<br />
<br />
So where are we now and what will we be doing in 2012? Here is a snapshot of progress so far. Further details on some of these projects can be found on this blog, and we will continue to explain our activities in more detail in future posts.<br />
<br />
<span style="font-size: large;"><b>Digitisation</b></span><br />
<ul>
<li><b>Archives</b>: With our in-house team of two photographers, we have digitised around 380,000 pages from the collections of Crick, Mourant, Medawar, Sanger, Wyatt, Grueneberg, and the Blood Group Unit. We have just started the Eugenics Society collection, which will carry on throughout the spring and summer.</li>
<li><b>Genetics Books</b>: This project has just begun, with up to 2,000 books to be digitised this spring by an external supplier, Bespoke Archive Digitisation, working on-site. </li>
<li><b>MOH reports</b>: A successful <a href="http://wellcomelibrary.blogspot.com/2011/12/medical-officer-of-health-reports-to-be.html" target="_blank">JISC funding bid</a> meant we could add the Greater London Medical Officer of Health reports to the pilot project. Conservation is underway, and digitisation will begin in a few month's time.</li>
<li><b>ProQuest</b>: We have partnered with ProQuest to digitise our pre-1700 printed books for Early European Books online, with over 1,000 books now digitised and around 13,000 to go. Those with subscriptions and anyone in the UK can view our first 400 books on the <a href="http://eeb.chadwyck.co.uk/home.do" target="_blank">EEB website</a> with more to come shortly (search for "Wellcome").</li>
<li><b>External content</b>: We have had the first delivery from one of our external partners, <a href="http://library.cshl.edu/" target="_blank">Cold Spring Harbour Laboratory</a>, including correspondence from the James Watson archive. This adds around 50,000 images to our digital archive collections, with more to come throughout 2012 and early 2013 from all partners.</li>
<li><b>Copyright and sensitivity:</b> Hand in hand with digitisation, we are assessing our content for sensitivity and copyright issues where necessary. Sensitive items (containing certain types of private information as defined by the Data Protection Act) are identified and flagged as unsuitable for online dissemination. <a href="http://wellcomelibrary.blogspot.com/2011/12/alcs-and-pls-to-provide-rights.html" target="_blank">Copryight clearance</a> of in-copyright works is underway with the help of the Authors' Licensing and Collecting Society, and the Publishers' Licensing Society. </li>
</ul>
<br />
<span style="font-size: large;"><b>Systems development</b></span><br />
<ul>
<li><b> Digital Asset Management</b> <b>& Storage</b>: <a href="http://www.digital-preservation.com/solution/safety-deposit-box/" target="_blank">Safety Deposit Box 4.1</a>, our digital asset management system, was extended to provide extra functionality for large sets of digital assets in 2011. This system is now in production. Our storage system, Pillar, now includes a Write Once Read Many (WORM) backup drive to ensure that our files are secure in the long term.</li>
<li><b>Workflow system</b>: We procured <a href="http://www.digiverso.com/products/goobi" target="_blank">Goobi </a>(Intranda Version) with bespoke modifications in 2011 to act as a workflow system, enabling us to track project progress, and to automate a number of activities (including ingest of content into Safety Deposit Box). This has recently been put into use in production, particularly for the Genetics books digitisation project. Soon we will be using Goobi for all digitisation projects, and to ingest our backlog of images.</li>
<li><b>JPEG 2000</b>: We now archive all our images in the JPEG 2000 (Part 1) format, and have an automated batching process set up with LuraWave. Soon, we will be implementing JPEG 2000 validation as part of this process to ensure all JPEG 2000s meet the correct standards before ingest.</li>
<li><b>Digital delivery</b>: A new digital delivery system is currently under development that will interoperate with Safety Deposit Box and our new website content management system, <a href="http://www.innovationdigital.co.uk/solutions/alterian-cm7.aspx" target="_blank">Alterian CM7</a>. We have commissioned CM7 developers <a href="http://www.digirati.co.uk/" target="_blank">Digirati </a>to carry out this development, which will be completed at the end of the summer. So far they have produced a proof of concept system that demonstrates an end-to-end sequence from retrieval of images from Safety Deposit Box using METS files created by Goobi, to displaying images online. They are adapting <a href="http://en.wikipedia.org/wiki/Seadragon_Software" target="_blank">Seadragon</a>, the MS viewer used by several other digital libraries, to meet our specific needs and design criteria.</li>
<li><b>Search and discovery</b>: <!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-GB</w:LidThemeOther>
<w:LidThemeAsian>X-NONE</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
</w:Compatibility>
<w:BrowserLevel>MicrosoftInternetExplorer4</w:BrowserLevel>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="true"
DefSemiHidden="true" DefQFormat="false" DefPriority="99"
LatentStyleCount="267">
<w:LsdException Locked="false" Priority="0" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" Priority="39" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" Name="toc 9"/>
<w:LsdException Locked="false" Priority="35" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" Priority="10" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" Priority="1" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" Priority="11" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" Priority="22" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" Priority="59" SemiHidden="false"
UnhideWhenUsed="false" Name="Table Grid"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" QFormat="true" Name="TOC Heading"/>
</w:LatentStyles>
</xml><![endif]--><!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin:0cm;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:10.0pt;
font-family:"Times New Roman","serif";}
</style>
<![endif]-->We are also making changes to our single search system, Encore. This work is
looking at providing better representation of archival metadata in Encore, and
also options for incorporating a full-text index. The purpose is to provide
access to all Library content - the catalogues as well as the digitised
materials - via a single interface.
</li>
</ul>
<br />
<span style="font-size: large;"><b>New website and user experience </b></span><br />
<ul>
<li><b>User experience-led design: </b>Last year the Library brought on board external suppliers <a href="http://clearleft.com/" target="_blank">Clearleft </a>-
user experience and web design experts - to help redesign the
information architecture and visual appearance of the new website. New
designs are already visible on the internal web development environment,
so further user testing of a real website can soon be done.</li>
<li><b>Transferring content:</b> The Library has carried out a full content audit of the current website, and prioritised content to carry across to the new site. The current site contains over 2,000 pages; this will be considerably reduced. The content carried across to the new site will be thoroughly edited to ensure it is up-to-date and consistent with the new site "style".</li>
<li><b>Creating new content</b>: New content will also be created once the content management system is in action, with a focus on the Foundation of Modern Genetics. This is a major part of the Library's aim to provide interpretative content to both researchers and the "curious public".</li>
</ul>
<ul>
</ul>Christy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.com1tag:blogger.com,1999:blog-6380997420931943041.post-34427225328572006072012-03-05T10:09:00.023+00:002012-03-12T10:08:19.542+00:00Filling the MOH Gaps<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCY6RpVpFdAU7dewu6Ezi4krG7Sj-I4NZzwcmm4XdKd2VMBtFWAOtsUZExhPaIDeAxw4QPE8on862iKKmKSWy7Sv6M96xCGapJXdmuZKKAYs1jKNy3uVxJ0WdpN1euzO9_AkBBSHvE0NWv/s1600/MOH.jpg"><img id="BLOGGER_PHOTO_ID_5717124847260884386" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; WIDTH: 217px; CURSOR: hand; HEIGHT: 151px" alt="" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCY6RpVpFdAU7dewu6Ezi4krG7Sj-I4NZzwcmm4XdKd2VMBtFWAOtsUZExhPaIDeAxw4QPE8on862iKKmKSWy7Sv6M96xCGapJXdmuZKKAYs1jKNy3uVxJ0WdpN1euzO9_AkBBSHvE0NWv/s200/MOH.jpg" border="0" /></a> The Wellcome Library has a great collection of Medical Officer of Health (MOH) Reports. These reports are stuffed full of grim and useful information from the 19th and 20th century, such as statistics on infant mortality. JISC is, very wisely, funding the digitisation of the London reports. There are some gaps in the Wellcome Library’s collection so, in order to make a really useful digital resource, we have been working out what is missing. This has not been straightforward.<br /><br />First we needed to check what we held and then we needed to make sure that our gaps really were gaps. We didn’t want to waste time looking for reports that were never created. The very first MOH report in Britain was produced in Liverpool in 1847. The first London report was produced in the following year but the early reports do not cover the whole of London. The Public Health Act of 1848 permitted local authorities to employ MOHs but, since it was not obligatory, only a minority did. The Metropolis Management Act of 1855 required MOHs to be appointed in central London but the big change came with the 1875 Public Health Act. From then until 1972 the production of MOH reports was pretty solid.<br /><br />Another challenge was getting to grips with the boundary changes. Over the years the administrative boundaries of London have altered several times. The current 32 London Borough boundaries date from 1965 when <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7xG4pSd2aXjVTOFwl6qZe-uyjoe6fXpBNBIMVUB1H1a9LnXCRbB5uyxvLq-7dePptE18HeHLt7PNa-8np8qkBsUEqQcTCOeg9wDNQRbTyS7jjU_1aPECk33_4jwuKso3Subi0VicMNBsU/s1600/MOH+gaps+2.jpg"><img id="BLOGGER_PHOTO_ID_5717138156174597810" style="FLOAT: right; MARGIN: 0px 0px 10px 10px; WIDTH: 349px; CURSOR: hand; HEIGHT: 220px" alt="" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7xG4pSd2aXjVTOFwl6qZe-uyjoe6fXpBNBIMVUB1H1a9LnXCRbB5uyxvLq-7dePptE18HeHLt7PNa-8np8qkBsUEqQcTCOeg9wDNQRbTyS7jjU_1aPECk33_4jwuKso3Subi0VicMNBsU/s320/MOH+gaps+2.jpg" border="0" /></a>Greater London was established. Before that there were 28 metropolitan boroughs plus various boroughs, urban and rural district councils in what is now, outer London. Before 1899 much of what we now think of as London was part of Kent, Middlesex, Essex or Surrey. The City of London has long gone its own distinctive way and the tangle of parish boundaries there is particularly confusing. Old maps and a book on administrative units by Frederick A. Youngs helped us to make sense of all these changes.<br /><br />We have decided to start from the centre and try to create a complete as record as possible for the 12 inner London Boroughs. We’ve got to the stage where we have a list of reports that we want to find. The next step is to track them down in other collections and ask if we can get them digitised.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6380997420931943041.post-56194758633934145892012-03-05T10:03:00.006+00:002012-03-06T10:17:41.754+00:00Watson and Crick LettersI've just had the privilege of reading a fantastic series of letters written in 1954 by James Watson and Francis Crick. They were written a year after they published their seminal article on the structure of DNA. In the letters the two men are exchanging ideas and their excitement shines through. They write about all sorts of things, for example, the importance of building space filling three dimensional models, confusion over how thymine fits into the helical structure and what the researchers at KCL are up to. In March 1954 Watson also expresses his frustration with the research process, “<em>The whole thing is puzzling and paradoxical (for could DNA be wrong) and is slowly driving me to despair and to loath nucleic acids</em>.” (PP/CRI/D/2/45)<br /><br />I got to read them because last month the first batch of digitised material arrived from <a href="http://library.cshl.edu/">Cold Spring Harbor Laboratory </a>in New York, one of the five external organisations contributing digitized material to the WDL pilot project. The James Watson archive is held at Cold Spring Harbor and contains the letters written to him by Francis Crick. The letters Watson wrote to Crick are held by the Wellcome Library.<br /><br />Later this year, when the WDL is launched, Watson and Crick’s correspondence will be digitally united. Lots of people will be able to read these letters (and lots of other stuff) online while the originals stay safely tucked away in their archival homes. I am excited about that!Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6380997420931943041.post-91220675678161482982012-02-21T16:09:00.000+00:002012-02-21T16:56:35.297+00:00The Medical Officer of Health reports project begins.<div>
<div>
<div>
<span style="font-family: Arial,Helvetica,sans-serif;">I recently started working on the </span><a href="http://wellcomelibrary.blogspot.com/2011/12/medical-officer-of-health-reports-to-be.html"><span style="font-family: Arial,Helvetica,sans-serif;">Medical Officer of Health digitisation project</span></a><span style="font-family: Arial,Helvetica,sans-serif;">.</span></div>
<div>
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">
</span><span style="font-family: Arial,Helvetica,sans-serif;">
</span><span style="font-family: Arial,Helvetica,sans-serif;">
</span><br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVEv-0EohVyGG0FrUzpnQoiXv2JhaJXQnZmkpi6QVHbiWoApgikAdtVC-qFw-6j9tGJP8dhXbQXZnUaaSR75y8z01RO-fEyraG6DT3v5ycVlwkabUyRqyPTnyzNKqePeW_qaAifdMIM0I/s1600/DSCN1771.JPG" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><span style="font-family: Arial,Helvetica,sans-serif;"><img border="0" height="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVEv-0EohVyGG0FrUzpnQoiXv2JhaJXQnZmkpi6QVHbiWoApgikAdtVC-qFw-6j9tGJP8dhXbQXZnUaaSR75y8z01RO-fEyraG6DT3v5ycVlwkabUyRqyPTnyzNKqePeW_qaAifdMIM0I/s200/DSCN1771.JPG" width="200" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial,Helvetica,sans-serif; font-size: x-small;">Spine with methylcellulose applied</span></td></tr>
</tbody></table>
<span style="font-family: Arial,Helvetica,sans-serif;">Since the beginning of December 2011, I have been spending the majority of my time in the conservation studio. I have been carrying out disbinding, cleaning and rehousing of the late 19th century Medical Officer of Health (MOH) reports that are bound by year. This is so that the digitisers can scan or photograph them for the project.</span><br />
<br /></div>
<div>
</div>
<div>
</div>
<div>
<span style="font-family: Arial,Helvetica,sans-serif;">The MOH reports in our collection are shelved in different sequences: Main, London and Provincial. However, all three sequences contain reports for London areas. Main and Provincial sequences are bound by geographical area and generally they're in a good condition but the London sequence reports are bound by year and tend to be in a poorer state due to their heavy use. There are about 80 bound volumes in the London sequence and they are all being disbound. Eventually we will be able to house all of the London reports together by geographical area.</span><br />
<br /></div>
<div>
<span style="font-family: Arial,Helvetica,sans-serif;">
</span><span style="font-family: Arial,Helvetica,sans-serif;">
</span><span style="font-family: Arial,Helvetica,sans-serif;">
</span><br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTdrRiEhvTEJfkDSpvM7Q4vQV1yM-UbnQOnJZ8DiL1o7jhQAylY9vZJpFLE-N7GPeFRF1U2A7m6Na4M-uOUQmSJgIZ6GSj1dKlIgv9tHf7TTO1GKOu-KljFBPArutLJTeHJ-HrdFYh2H0/s1600/disbinding+test+08+11+%25284%2529.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><span style="font-family: Arial,Helvetica,sans-serif;"><img alt="" border="0" height="150" id="BLOGGER_PHOTO_ID_5711348260932807634" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTdrRiEhvTEJfkDSpvM7Q4vQV1yM-UbnQOnJZ8DiL1o7jhQAylY9vZJpFLE-N7GPeFRF1U2A7m6Na4M-uOUQmSJgIZ6GSj1dKlIgv9tHf7TTO1GKOu-KljFBPArutLJTeHJ-HrdFYh2H0/s200/disbinding+test+08+11+%25284%2529.jpg" style="margin-top: 0px;" width="200" /></span></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial,Helvetica,sans-serif; font-size: x-small;">Lining coming off</span></td></tr>
</tbody></table>
<span style="font-family: Arial,Helvetica,sans-serif;"></span><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTdrRiEhvTEJfkDSpvM7Q4vQV1yM-UbnQOnJZ8DiL1o7jhQAylY9vZJpFLE-N7GPeFRF1U2A7m6Na4M-uOUQmSJgIZ6GSj1dKlIgv9tHf7TTO1GKOu-KljFBPArutLJTeHJ-HrdFYh2H0/s1600/disbinding+test+08+11+%25284%2529.jpg" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><span style="font-family: Arial,Helvetica,sans-serif;"></span></a><span style="font-family: Arial,Helvetica,sans-serif;">In order to take the bound reports apart, I first need to remove the cloth case and spine linings whilst keeping the pamphlets intact. I use a 4% methylcellulose solution to break down the binding animal glue. A major challenge with this is that each volume’s binding breaks down at a different rate so it requires constant checking to avoid damaging the paper. On average it takes a couple of hours just to remove the spine linings. We recently purchased a <i>pink</i> portable clothes steamer to try to speed up the process. I haven’t tried this yet but we are hopeful that this will work faster.</span></div>
<div>
</div>
<div>
</div>
<div>
<span style="font-family: Arial,Helvetica,sans-serif;">
</span><span style="font-family: Arial,Helvetica,sans-serif;">
</span><span style="font-family: Arial,Helvetica,sans-serif;">
</span><br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; text-align: right;"><tbody>
<tr><td style="text-align: center;"><span style="font-family: Arial,Helvetica,sans-serif;"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5711348262773734258" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6GRwksQjwhC4NnEtZPYNaCu-1pGHz2GGM3MvI6hfhftHKINLLhdLAM9o57_gcEs2P-cUkrvUS-K301i8zBRTlME-uabpBPGlJucIIOk29prgodIjt592k2UGADtJCy3K98O1ZRmBu_Xo/s200/disbinding+test+08+11+%25286%2529.jpg" style="height: 150px; margin: 0px auto 10px; width: 200px;" /></span></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial,Helvetica,sans-serif; font-size: x-small;">The aftermath of removal</span></td></tr>
</tbody></table>
</div>
<div>
</div>
<span style="font-family: Arial,Helvetica,sans-serif;">Another important part of the project is creating <span style="line-height: 115%;">separate bibliographic records for each report</span>. We have decided to catalogue these as monographs in order to improve searching and allow users to find reports by fields such as geographic area, Medical Officer's name and date of the report. </span><br />
<br /></div>
<div>
<div>
</div>
<div>
<span style="font-family: Arial,Helvetica,sans-serif;"></span></div>
<div>
<span style="font-family: Arial,Helvetica,sans-serif;">Whilst I am going through the reports, I have been finding some very interesting snippets. I will include some as I continue to blog so look out for them!</span></div>
</div>
</div>Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-6380997420931943041.post-47853715054033789762011-11-28T09:34:00.001+00:002011-12-07T17:41:21.349+00:00Guest blog post: Digitising early printed books at the Wellcome Library<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwBnHHbi50Dcr4VUPN0we5Xvsxt-s3BPzu2P_pZdK0sZ21DYd8A3EOv2pJB4KMAeldFY77V_102QzH0kjaruwLQhpj6eIvjXJiSecvDJXjA4l4Oaw14njMTYj9_JVrW1LdhDy0MgcUGxFC/s1600/imageserv.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="277" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwBnHHbi50Dcr4VUPN0we5Xvsxt-s3BPzu2P_pZdK0sZ21DYd8A3EOv2pJB4KMAeldFY77V_102QzH0kjaruwLQhpj6eIvjXJiSecvDJXjA4l4Oaw14njMTYj9_JVrW1LdhDy0MgcUGxFC/s320/imageserv.jpg" width="320" /></a></div>
<i>Karine Larose and Jeremy Uhl are working on the Early European Books <a href="http://wellcomelibrary.blogspot.com/2011/07/digitising-our-early-european-printed.html">digitisation project</a> at the Wellcome Library. They are empolyees of Diadeis, a digitisation company based in France, and work on site alongside the rest of the Wellcome's digitisation team. In this guest post, Karine and Jeremy explain what they do here at the Wellcome Library: </i><br />
<br />
<br />
<br />
<br />
Hello, Karine and Jeremy here from <a href="http://www.numen.fr/numen-activites-offres-num-diadeis-patrimoniale-uk.aspx">Diadeis</a>. We are the production team currently working on the digitization project for ProQuest at the Wellcome library. The following will be a sort of 'day in the life' of our department. We hope you enjoy it and would love any comments, questions, or suggestions.<br />
<br />
Each day, when we start, new books are delivered to us by Matt Brack, the Digitisation Support Officer, for digitisation. We then register them in the database (also known as the check-in process). First we assess the books to determine if they are suitable for scanning. We might need to refuse a book if<br />
<br />
<ul>
<li>the book is too big or thick for the scanning machines to handle (usually higher than 42 cm and wider than 30 cm) </li>
<li>the book cannot open more than 100 degree (later in the project these books will be scanned using different equipment) </li>
<li>The book is too fragile and we feel that we cannot scan it without causing more damage. </li>
</ul>
Also during the check-in process we might notice other minor problems with the books but not so severe as to refuse them. In that case we usually put them on hold. An example would be if most of the pages are stuck/glued together. ‘Problem books’ on hold are placed on a separate shelf, and we later consult with Matt as to what action has to be taken. Simultaneously, the books we scanned on the previous days are given back to Matt and we update the database to show that they have been returned to the library.<br />
<br />
Before scanning the pages of the books, the spine and edges are photographed first. This step is to show the condition of the book before we start the scanning. We do our best during the scanning process to be careful with the books, to return them in the state they were delivered. The books are scanned using two main scanners, one for restricted angles and one for opened books. A few problems we sometimes have when scanning books are:<br />
<br />
<ul>
<li>Huge fold-outs that cannot be scanned with the current equipment </li>
<li>missing pages </li>
<li>pieces of a book coming apart (usually the spine) </li>
</ul>
<br />
Our most common problem is dust! This includes tiny book pieces, occasionally hair, once we even found tiny bits of cereals in a book. There are lots of surprises and we never know what we may find in between pages of a book. Sadly we have yet to find any money in any of the books ;). Several times during the day we have to clean the glass panel; for that we use a special cloth and a vacuum cleaner. The archive office kindly supplied us with alcohol wipes which has a disinfection effect. Also because some of the books have leather covers, our hands can become oily after handling them and we have to wash our hands regularly.<br />
<br />
Once the scanning day is over, we pack all the images in a packaging software and send them off to the indexing team for quality control, cropping and indexing.<br />
<br />
Working on this project at the Wellcome library has been fun and exciting because we get to see rare books every day and provide digital access to them to people all over the world. We are pleased to be given the chance to explain our daily routine to the followers of this blog and look forward to any comments they might have.<br />
<br />
We would like to thank Matt for being so flexible with our timetable and bringing us lots of books everyday and Christy for her support and giving us the opportunity to write for the blog. For those of you who work for the Wellcome library do come along and say "hello" to us, the room may be dark but I assure you we are here!<br />
<br />
<i>Authors: Karine Larose and Jeremy Uhl </i><br />
<br />
<i>NB: The first batch of 400 books can now be seen on the <a href="http://eeb.chadwyck.com/home.do">EEB website</a> (do a Quick Search for Wellcome). These are freely available to Wellcome Library members and everyone in the UK. </i>Christy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.com2tag:blogger.com,1999:blog-6380997420931943041.post-83984475278581751972011-09-21T09:14:00.002+01:002012-03-05T18:59:13.422+00:00Preserving our digital assets #2 - WORM storageThis post follows on from my <a href="http://wellcomedigitallibrary.blogspot.com/2011/07/preserving-our-digital-assets-1-sdb4.html">earlier description</a> of our DAM system (Safety Deposit Box or SDB), which manages long-term preservation and access to our digital assets. Here we turn to the back-end of the back-end: storage.<br />
<br />
There are a number of requirements that must be met to safely storing digital assets. The storage solution must be:<br />
<ul>
<li>Secure (behind a firewall)</li>
<li>Robust (able to manage points of failure in the disks)</li>
<li>Replicable (multiple copies on multiple sites)</li>
<li>Scalable (able to handle tens of millions of files)</li>
<li>Quick to access (the archived files are also used for delivery) </li>
</ul>
After considering different systems and suppliers – including robotic tape back-up – we settled on a solution that gives us the confidence we need for long-term preservation, and fits well with the Trust’s existing storage infrastructure. Our existing storage suppliers, <a href="http://www.pillardata.co.uk/">Pillar Data Systems</a>, have extended the existing RAID5 enterprise storage system used for all the Wellcome Trust’s business needs by incorporating a “Write Once, Read Many” (WORM) back-up storage server for use by the Wellcome Digital Library. Associated management software copies files from the main storage server to the WORM, and monitors the main server for file errors that can be “healed” using the WORM copy.<br />
<br />
To explain this in a bit more detail, related to our primary requirements:<br />
<br />
<b>Security</b> means that only authorised users or systems are able to access the files, and that unauthorised deletions or changes are guarded against. Locking master files behind a firewall is the main form of defence from unauthorised external access (i.e. hackers, or the ability to download files by "guessing" the network path and filenames). However, we still faced the prospect of file deletion, changes or corruption due to system failures, and accidental or malicious actions by otherwise authorised users. In order to eliminate this possibility, the Trust IT Department recommended permanent, on-line WORM storage as our back-up solution. Files stored on the WORM can be accessed, but they cannot be overwritten or deleted. This means that we have a permanent back-up of every digital file that cannot be tampered with.<br />
<br />
<b>Robustness </b>is tied up with the WORM system - although the RAID5 storage we normally use is highly robust. It distributes the bit stream of digital files in such a way that points of failure generally do not damage the entire file, allowing it to be reconstituted, while short-term back-ups allows complete recovery of damaged or lost files. The WORM system adds a further element of confidence. Once a new file is stored on the main servers, it is copied to the WORM drive. The software managing this process checks the files on the main servers periodically, and if it finds a mis-match with the WORM drive (a lost or corrupted file on the main server), it pulls a copy of the WORM'ed file to the main server, thus "self-healing" the damage. <br />
<br />
<b>Replicability </b>means that there is more than one copy of the file. One lives on the main servers at the Wellcome Trust offices, and the other is stored outside of London. If either server is damaged in a serious accident, one server remains to keep the content safe.<br />
<br />
<b>Scalability </b>is important, as we are creating millions of images during the pilot project, and will create up to 30 million images over the longer term as we digitise the Wellcome Library. All the systems that are in place must be able to increase capacity - both in terms of hardware and processing software. The system we selected is scalable – we simply need to add storage “bricks” and “racks” (additional hardware), and processing units to manage that additional hardware, as our data store grows over time.<br />
<br />
<b>Speed of access </b>is also key. In order to keep our storage footprint as small as possible, we are using one file as both the master archive file (preserved in the long-term, and backed-up to the WORM) and as the dissemination file. Our DAM, SDB, will maki copies of the archived files available to the front-end delivery system as the user requests images via the Wellcome Digital Library (or via back-end administrative systems), and this must be handled as quickly as possible – something RAID5 is particularly well suited for.Christy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.com1tag:blogger.com,1999:blog-6380997420931943041.post-76751129250885196502011-07-04T10:14:00.001+01:002012-03-05T18:59:28.690+00:00Preserving our digital assets #1 – SDB4The successful long term management of digital assets is a key concern for us as we build our digital resource. Until recently, we did not have a dedicated system in use for storing and managing master files for any digitised content, and our file backup system was not idea for dealing with large sets of data. Files were managed via simple filesharing on dedicated storage servers. Backups were only created for some content, and even then the backups were not permanent.<br /><br />When the <a href="http://http//wellcomelibrary.blogspot.com/2010/08/wellcome-library-launches-major.html">Wellcome Digital Library </a>was initiated it quickly became clear that we needed a dedicated system to manage our digital masters - something that was scalable, robust, and could handle all the digital formats we create or procure - including <a href="http://en.wikipedia.org/wiki/Born-digital">born digital </a>material. We also needed a secure storage system with offsite, permanent backup capability. This blog post describes our digital asset management system; a future blog post will provide details of our secure storage solution.<br /><br />We already had an existing digital asset management system in place to manage born digital archives: Safety Deposit Box (SDB), developed by <a href="http://www.digital-preservation.com/">Tessella</a>. This system incorporates a suite of tools designed to manage and preserve digital files. It provides a context in which administrative and descriptive metadata is associated with all ingested content. SDB can be combined with tools that can "migrate" files from one format to another to counter format obsolescence and therefore ensure the longevity of the data. In other words, when Word 2010 is no longer supported by Microsoft, SDB can help migrate these files to a current format that is supported by software available at that time.<br /><br />However, in order to manage preservation of large sets of digitised content, SDB “out of the box” was not entirely able to meet the needs of the Wellcome Digital Library. We carried out a <a href="http://liber.library.uu.nl/publish/issues/2010-1/index.html?000490">Feasibility Study </a>in 2010 to determine whether it would be possible to use a modified version of the system. The research and prototype system we commissioned proved that it was indeed feasible to use SDB with certain software extensions.<br /><br />In the spring of 2011, we <a href="http://www.tessella.com/2011/05/press-release-comprehensive-archiving-solution-to-manage-wellcome-trust%E2%80%99s-digitised-and-born-digital-assets/">commissioned </a>Tessella to extend and install the newest version of the software (SDB4). This work is now complete and in the testing phase.<br /><br />Key preservation functionality that SDB4 provides is listed here (some of this was “out of the box”, some were developed as extensions to the core system or as modules):<br />
<ul><br />
<li>Automated ingest: automated SDB workflow to create and ingest a “submission information package” (SIP) – a bundle of content files and metadata forming a complete “object”</li>
<br />
<li>Multiple manifestations: ability to associate all the different manifestations of an object together (e.g. master video file, broadband versions, narrowband versions, transcripts, etc.)</li>
<br />
<li>SQL Server database: stores and indexes administrative and descriptive metadata describing objects stored in SDB</li>
<br />
<li>Characterisation: use of the JHOVE, DROID and PRONOM characterisation tools to extract essential technical information about digital files to be stored in the database</li>
<br />
<li>Format Migration: SDB4 builds on the PLANETS framework to support preservation planning for mitigation of obsolescence</li>
<br />
<li>Integrity checking: creates and stores a unique SHA-1 hash code for each file that can be used to test validity of the file over time</li>
<br />
<li>Provide access to content: delivers content to external systems using an API</li>
<br />
<li>Automatic export of administrative metadata: allows us to store metadata such as unique SDB identifiers in our workflow system in order to deliver files to the user</li>
<br />
<li>Administrative interface: allows administrators access to the content and the database, and to generate reports</li>
</ul>
<br />
This is by no means a complete description of SDB’s functionality, but the above provides a flavour of the most important features for long-term asset management and access to master files. Much of the development work that was done to extend SDB4 is likely to have an application for other users, not just the Wellcome Digital Library. Where possible, extensions are generic – although system requests/commands and metadata mappings may be highly specific to us.<br />
<br />
We hope that our efforts in specifying our own needs will benefit other SDB users who see the value in long term preservation for digitised materials, and the efficiencies of combining born digital and digitised content into a single system strategy.<br />
<br />
In future, SDB will interface with our workflow system and the image/media servers employed by our digital delivery system. These latter two systems have not yet been implemented.Christy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.com8tag:blogger.com,1999:blog-6380997420931943041.post-46790870203355130422011-04-04T09:54:00.017+01:002011-04-04T15:14:11.489+01:00Arabic manuscripts online - Fihrist launch<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI2aIJ6Rq7TuDLkF1202zXp0279lwL4_xYkD2NayK2f-IdeEfOv1WOnTgFhFvjbWWHiJZB5cCrapltplr69GSOp4XMCLwPBt1fjTQ8qSehx5DFtR-7IKf8sPmC0bDy3JLGp0gqImxtD0pL/s1600/Fihrist.jpg"><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 400px; height: 62px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI2aIJ6Rq7TuDLkF1202zXp0279lwL4_xYkD2NayK2f-IdeEfOv1WOnTgFhFvjbWWHiJZB5cCrapltplr69GSOp4XMCLwPBt1fjTQ8qSehx5DFtR-7IKf8sPmC0bDy3JLGp0gqImxtD0pL/s400/Fihrist.jpg" alt="" id="BLOGGER_PHOTO_ID_5591659156362997762" border="0" /></a><br /><a href="http://www.fihrist.org.uk/home">Fihrist</a>, an online catalogue for Islamic manuscripts held at the Bodleian and Cambridge University Library (CUL), was launched on 28 March. Developed by the the <a href="http://www.bodleian.ox.ac.uk/bodley/library/specialcollections/projects/ocimco">OCIMCO</a> (Oxford & Cambridge Islamic Manuscripts Catalogue Online) project, the catalogue uses the <a href="http://wellcomelibrary.blogspot.com/2010/08/guest-post-elena-pierazzo-on-arabic.html">TEI/XML</a><a href="http://wellcomelibrary.blogspot.com/2010/08/guest-post-elena-pierazzo-on-arabic.html"> schema</a> created as part of the <a href="http://library.wellcome.ac.uk/doc_wtdv027097.html">WAMCP</a> (Wellcome Arabic Manuscript Cataloguing Partnership) project to structure the descriptions and to provide for future enhancement of those records.<br /><br />Fihrist contains around 10,000 catalogue entries, retroconverted from hard copy lists and card catalogues at the Bodleian and CUL, providing an integrated online search tool for these large collections. Searching for a manuscript or a work shows you a list of works contained in the relevent manuscript, with separate descriptions provided for each work. The catalogue records are currently very brief, but the OCIMCO project "will eventually provide detailed manuscript descriptions that will include digital representations of the manuscripts themselves. The TEI/XML schema provides an extensible framework that will allow for these future enhancements" (<a href="http://www.fihrist.org.uk/about"><span style="text-decoration: underline;">About us</span></a>).<br /><br />The project recieved funding from the JISC’s <a href="http://www.jisc.ac.uk/islamdigi">Digital Resources for Islamic Studies</a> programme - a scheme that also part-funded the WAMCP project.<br /><br />At the Fihrist launch at Clare College, Cambridge, a series of presentations from the OCIMCO project managers and staff provided further details of the collections and the technical implementation of the project; a presentation from the JISC on the Islamic Studies programme provided background to the funding scheme; a joint presentation was given by Richard Aspin and Nikolai Serikoff of the Wellcome Library, and Gerhard Brey of King's College London on the WAMCP project; and we heard about the like-minded <a href="http://www.soas.ac.uk/ysimg/">Yale/SOAS Islamic Manuscript Gallery</a>.<br /><br />The day then turned to a follow up project that Oxford is carrying out over the summer to create a union catalogue of Islamic manuscript catalogues (the <a href="http://www.jisc.ac.uk/whatwedo/programmes/digitisation/islamdigi/islamicstudies.aspx">Islamic Studies Gateway</a>). This project was recently awarded JISC funding, and will seek to develop Fihrist further to "provide cross searching of existing online manuscript resources ... that at present do not have a significant internet presence."<br /><br />"In addition to creating the gateway itself, the aim of the Fihrist is to create a sustainable user community of Islamic manuscript metadata standards and cataloguing tools to ensure a long term commitment by stakeholders to supporting and developing the Fihrist beyond the lifetime of the project."<br /><br />The WAMCP partnership (comprising the <a href="http://library.wellcome.ac.uk/">Wellcome Library</a>, <a href="http://www.bibalex.org/Home/Default_EN.aspx">Bibliotheca Alexandrina</a>, and <a href="http://www.kcl.ac.uk/artshums/depts/ddh/redirect.aspx">King's College London</a>) will collaborate on this project by providing open access to metadata for 500 manuscripts. This metadata, with digitised manuscripts, is expected to be publically available online via the WAMCP website from Summer 2011.Christy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.com0tag:blogger.com,1999:blog-6380997420931943041.post-62598675018351225112011-03-23T14:30:00.007+00:002011-03-23T15:22:04.667+00:00The problem with orphans…<a href="http://images.wellcome.ac.uk/ixbin/hixclient.exe?MIROPAC=C0063524"><img id="BLOGGER_PHOTO_ID_5587294743550823794" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; WIDTH: 320px; CURSOR: hand; HEIGHT: 221px" alt="" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTdu8HJi-W8kQgtViSUVDIxD1s7926pDwlvOaxs3T-xBQbposX2TxSx7w2-s58TMFV6P7dg_B2MQPRIn4N0auKETSoXjgmSK2fs48KS2Iy78SSuR9s_7CQfpiFufBQb5SEwv0Sm0-Tb0I/s320/genetics+books.jpg" border="0" /></a><br />Sooner or later, anyone planning to digitise works created in the last hundred years or so has to face the problem of what to do with those which are still in copyright but for which no copyright holder can be traced: so-called "<a href="http://en.wikipedia.org/wiki/Orphan_works">orphan works</a>". On 17 March, three of us travelled a few hundred yards down the road to attend a workshop on this very issue. The workshop was arranged as part of the "<a href="http://www.whoownstheorphans.org/Home.html">Who owns the Orphans?</a>" project from the <a href="http://www.beyondtext.ac.uk/">Beyond Text</a> programme, sponsored by the UK <a href="http://www.ahrc.ac.uk/Pages/default.aspx">Arts and Humanities Research Council</a>. The morning focused on the highlights of the interim report which has been produced while the afternoon was given over to panel discussions on different aspects of the problem. The attendees came from a range of perspectives, including representatives from libraries, archives, museums and galleries, as well as creators of content – authors and photographers – who brought a different angle to the discussions.<br /><br />The day began with an admission that it was tricky even just to define orphan works, and it quickly became evident that there is no easy solution to the problem. Various approaches were proposed, including compulsory registration and blanket licensing schemes, but all have their drawbacks and none really address the problem of the historical orphan works which exist in all collections. Many institutions admitted to taking a risk-managed approach to orphan works in order to maximise what could be made available to the public online. The sense from the participants was that as long as the material in question was not of high commercial value, then making it available after a due diligence search for the copyright holder had drawn a blank might be infringing copyright but was preferable to not making these cultural works widely available.<br /><br />Compulsory registration of creative works is already the norm for the music industry, hence they do not have the same problems with orphan works faced by much of the cultural sector. However, imposing compulsory registration on other creators such as authors and photographers could be more difficult: the records of copyright registration which ended in 1912 show that few people actually bothered to register their work last time compulsory registration existed. In any case, this still leaves the problem of tracing subsequent copyright holders after the death of the original creator and it does not alter the position of historical orphan works.<br /><br />Licensing schemes are favoured by the EU, but they also have their problems, both for authors who wish to retain control of their work and for cultural institutions who are reluctant to pay collecting societies to license works where they have no contact with the author or their heirs. Some collecting societies might be happy to license orphan works, setting aside money to pay out in the event of a copyright holder making themselves known. However, although this would in effect act to indemnify the institution who licenses the material, it was pointed out that this would really be no different to taking out insurance.<br /><br />While no-one wants to infringe copyright, if a copyright holder cannot be traced after a due diligence search has taken place, most people working in the cultural sector consider that making material available to the public online is part of their obligation as guardians of the material. Some orphan works may have low commercial value, or their value may come from their position as part of a whole collection. It was also noted that people who have inherited copyright in these works are often unaware that they own rights in items, but once the work becomes commercially valuable, these rightsholders magically appear.<br /><br />It seems likely that current practices will continue for the time being, due to a lack of viable alternatives. However, the outcome of the <a href="http://www.ipo.gov.uk/ipreview.htm">Hargreaves Review</a> may have an impact on how orphan works are dealt with in the future, and perhaps the final report from the “Who Owns the Orphans?” project will be able to suggest practical solutions to what is acknowledged to be an issue across the cultural sector.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6380997420931943041.post-50419280581525182072011-02-28T19:27:00.003+00:002011-02-28T19:43:12.165+00:00Wellcome Library joins Strategic Content AllianceThe Wellcome Library is pleased to announce that we’re the latest partners to join the <a href="http://www.jisc.ac.uk/contentalliance">Strategic Content Alliance</a>. You can read the press release <a href="http://www.jisc.ac.uk/news/stories/2011/02/scawellcome.aspx">here</a>.<br /><br />Why join? Well, as our digitisation programme gathers pace, one of the key issues we face is making sure that our plans are not developed in isolation. We’re in a lucky position: unlike many libraries and archives, we have the financial resources of the <a href="http://www.wellcome.ac.uk/">Wellcome Trust</a> to draw upon to support our work. But one of our aims is to make sure that our digitised content is made available to – and is useful to – as wide a range of users as possible, and to make sure that we achieve best value for, and embody best practice in, our programme.<br /><br />The aim of the Strategic Content Alliance is to bring together key public and not-for-profit sector organisations involved in the creation, management and exploitation of digital content for the common good. By facilitating high-level discussion between partners, the SCA aims to ensure the maximum return on public sector and charitable investment in digitisation. Being part of the Strategic Content Alliance enables us to keep abreast of what others – including the <a href="http://www.bbc.co.uk/">BBC</a>, <a href="http://www.jisc.ac.uk/">JISC</a> and the <a href="http://www.bl.uk/">British Library</a> – are up to, and to identify and pursue opportunities for partnership. Some of these are around the sharing of expertise (such as <a href="http://beta.digipedia.org.uk/">Digipedia</a>, a pilot project developed by SCA), tools to make digital content accessible, and business models that ensure the sustainability of the content we create and the repositories that house it (see for example <a href="http://sca.jiscinvolve.org/wp/2011/02/21/workshop-new-and-emergent-internet-business-models-in-updated-case-studies-on-sustainability/">this post</a> on the <a href="http://sca.jiscinvolve.org/wp/">SCA blog</a> about work being undertaken by Ithaka S+R for the SCA). Others are about aggregating content and finding ways to complement each others’ work (as with the recent JISC meeting on the <a href="http://sca.jiscinvolve.org/wp/2011/02/21/first-world-war-digital-commemoration-roundtable/">First World War Commemoration</a>). The SCA also acts as a coordinating body to provide sound business intelligence and ensure effective advocacy on issues that cut across our shared interests, such as copyright and orphan works. Nor is the benefit limited to our in-house programmes: our involvement with the SCA will also help to inform the Wellcome Trust’s strategy towards the funding of digitisation projects elsewhere.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6380997420931943041.post-75486917455502466752010-12-06T08:33:00.004+00:002011-05-06T09:44:34.856+01:00Workflow Tools Workshop in Bath, 30 November 2010<div>On a freezing cold November day I travelled down to Bath to attend a one-day event run by <a href="http://www.ukoln.ac.uk/">UKOLN </a> as part of the <a href="http://devcsi.ukoln.ac.uk/">DevCSI program</a>. Mahendra Mahey, Research Office with UKOLN, was our host for the day. What may seem like an esoteric activity for the few is actually a fascinating subject for those with a logical and ordered approach to things. We’ve used some business modelling techniques in developing the basic model for our <a href="http://library.wellcome.ac.uk/doc_wtx052496.html">Workflow Tracking System</a> so the workshop was very relevant to us. <br /><br />The aim of the day was to introduce a small range of open source workflow tools to a wider audience and to set out the role and value of using software to model business processes. The day consisted of a series of presentations and product demonstrations. <br /><br />The event brought together vendors, users and project managers who are interested in learning more about workflow and business process modelling tools. In some way or another all attendees were involved in planning or developing workflows within their own institutions. <br /><br />The day started with a presentation by Tammo van Lessen, who gave an overview of the basics of business process modelling, languages such as BPEL (Business process Execution Language) and BPMN 2.0 (Business Process Model and Notation). Business process modelling is just a formal way of describing the processes that a business uses to do what it does. It can be used to describe each step in a process, the interdependencies between those steps and sets out a sequence in which steps must occur. Tammo talked about the history of business process modelling and how the tools have become more interactive and how they can now be used to directly build web based interactive services.<br /><br />Amol Vedak gave a demonstration of the <a href="http://www.intalio.com/bpm">Intalio Business Process Management System</a> and discussed how it could be used to as a tool for business transformation. Intalio is designed to combine the skills of business analysts and IT people. Groups of experts can use their respective skills to design and build business processes using a common language and set of tools. The simplicity of the tools means that processes can be quickly designed, tested and implemented.<br /><br />The <a href="http://wso2.com/">WS02 Business Process Server</a>, discussed by Paul Fremantle, is an open-source BPEL. Paul demonstrated the ease with which a workflow could be built diagrammatically. At the same time the software builds real web services that can be used by people or systems and integrated into an organisations business.<br /><br />Of particular interest was the Taverna suite of scientific workflow tools, demonstrated by Stian Soiland-Reyes. This tool can be used to build the workflows that query public data sources. Taverna can use local tools or third party scripts to query services such as EBI’s BioMart. Taverna recognises that many scientists have data or their own tool sets that are held locally and it provides a means by which these can be used by others.<br /><br />At the end of the day there was a series of ‘Lightening talks’ given by some of the workshop attendees looking at workflow related projects that they are currently working on. This provided a set of practical real-world examples of how business process modelling could be used.<br /><br />The lessons of the day were quite clear. Using business process modelling tools is an effective way to help us better understand the processes that we use in our business, and it can help us design better more efficient processes. In better understanding how we do what we do we can better understand the risks to our business, design more efficient processes and do so relatively quickly and easily. We’re also able to use these techniques to run ‘what if’ scenarios and to test processes under different scenarios. All of which can be done in a virtual world before implementation.</div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6380997420931943041.post-59017807451262506022010-11-29T15:33:00.003+00:002010-11-29T15:37:59.060+00:00Wellcome Library releases an ITT for a Workflow Tracking SystemIf you’ve been reading our blog regularly you’ll know about how the <a href="http://www.wellcome.ac.uk/News/2010/News/WTX062533.htm">Library</a> plans to transform itself into a groundbreaking digital resource, allowing access to much of the Library’s material in digital form.<br /><br />As part of this program we’ve just released an ITT for a <a href="http://library.wellcome.ac.uk/doc_wtx052496.html">Workflow Tracking System</a> (WTS). We’re looking for a system that will track and manage the processes around creating digital content – whether that content is digitised by us, digitised externally or born digital archival material- and automating that activity as much as possible.<br /><br />Within the Library, staff who want to add content to our Digital Library will do so using the Workflow Tracking System. This means using the WTS to record that all digital content, e.g. digitised books or archival collections, has been created correctly, has had its descriptive metadata attached, is converted to JPEG2000 (or some other appropriate format) and is ingested into our digital object repository. The WTS will also create metadata encoding and transmission standard (METS) files. These will be used by the front end system to deliver digital content to our users.<br /><br />Expressed simply, the WTS will play a critical central role in ensuring that all digital content that is destined for our Digital Library is created, quality controlled and ingested accurately and efficiently into the Library’s repository.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6380997420931943041.post-92008905589340322542010-11-24T14:57:00.002+00:002010-11-24T15:06:49.060+00:00Wellcome Trust hosts JPEG 2000 seminarThe <span style="font-style: italic;"><span class="blsp-spelling-error" id="SPELLING_ERROR_0">JPEG</span> 2000 for the Practitioner</span> seminar was hosted by the <span class="blsp-spelling-error" id="SPELLING_ERROR_1">Wellcome</span> Trust on 16 November, 2010. This was organised by the <span class="blsp-spelling-error" id="SPELLING_ERROR_2">Wellcome</span> Library, and achieved a sell-out crowd of over 80 people.<br /><br />The aim of the seminar was to look at specific case studies of <span class="blsp-spelling-error" id="SPELLING_ERROR_2"><span class="blsp-spelling-error" id="SPELLING_ERROR_3">JPEG</span></span> 2000 use, to explain technical issues that have an impact on practical implementation of the format, and explore the context of how and why organisations might choose to use <span class="blsp-spelling-error" id="SPELLING_ERROR_3"><span class="blsp-spelling-error" id="SPELLING_ERROR_4">JPEG</span></span> 2000. The <span class="blsp-spelling-error" id="SPELLING_ERROR_5">Wellcome</span> Library started investigating <span class="blsp-spelling-error" id="SPELLING_ERROR_6">JPEG</span> 2000 as a strategy for storing its archival master images in 2009, and has recently started converting its backlog of images into <span class="blsp-spelling-error" id="SPELLING_ERROR_7">JPEG</span> 2000.<br /><br />The <a href="http://jpeg2000wellcomelibrary.blogspot.com/2010/10/jpeg-2000-seminar-draft-programme-now.html">programme </a>for the event was posted to our <span class="blsp-spelling-error" id="SPELLING_ERROR_8">JPEG</span> 2000 blog, and further <a href="http://jpeg2000wellcomelibrary.blogspot.com/2010/11/jpeg-2000-seminar-edited-highlights-1.html">blogs </a>provide edited highlights of the varied and informative talks given on the day. All the presentations are available <a href="http://www.dpconline.org/events/details/19-jpeg-2000-for-the-practioner?xref=19">online</a>, hosted by the <a href="http://www.dpconline.org/">Digital Preservation Coalition</a>. The twitter stream from the day can be seen by searching the <span class="blsp-spelling-error" id="SPELLING_ERROR_9">hashtag</span> #<a href="http://twitter.com/#search?q=%23jp2k10"><span class="blsp-spelling-error" id="SPELLING_ERROR_10">jp</span>2k10</a>.Christy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.com0tag:blogger.com,1999:blog-6380997420931943041.post-34143735036977898322010-10-29T11:42:00.011+01:002010-11-01T09:26:47.156+00:00100,000 image milestone, and how we did it<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuOH5XHBPHJ0-nKRf5FuwXpkO85maNa2TFVRE9kHnf4ZRnBCIiKsHwaU9H7jAo7PXd2vpZZMh6sqt9Tb7iSwviL6eD41e-VEIZxtp3LzoQ7ij-ciRoAb0e5DWguXKCh0wD31THbna1rVBF/s1600/cameracolumn.jpg"><img style="float: left; margin: 0pt 10px 10px 0pt; cursor: pointer; width: 213px; height: 320px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuOH5XHBPHJ0-nKRf5FuwXpkO85maNa2TFVRE9kHnf4ZRnBCIiKsHwaU9H7jAo7PXd2vpZZMh6sqt9Tb7iSwviL6eD41e-VEIZxtp3LzoQ7ij-ciRoAb0e5DWguXKCh0wD31THbna1rVBF/s320/cameracolumn.jpg" alt="" id="BLOGGER_PHOTO_ID_5534506909561634866" border="0" /></a>A significant milestone was reached this week, with 100,000 images of Crick papers photographed. The Crick collection, previously <a href="http://wellcomedigitallibrary.blogspot.com/2010/09/digitising-archives-wellcome-library.html">described</a>, includes around 285,000 images - over half the total amount to be digitised as part of the <a href="http://library.wellcome.ac.uk/doc_WTX057852.html">Archives digitisation project</a> over a period of 20 months by Laurie Auchterlonie and Tom Cox in the <a href="http://wellcomelibrary.blogspot.com/2010/07/behind-scenes-imaging-services.html">Imaging department</a>.<br /><div><br />Laurie and Tom use <a href="http://www.canon.co.uk/for_home/product_finder/cameras/digital_slr/EOS_5D_Mark_II/">Canon Mark II </a>digital SLR cameras mounted to columns attached to two copy stands. These allow the cameras to be automatically raised and lowered according to the size of the items being digitised. This is a fairly typical imaging set-up for this type of material. However, with a project this large, it is important to ensure that the minimum amount of time is taken to photograph, edit and manage the images. In spring 2010, the photographers spent time developing a workflow that would allow them to digitise the highest number of items possible, whilst not compromising on quality or care in handling. It was in fact found that time-saving measures actually resulted in higher quality images, and minimised the amount of handling required.</div><br /><div> </div><br /><div>For example, "live view" screens allow the photographers to easily see and adjust the alignment of each item on the copy stand, and the degree to which it fits the frame of view. This saves time as the photographers do not have to look through the viewfinder (difficult when the camera is 6 or more feet above the ground), or take multiple shots to get it right. Post-processing work has also been almost completely eliminated, as has the need to reshoot items at a later date.</div><br /><div>Purchasing higher columns limited the number of times lenses had to be changed. Larger items require the cameras to be raised quite high, and if the column isn't high enough, the photographer has to change to a different, shorter or wide-angle lens. The flexibility built into the workflow by these measures is highly advantageous when dealing with heterogeneous materials such as personal archives.<br /></div><br /><div>Other aspects of the workflow that had to be specifically tailored to archival collections was the storage and foldering of images so that they could easily be found and identified. Using the existing archive catalogue hierarchies, the foldering system allows the user to pinpoint the exact file or item to be viewed to create copies for users, or to carry out QA against the original items (a sample of images is checked against the originals by Julia Nurse, who <a href="http://wellcomedigitallibrary.blogspot.com/2010/10/papers-papers-and-yet-more-papers.html">prepares </a>the items before photography, to ensure that filenaming is accurate and that items aren't being missed). Eventually, these folders will be rendered obsolete, as we implement a digital asset management system that will restructure the archive storage of our images on ingest. But it is important not to underestimate the need to access images during the pre-ingest process of digitisation and QA. And if you do not have a digital asset management system, it is even more important that ease of access is factored in from an early stage.</div><br /><div>A bit of preparation and testing makes a huge difference when setting up a new workflow. Even a minute saved per item means a large overall time saving when spread over hundreds of thousands of images.</div>Christy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.com0tag:blogger.com,1999:blog-6380997420931943041.post-32688009046810669102010-10-13T09:00:00.002+01:002010-10-13T09:06:46.928+01:00Papers, papers and yet more papers … preparing the Crick Archive for digitisation<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiATlqtOb9yjAluW0ywGllgkiKsKF6uvvFL1ENP5Cu8-m-lTu7hfaECnuqCmvaYGSNy-Ns3ssT37h82CtxM1g9qsgxx62mPPsRT1zlrHEGKvnZXodd2kd9_VZpXxpV00E64CRT1pDkq6RH-/s1600/Cricksketch.jpg"><img style="float: left; margin: 0pt 10px 10px 0pt; cursor: pointer; width: 238px; height: 320px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiATlqtOb9yjAluW0ywGllgkiKsKF6uvvFL1ENP5Cu8-m-lTu7hfaECnuqCmvaYGSNy-Ns3ssT37h82CtxM1g9qsgxx62mPPsRT1zlrHEGKvnZXodd2kd9_VZpXxpV00E64CRT1pDkq6RH-/s320/Cricksketch.jpg" alt="" id="BLOGGER_PHOTO_ID_5527437216953323010" border="0" /></a>When first faced with preparing around 300 boxes of Francis Crick’s personal papers for digitisation, I have to confess my heart sank. A far cry from the last very visual digitisation project of 3000 AIDS posters, I was daunted, not only by the very different content of this collection, but the sheer size of it – an estimated half a million items this time. How wrong I was. I feel privileged to have been given the opportunity to delve into one of the most incredible minds of our lifetime.<br /><br />Although we tend to associate him only with his (and Jim Watson’s) discovery of the double-helix sequence of DNA – and there is plenty of fascinating correspondence within the archive related to this - it is his research on the mind and consciousness in his latter years that is truly ‘astonishing’ as he would put it. Through his endless correspondence with both fellow scientists and the general public, we get a real sense of his probing analysis of what makes our brains tick.<br /><br />It is very easy to get side-tracked from such a collection but I have to remember that my main task is to ensure the papers are in a suitable condition for the photographers to shoot: a daily scour of the collection is required to remove existing staples and flatten pages but occasionally a conservator is required. For example, Crick’s heavily folded (since 1955) tracing sketches and calculations of <a href="http://bit.ly/9VUt2V">Collagen Long Spacings</a> required specialist equipment to flatten out.<br /><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9Vyv7hPYXetjA8-Ohmiqyy8rayVS_sKqNqQDN7mEaRhIsPx5fZMcT9nbilyxt2MNq2678b6BglX7azds__qiDA0ErOxtVoaZRJ4p2VDf6N7SztfmC5QCxm9GNd-3KE8XjL5moIJIDeN3z/s1600/Crick-lecturing.jpg"><img style="float: left; margin: 0pt 10px 10px 0pt; cursor: pointer; width: 220px; height: 320px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9Vyv7hPYXetjA8-Ohmiqyy8rayVS_sKqNqQDN7mEaRhIsPx5fZMcT9nbilyxt2MNq2678b6BglX7azds__qiDA0ErOxtVoaZRJ4p2VDf6N7SztfmC5QCxm9GNd-3KE8XjL5moIJIDeN3z/s320/Crick-lecturing.jpg" alt="" id="BLOGGER_PHOTO_ID_5527437221071951042" border="0" /></a>Once a particular batch has been checked, data spreadsheets are then produced for the photographers so that they know what to expect in each box – included in this data is an estimate of the percentage of OCR’able (Optical Character Recognition) text, a record of the current location of a particular batch and notes for the archivists’ attention. While doing this I cannot help but siphon off particularly interesting information which has and will continue to be used for publicity about the project – see the recent BBC Audio <a href="http://www.bbc.co.uk/news/science-environment-11016755">Slideshow</a>.<br /><br />Further blogs providing updates on the digitisation project will follow in due course.<br /><br /><br /><span style="font-size:100%;"><span style="font-style: italic;"><br /><br /><br /><br /><br /><a href="http://images.wellcome.ac.uk/ixbin/hixclient.exe?MIROPAC=L0051869">Top image</a>: Crick's sketch of genetic code, 1965 (<a href="http://bit.ly/aCpvOR">PP/CRI/E/1/13/10</a>)</span><br /><span style="font-style: italic;"><a href="http://images.wellcome.ac.uk/ixbin/hixclient.exe?MIROPAC=L0042633">Bottom image</a>: Francis Crick lecturing at Cambridge University (<a href="http://bit.ly/9t1qf7">PP/CRI/A/1/2/9</a>)</span></span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6380997420931943041.post-91815905341229545242010-10-01T11:50:00.007+01:002010-10-02T14:47:39.001+01:00Preparing and planning a large archives digitisation project<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZSi2JwxdNsg3n4_3oHH9a2KElwmVsoARieeD4_0fyoHUg8yUuKOvJENC-Jv63sCntqshuJ55zgBeMb2eMI5GykNoPDcnUrktHr58Z-61Q9MXHljOjTZyDZFOK94n5X5OMLTa6x8IbX_RI/s1600/archives.jpg"><img style="float: left; margin: 0pt 10px 10px 0pt; cursor: pointer; width: 132px; height: 200px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZSi2JwxdNsg3n4_3oHH9a2KElwmVsoARieeD4_0fyoHUg8yUuKOvJENC-Jv63sCntqshuJ55zgBeMb2eMI5GykNoPDcnUrktHr58Z-61Q9MXHljOjTZyDZFOK94n5X5OMLTa6x8IbX_RI/s200/archives.jpg" alt="" id="BLOGGER_PHOTO_ID_5523029582583705074" border="0" /></a>Archives digitisation is currently underway in our Imaging Studio, with two full-time members and two part-time members of Library staff dedicated to preparing and digitising the items. We will talk more specifically about the work being carried out on these materials on this blog in the near future, but first we present an introduction to the setup and planning of the project.<br /><br />Once the theme was chosen (Modern Genetics and its Foundations), and relevant collections identified (see previous <a href="http://wellcomedigitallibrary.blogspot.com/2010/09/digitising-archives-wellcome-library.html">blog post</a>), we realised that we had quite a large job on our hands. The scope of the project was bigger than anything we had done before: 620 boxes of material, containing around 800 pages each, adds up to around half a million pages to be digitised.<br /><br />Based on a series of tests, we estimated the project would take 2 years to complete – starting with the preparation of the material in advance, with photography coming into play a few months later. Two full-time staff would be focused on imaging the material, with two part-time member of staff preparing, tracking and assessing the items.<br /><br />There are a range of logistical issues to bear in mind when planning and starting up a project of this nature. The boxes are stored in the basement stores, and had to be retrieved for a period of some months while the material was being worked on. We divided the collections into batches of a size that could be imaged in a period of 4-6 weeks and retrieve and return each batch as a unit, tracking all movements on a spreadsheet. The tracking spreadsheet also records information such as location of each box in the batch, notes from the preparer for the archivists, photographers, and/or conservation staff, and the percentage of items in each box that can be OCR’d among other things.<br /><br />We put a <a href="http://library.wellcome.ac.uk/doc_wtx058080.html">notice </a>on our website of the entire schedule of archives to be digitised, so readers could see at a glance what would be unavailable and when. The catalogue records are also amended to show where material cannot be reserved. Each time a batch is retrieved, checked out, checked in, or dates altered, this has an impact on the website and two different cataloguing systems (the <a href="http://library.wellcome.ac.uk/node49.html">Archives and Manuscripts Catalogue</a>, and the <a href="http://library.wellcome.ac.uk/">Library Catalogue</a>), so communication with the departments responsible for retrieval and <span class="blsp-spelling-error" id="SPELLING_ERROR_0">metadata</span> was key.<br /><br />The preparation staff were trained in advance by the conservation team so they could carry out basic stabilisation and first-aid work on the materials if required for digitisation. The photographers ran multiple tests on different equipment and with different cameras to ensure the <span class="blsp-spelling-error" id="SPELLING_ERROR_1">workflow</span> was efficient and appropriate to the formats of the material, the anticipated end use of the material, and to ensure proper <span class="blsp-spelling-error" id="SPELLING_ERROR_2">QA</span> could be accommodated. Preparation and imaging takes place in the Imaging Studio - ensuring that all staff are in close proximity and able to communicate easily with each other. The Imaging Studio was refitted with desks, shelving and equipment to make sure all the boxes in process at any one time could be accommodated. A further planning issue was in determining how to assess and record different levels of sensitivity of information contained in the archives. We are currently developing a policy for access to archives that takes account of online display, and this has informed the <span class="blsp-spelling-error" id="SPELLING_ERROR_3">workflow</span> for assessment.<br /><br />This project required liaison between several different departments and stakeholders in the Library in order to set up a suitable <span class="blsp-spelling-error" id="SPELLING_ERROR_4">workflow</span>. In future, we hope that <span class="blsp-spelling-error" id="SPELLING_ERROR_5">workflow</span> issues will be streamlined further by procuring a <span class="blsp-spelling-error" id="SPELLING_ERROR_6">Workflow</span> Tracking System that will serve to centralise tracking and monitoring of all digitisation projects. We anticipate that this pilot project will enable us in future to plan effectively for much larger digitisation projects as we work towards the digitisation of all suitable material held in the <span class="blsp-spelling-error" id="SPELLING_ERROR_7">Wellcome</span> Library.Christy Henshawhttp://www.blogger.com/profile/13179015500410216822noreply@blogger.com0