Software and tear: the challenge of archiving born-digital content

A computer hard drive arrives on Tom Ackroyd’s desk. Ackroyd, the Digital Collection Team Leader at Ngā Taonga, deals with this kind of thing all the time. He plugs the drive in to his network-isolated computer and sees what it holds. Ackroyd collects ‘born-digital’ material. ‘That means the production was shot and cut in an entirely digital way; in the digital realm,’ he explains. Moving images can be shot on physical film, or they can be recorded to videotape or other formats. Sound can be recorded to magnetic tape or cut direct to a disc.

‘In recent years though,’ reflects Ackroyd, ‘we’ve seen the movement away from material being stored physically to purely digital files, stored on hard disks and servers, and moved around on the Internet using services like Dropbox or Google Drive.’ However it’s made, the audiovisual material created today becomes the archived material of tomorrow.

Digital Collection Team Leader Tom Ackroyd. Photo by Jo Dixon-Didier.

As you can imagine, born-digital covers a huge amount of content. Fortunately for Ngā Taonga, Ackroyd has a good background in working with and managing digital assets. His early career began with basic videotape editing, before gaining experience at an advertising agency using Final Cut Pro. ‘A lot of my work was with large numbers of digital audiovisual assets,’ he explains. ‘They needed management and I developed an understanding of archiving.’ After that he was ideally placed to bring that experience and knowledge to Ngā Taonga.

The flow from videotape to digital is reflected around the world over the last few decades – aside from niche film or tape productions, almost all new media created is entirely digital.

‘There is a boom of born-digital content: Instagram stories, Facebook videos, Tik Toks, and gifs on Twitter,’ explains Ackroyd. ‘It’s a fire hose of audiovisual digital material flying around on social media and places like YouTube.’

This deluge is at the crest of a wave of changes. ‘Institutions across the planet are dealing with this explosion in the democratic production of audiovisual materials. With more and more people creating content, it’s started us on a journey towards new ways of dividing up responsibilities and collection strategies.’ As it stands, while Ackroyd collects born-digital material, his colleagues who collect film or radio also bring in digital content.

As long as films remain under people’s beds or videotapes sit in post-production houses, ‘then we will need to have the expertise of particular members of staff to deal with those materials and formats.’ Despite the broad scope of digital material, ‘90 percent of what I collect is from traditional modes of production – film, short film, television or internet video.’

The audiovisual content created today becomes the archive of tomorrow, but keeping on top of what to keep – especially outside film, television and news – is tricky. The Selection and Acquisition Policy (SAP) is a guide for collecting. ‘It is a massive help because otherwise we might want to collect everything – and that’s an impossible task.’

Ackroyd’s job also involves maintaining relationships with the screen production industry. Recent acquisitions include the Loading Docs series of short documentaries and The Stories of Waitara. The latter came about because Ackroyd had worked with the producers on their previous series The Stories of Ruapekapeka. Both Stories and Loading Docs are shared and streamed online and are well supported by funders like NZ On Air. But what happens to low budget independent productions? ‘Without a concerted effort from institutions like Ngā Taonga, productions like these are arguably at risk of being lost in the future because they are not supported by established broadcast or post production film infrastructures.’

‘There are many different approaches not just to collecting but looking after digital materials,’ explains Ackroyd. ‘I am in frequent discussion with colleagues inside the organisation and people in the industry because this field is in a state of flux.’

Ngā Taonga, and Ackroyd in particular, is thinking about archiving in a way that some commercial providers aren’t. ‘We reach out to content makers or production houses to suggest that they might like to deposit their titles with us. Often they are delighted to be asked and hadn’t thought about it before. A different kind of reaction might be, “Oh, why would you want to collect this as it’s on YouTube?” I respond: “Well, how long is YouTube going to be around?” It’s a commercial operation and when it comes to an end – which it will, just as MySpace did – what happens to that archive and who’s responsible for it?’

Ackroyd works his way through the hard drive and runs a virus scan to protect the Ngā Taonga networks. Once cleared, the files are transferred to a working folder and securely backed up to two separate locations. The hard drive is kept until the deposit is brought fully into the collection and the entire accession is done – this occurs once the files are in the digital vault. ‘In the archive, film goes to a physical vault with controlled conditions. Similarly, born-digital content goes into our digital vault which we call Kohinga. It’s part of a storage area network (SAN) across two different sites and comprises servers, hard drives and LTO (linear tape open) archival data tape.’

Digital Collection Team Leader Tom Ackroyd. Photo by David Klein.

It’s important that the files remain true to the original. The ICT department at Ngā Taonga has helped develop a ‘checksum’ process. ICT Operations Manager Gary Jarvis explains: ‘When we copy data on and off tapes or drives, we need to make sure that after each copy process every single bit, every ‘1’ and ‘0’, is exactly the same as it started.’ This ensures that the file remains a faithful clone of what was deposited with us.

The aforementioned fire hose of digital audiovisual content has come about through technology changes that have hugely who can tell stories on screen. The costs associated with moving image production have decreased immensely over time, from early film to hand-held cameras to videotape to smartphones and affordable digital cameras. ‘Most people have the ability to immediately pull a computer out of their pocket and shoot cinema grade video,’ reflects Ackroyd. This can then be edited and shared online. This means that almost everyone can tell their own story, not just the likes of Hollywood studios.

Some of the ease of this creation and access can lead to problems however. You can hold up a piece of well-preserved film and see the image decades later; digital equipment and files have upgrade cycles. Occasionally a supplied drive is broken or files are corrupted and Ackroyd will have to contact the depositor. Out of the hundreds of hard drives received over the years, only twice has a drive come in with a virus or some malware on it – these get sent back.

The issue with technology upgrade cycles is further compounded by file format compatibility. The formats for producing audiovisual material change over time, compression and data storage technologies change and different video file codecs become fashionable or are pushed by business interests. ‘As an archival institution we want to make sure that the material we preserve has the potential to last longer than proprietary formats that come and go in a commercial environment,’ Ackroyd states. Digital material is converted to a format that is better suited for long term storage and maximum usability. ‘There’s an ongoing and exciting discussion across the globe about how best to preserve digital formats, especially when you consider there have been thousands of different codecs in use over time.’

Archives need to make sure that those files can be played far into the future. The solution is often a combination of transcoding and open source formats. These formats mean that no proprietary software or hardware is required for it to play. ‘Open-source communities such as the FFMPEG project are of immense value to archival institutions,’ Ackroyd says. ‘The people involved might work for archives. They have an eye on exactly the problem of making sure that things last into the future. Relying on commercial companies to provide those solutions can be risky even while they’re operating.’

With the files safely stored in the digital vault, Ackroyd can then work on describing the title. Colleagues will move the recording through a workflow until it finds its way to the online catalogue. Born-digital material is the present and future, but also the past. Like any audiovisual recording, it captures the people who made it and tells a story. That story has now become increasingly democratised by technology. Archives want to collect ‘new’ material, because at each moment in time it describes how we lived our lives. However it’s made, when it is safely backed up in a digital vault, it will be able to do that.

Ngā Taonga newsletter

Sign up for our regular newsletter about projects we're working on and fascinating items from our collections.