Culture

Digital archives in an algorithmic world

It’s on a hard drive. It’s in the cloud. And it’s so, so very fragile.

With phones in hand from dawn to dusk, humanity walks with one foot firmly planted in the digital world. Like footprints in sand, the record of its journey may be in peril.

At the turn of the millennium, the words “the internet is forever” seemed both a promise and a warning. But sites hosting files collapse, or prune projects that no longer exist elsewhere, and articles fold quietly into nothingness as companies change hands or put up paywalls.

In 2014, the Harvard Law Review published an article stating half of the websites cited in US Supreme Court opinions no longer hold the information they were cited for. Eight years later, the Pew Research Center reported about 40% of website links from 2013 no longer go anywhere. Digital history disappears.

Furthermore, the surface area of the internet may be shrinking. Paul McCarthy and Marian-Andrei Rizoiu in The Conversation suggest the majority of attention on major social media platforms, like Reddit and Twitter (X), is focused towards just ten domains, and visibility of any page outside the 1000 most populous ones is in decline.

The internet is also becoming less openly visible. Since 2021, Facebook requires visitors to log in with an account to view posts at all, with Twitter following suit in 2023. Questions and answers once posted to searchable web forums now live quietly out of sight and search in chatrooms like Discord, and the body of public knowledge shrinks with it.

Acquiescing to the walled garden has its own costs. Users have vanishingly little control over the data they’ve shared or how they’re tracked. In this way, the digital record is frighteningly permanent, and leveraged in ways that harm its most vulnerable users.

Outside the login thresholds of the social media giants, exposed websites are experiencing a new deluge of traffic, but none of it is human. In July 2026, Forbes reported that Cloudflare, which routes and connects significant portions of the internet’s traffic, was receiving more web traffic from AI bots than from humans 18 months sooner than anticipated. By October, that percentage approached two thirds.

A graph shows bot traffic peaked at 71 per cent on 12 Oct 2026. Average bot traffic from Oct 8-14 is 65.2 per cent.
On October 12, 71 per cent of internet traffic through Cloudflare came from bots. Image sampled from Cloudflare Radar under CC BY 4.0.


Smaller web forums that ordinarily see a few hundred visitors per day may suddenly see tens of thousands of them, all bots scouring every post and link to feed to language models and driving up the cost to host a website.

A webforum statistics panel shows 337 users online today, and most online ever on 21 April at 102,791.
Only ten new accounts registered on April 21 2026, despite traffic increasing by orders of magnitude. Image taken from the Simple Machines Community Forum.

In one particularly incensed blog post, a San Francisco nightclub, the DNA Lounge, said its monthly website hosting costs had quintupled from around $A1000 to $A5000, courtesy of the incredible deluge of AI bots scraping it for training.

Other sites simply stop seeing traffic. Once they’re crawled, AI summaries replace page visits. Websites subsisting on advertising revenue collapse, taking articles, images, and craft along with them, potentially forever.

The internet remains a young technology, and a vulnerable one at that. Artists, writers, histories, creators, languages, cultures, all have bloomed its wake, but the record of its rapid transformation may wither and vanish as the trust placed in the technology dissolves alongside so many dead websites.

What gets saved, and who decides?

A hard drive leans against brickwork.
Records get left behind as technology moves and evolves.

It’s not the first time a record has been imperilled. Even before the internet, the rapid development of new media formats– magnetic tape, floppy drive, optical disc, compact disk, and so on– produced new challenges to keeping records as the technologies and means of using them were replaced. They won’t last forever on a shelf, either- hard drives and disks slowly disintegrate much like magnetic tapes do in a process called bit rot.

Dr James Doig works in the Digital Archives Preservation and Research department of the National Archives of Australia and knows firsthand the challenge of managing a digital record. His mantra: active ongoing management.

“Digital information is so fragile,” he says. “If you’re not actively doing something, then you’re going to lose it. You can’t leave something in a digital format and expect that you’re going to be able to access it in five years’ time. You won’t be able to.”

He recounts the example of the Domesday Book, a survey of England made in 1086. The British Broadcasting Corporation made a massive undertaking in the 1980s to digitise the book onto LaserDiscs for sale to education institutions. Within fifteen years, the discs were obsolete: discs were scratched, readers were failing, and the software tying the system together was largely incompatible with current-day computers.

The original manuscript, meanwhile, still survives in the National Archives of London, 940 years after it was written.

Much of the data borne of the internet has no such physical copy or national archive to lovingly tend it, however. As for the government records Doig keeps, he estimates it’s only 3-5 per cent of them.

A hard drive nestled into foliage.
Data that is not maintained regularly is at risk of loss.

Despite the pace of change, digital cultures thrive and collapse within temporary technologies. Recording the history of that technology may contextualise other cultures, technologies, or even the formative years of entire industries.

Take Adobe Flash, for example.

Matthew Lucis, an independent game developer, is working with the National Film and Sound Archives of Australia to save and preserve the creative works of Australians built with Flash. Flash was free education software that many people had access to, and early developers and hobbyists latched onto it to create games and animations well before better tools were available to the public.

And then Flash died.

Apple’s chief executive officer, Steve Jobs, put out his Thoughts on Flash letter in 2010– the infant years of the iPhone– and support for Flash on the burgeoning mobile device market ended a year later. Its use waned until full deprecation in 2020, and perhaps for good reason. Flash was chronically vulnerable to hacks and computer viruses, ran poorly, and was proprietary.

Still, hobbyists had no say about its end and limited recourse to save or archive creations built with it.

That didn’t stop the community. The website Newgrounds built a tool to run the now-discontinued software it hosted, and another Australian enthusiast “BlueMaxima” built the downloadable archive Flashpoint to preserve Flash software across the wider internet. They stepped into to preserve their work when companies walked away from it.

It may not sound important, but those archived Flash programs reflect the infancy of an indie games market that WorldMetrics estimates to be worth $A68.6 billion by 2027. Lucis says simply, “Recording why Flash is important is recording why indie games are important. We need to remember where we’ve come from at each stop of the way.”

Lucis has had some success procuring programs for the NFSA, but the development files– the code, project notes, the wit and spirit of how they were built– are frequently painfully absent.

“This is a thing that is going to happen to pretty much everything. The death & crash of Flash is unfortunate, but every media will suffer this.”

A scuffed hard drive rests beneath vines.
Technologies inevitably die out.

Whether or not consumers ought to be responsible for archiving creative works is another matter. A single hobbyist’s archive may be a singular point of failure. If one hobbyist has championed the the task of preservation, it becomes one archive that can be eradicated with a hard drive failure, computer virus, or simple natural disaster like flooding.

Additionally, an archive is fundamentally about both the preservation and sharing of information, which is complicated by soaring hosting costs and copyright challenges. One hobby archivist is unlikely to have the resources to manage.

Dr sky croeser [sic] is a lecturer in Curtin University’s Internet Studies department. To them, it’s really a question about community.

“Maybe it’s not about you as an individual having this responsibility to archive. Maybe it’s about thinking about how you do that as a community, how you create an archive that isn’t just static but is able to be accessed by people in important ways,” they say.

A profile photo of Dr sky croeser in front of a bookshelf.
Dr sky croeser studies how people seek agency with the technologies they use.

The task of archiving media also assumes the files exist in the first place.

A 2026 report from the Australian Communications and Media Authority showed the percentage of Australians streaming from media services nearly tripled to 68 per cent since 2017. Streaming trades ownership for convenience, and without ownership, viewers may have no recourse when their chosen streaming service removes media. Sometimes there is no legal way to watch a movie or documentary lost between licensing agreements.

Worse, viewers that have actually purchased digital media may find it suddenly excised from their libraries when those licensing agreements pass. On September 1 2026, Sony removed 551 different movies and shows from United Kingdom residents’ accounts when a licensing agreement with StudioCanal ended. This mirrors a similar Sony snafu in 2024 when users’ “forever” libraries with Funimation were eradicated in a merger with streaming service Crunchyroll.

Digital ownership is as ephemeral as digital storage.

For smaller artists and creators, simply sharing more is no true answer, either. Publishing on social media giants or just their own websites gives them little leverage to protect their work or decide how it’s used. Facebook, Instagram, Twitter, Twitch, Tumblr, all by default feed user content to language learning models even if it’s not publicly available for the heaving mass of bot traffic to scour.

It’s also unknowable if deleted material is truly gone, or if it’s actually excluded from AI training. “’The cloud’ is just somebody else’s computer,” croeser warns. “They might do things with your data that you weren’t expecting them to.”

A CD leans against the scuffed hard drive upon brickwork. The many puncture marks in the disc resemble the sparkle emoji visible in the center of the disc.
Most people have limited control over how AI processes their data.

It’s an imbalance of power, and one that tilts further away from people with the vast and valuable troves of personal information companies harvest from their users, offered willingly or otherwise. Browsing habits, shopping choices, search history, social circles, work location, home address, every comment made in anger and every heartfelt message, all get squirrelled away into behemoth corporate archives.

According to croeser, to archive is to decide what data matters. As companies draw more disparate data together and decide, well, all of it matters, the purpose of that archive must be questioned.

“What is it for?” they ask. “What is the story that you’re trying to tell? Who is your audience and who are you gathering data on or whose whose story are you telling?”

For companies like Meta and Google, the story buried in those massive archives is for marketing to an audience of shareholders. It’s also shaping “the algorithm” to keep users engaged, no matter the cost to their well-being. For others, like Palantir, it’s for matching addresses to “potential deportation targets” in America’s ongoing pogroms against immigrants.

And for generative AI, it’s for producing text, images, and audio that’s increasingly indistinguishable from reality. Pink-slime journalism fabricates local news for towns without a local paper, and deepfake images and videos make scams convincing and sway elections. It dictates reality in the absence of its original sources and rewrites history in the graveyard of an internet it is actively choking out.

A scuffed hard drive leans against stonework. An ethernet cable is wrapped around it. One end of the cable is severed.
As websites are forced offline under the incredible weight of AI scrapers, data may be lost forever.

It’s not all hopeless, of course.

Communities like those around Flash, giants like archive.org, special interest communities like the Archiveteam Wiki or Lost Media Wiki, or boards like Reddit’s DataHoarder subreddit, all work to preserve whatever they can. People like Lucis are speaking to their communities and industries on how better to preserve their works.

Perhaps, in thirty years, hobby archivists will emerge with their arrays of hard drives and restore slivers of a public record long since replaced by newer technologies, much as movie director Martin Scorsese did with his thousands of VHS tapes of recorded television. The ephemera of today may get recast as history long after it was thought lost.

Ironically, a healthier internet might embrace some of the gatekeeping smothering it now. croeser says smaller federated networks already exist in places like Mastodon, where users have more control over what they share, how long it exists, and who sees it. They imagine an internet where the physical and environmental costs to maintain it are realised, free from the digital arms race of data centres and AI scrapers.

“It’s very hard to imagine a world where we have a better internet, a better economic system, a better democracy,” they say. “But I think it’s also really useful to keep that imagination alive and keep thinking, well, what do we want it to look like?”

“And I think many people would enjoy an internet that isn’t about infinite scrolling, infinite content (much of it untrustworthy), increasingly hot summers, increasing flooding and fires, but is about really connecting with each other, where we have both privacy and the ability to share what we want to with each other and with the world, and to keep the data that we think is important.”

“And, at the same time, have limits on what the internet does.”

The cloud is just somebody else’s computer, after all. Perhaps it’s time to re-evaluate what’s entrusted to it.