Skip to content
Logo Investereum
Hit enter to search or ESC to close

From dark data to usable archiving

You have probably heard of Big Data and how it’s used to feed machine intelligence or pattern recognition in stock exchanges or weather forecasts. But do you know what ‘dark data’ is? Simply put, it’s all data that lies around unused, useless or unusable.

For instance, a company may make the smart decision to go from paper archives to digital archives. They scan in all documents and bundle them in a big image library, only for those files to remain unread, unfindable by search engines and not structured in a manner that other programs (or even people) can freely use them.

Fishing in the data lake

IBM estimates that about 90% of all of the world’s data is, in fact, dark data. One possible solution comes from data lakes: vast reservoirs of structured, unstructured and dark data where specialized software packages can trawl through and make the most of this mass of data.

While this is useful for the big fish who have the resources and capacity to devote to building data lakes, for most companies or organizations, this takes too much effort. And even for the big ones, it might be smarter to change their archiving processes altogether. That’s where blockchain comes in.

Cloud to paper

If you use blockchain technology for archiving, you’ll have the benefit of verification: the blockchain will verify whether the new block really belongs there or not and matches previous records. Its distributed and encrypted nature will also make it very hard for one instance to tamper or destroy your records. You can’t set fire to the cloud.

This in itself doesn’t solve the problem of dark data, but at least it’s a step up to add more structure and regularity in your archiving process. Other technologies can help, too.

The smart reader

We should remind ourselves that transitioning to the paperless office isn’t merely about saving an image of all your documents (and since “the paperless office” always seems to be on the horizon but never quite the reality, it’s a safe bet to assume some form of paper will always remain).

Two technologies have evolved quite a bit over the last decade: OCR and PDF. OCR stands for “optical character recognition”, and is used to turn analog text into digital text readable by most text processing software. PDF, or “portable document file” allows people to add metadata and structure so that they can be easily found by search engines.

One archive, one answer

By employing OCR, PDF and blockchain in tandem, you can build an impressive archive of records that is easily accessible, readable and usable by other smart software. For instance, you can let an algorithm go to town on your archives to calculate the most likely month that your customers will end up missing a payment. Or, your archives can tell you when your employees are most likely to fall sick.

The longer your archiving blockchain becomes, the more impressive it will be as a vehicle for archiving and research needs. Adding reliable PDF generation to the mix – which, by the way, you can also use for watermarking and signatures – gives you a clear framework for archiving that can be learned and used by all admins and requires the least amount of proprietary busywork.

The healthcare proposition

One industry where archiving is especially important is healthcare. Being able to track a patient’s health issues over the years is a boon to any hospital or doctor’s office, and while digitalization is already helping healthcare professionals in getting a more accurate picture of the patients they serve, blockchain can up the ante.

By adding encrypted new records to each patient’s personal blockchain, not only will new data be verified, they will automatically be more secure as well as accessible to the various doctors and specialists that the patient gets in touch with. In addition, it also removes the need for the patient to explain their health history to each new specialist.

Oof. This all still sounds a bit hard?

If it does, that’s because the first steps are never easy. But that’s why we are here to help. This and other blockchain applications can be very beneficial for organizations of any size, and we’re thrilled to help them discover the benefits. Give us a shout if you’d like to discuss your challenges and ideas with us – no strings attached.

Verified on

March 29th 2019, 08:27

Categorized: General