What You Need To Know About Dark Data

Tom Taulli Tom Taulli
November 4, 2019 Big Data, Cloud & DevOps

multiple layers binary code background. number one and zero in pattern moved upward. GETTY 5

The concept of dark data sounds ominous, even sinister. But it is very important in the technology world. “To make it more relatable, dark data is like all of the photos on your devices,” said Sky Cassidy, who is the CEO of MountainTop Data. “Most of them will never be used or even viewed again, but they are there. So as for dark data, it’s all the information companies collect in their regular business processes, don’t use, have no plans to use, but will never throw out. It’s web logs, visitor tracking data, surveillance footage, email correspondences from past employees, and so much more.”

For most companies, there is usually an enormous amount of dark data.  According to Rahul Telang, who is the professor of information systems at Carnegie Mellon University's Heinz College, its about 90% or so. 

While dark data may never be used or be useful for many organizations, its something that should not be ignored. “One key area of challenge is to manage dark data in order to minimize risks and liabilities in information governance, that is, regulatory compliance, litigation, records-keeping, privacy and records management,” said Kon Leong, who is the CEO of ZL Technologies. “Separately, while analytics is a very active area of focus for data managers, it is now critical that analytics and governance be addressed simultaneously going forward. New privacy laws, for example, now cover all data, including data in analytics repositories.”

But dark data should not just be about handling regulatory issues. This type of information may ultimately prove quite useful for deriving insights for managers. We are already seeing this with software that helps automate and streamline operations, such as with RPA (Robotic Process Automation).

“Cognitive automation is the answer,” said Prince Kohli, who is the CTO of Automation Anywhere. “By adding structure to unstructured content, cognitive RPA helps you automate invoice, purchase-order, and mortgage application processing—all of which rely on the dark data stored in documents, images, emails, and more. At Automation Anywhere, we believe that within five years, knowledge workers will be freed from the task of extracting information from unstructured content. They will then be empowered to do what they do best: make decisions, handle exceptions, and interact with customers, partners, and each other to advance business objectives.”

Then what are some of the best practices with dark data? What can be done to get the most value from it?

Well, a first step is to use data classification so as to get a sense of what you are working with. “We are seeing a number of vendors from the backup and recovery market segment come to market with solutions to help better do this and make it easier for re-use,” said Christophe Bertrand, who is a Senior Analyst at Enterprise Strategy Group (ESG). “Taking a holistic perspective is necessary, and leveraging existing processes that already move and manage data is critical.”

After this, you can look at what data has some potential and what is essentially unnecessary. “You can reorganize the data logically in a repository so employees can find the documents faster,” said Ilia Sotnikov, who is the VP of Product Management at Netwrix. It’s also a good idea to put together a data strategy as well as a governance policy.

“Working with dark data is not a one-time project,” said Sotnikov. “Over time, you need to improve it.”

  • Experfy Insights

    Top articles, research, podcasts, webinars and more delivered to you monthly.

  • Tom Taulli

    Tags
    Big Data & Technology
    Leave a Comment
    Next Post
    Pump up the Volumes: Data in Docker

    Pump up the Volumes: Data in Docker

    Leave a Reply Cancel reply

    Your email address will not be published. Required fields are marked *

    More in Big Data, Cloud & DevOps
    Big Data, Cloud & DevOps
    Cognitive Load Of Being On Call: 6 Tips To Address It

    If you’ve ever been on call, you’ve probably experienced the pain of being woken up at 4 a.m., unactionable alerts, alerts going to the wrong team, and other unfortunate events. But, there’s an aspect of being on call that is less talked about, but even more ubiquitous – the cognitive load. “Cognitive load” has perhaps

    5 MINUTES READ Continue Reading »
    Big Data, Cloud & DevOps
    How To Refine 360 Customer View With Next Generation Data Matching

    Knowing your customer in the digital age Want to know more about your customers? About their demographics, personal choices, and preferable buying journey? Who do you think is the best source for such insights? You’re right. The customer. But, in a fast-paced world, it is almost impossible to extract all relevant information about a customer

    4 MINUTES READ Continue Reading »
    Big Data, Cloud & DevOps
    3 Ways Businesses Can Use Cloud Computing To The Fullest

    Cloud computing is the anytime, anywhere delivery of IT services like compute, storage, networking, and application software over the internet to end-users. The underlying physical resources, as well as processes, are masked to the end-user, who accesses only the files and apps they want. Companies (usually) pay for only the cloud computing services they use,

    7 MINUTES READ Continue Reading »

    About Us

    Incubated in Harvard Innovation Lab, Experfy specializes in pipelining and deploying the world's best AI and engineering talent at breakneck speed, with exceptional focus on quality and compliance. Enterprises and governments also leverage our award-winning SaaS platform to build their own customized future of work solutions such as talent clouds.

    Join Us At

    Contact Us

    1700 West Park Drive, Suite 190
    Westborough, MA 01581

    Email: support@experfy.com

    Toll Free: (844) EXPERFY or
    (844) 397-3739

    © 2023, Experfy Inc. All rights reserved.