Panning for gold in your data stream

Data analytics have become part of the information lifeblood. Simple dashboards provide easy access to powerful business insights, almost in real time. Data, drawn from a variety of massive internal and external sources, reveals key performance indicators and other derived knowledge that plays an essential role in a company's strategic plans.

The technology has revolutionized how corporations are managed, and the amount of data that can be and is collected continues to increase. So using that data for a competitive advantage or even just to help focus the business will continue to be a critical task.
But what if you don't know what you're looking for?

Three kinds of information

One might argue that there are three classes of knowledge:
  • Information we know that we know
  • Information we know that we don't know
  • Information we don't know that we don't know
Traditionally, statisticians and analysts have worked from a data model mindset. You create a mathematical model that describes how a given input results in an observed output. You then test the model against data sets to determine a p-value that should tell you if the model is correct. If it doesn't accurately predict the results, you tweak the model - or throw it out and start over - until you find something that works.