The 11 Commandments of Business Driven Data Analytics

There are some basic and well-recognized principles involved in business-driven data analytics, which we'll look at as a set of "The Eleven Commandments".

1. You Should Not Violate Security Policy

   The first commandment of ensuring your big data policies support your business is that you must be rigorous about your security policy – developing it, training in it, maintaining it, obeying it. This principle seems obvious, but between the increasing sophistication of attackers and normal business events like employee turnover, security policy often does not keep up with an organization's needs.

   Chances are that your company has a security plan in place that was selected for its technical features. Your business intelligence tools should be able to take advantage of that security already in place. Many Big Data management systems like Ranger, Sentry, and Mongo come with security modules that can make it easier, to implement a unified security system. These modules enable you to plug right in to existing architecture so that the whole system right up to the application layer has the same user information and allows access the same way, at the same level, to the same users.

2. You Should Not Trust Invalidated Data

   An important factor in a security policy is validation and verification of data. Your business intelligence is only as good as the data you use. Data should be validated on input, both to ensure that the data is real and relevant from the start, and monitored continuously to confirm that there hasn't been a security breach that might have affected data quality. In addition, it's important to ensure that both data and user information remain consistent across systems.

3. You Should Not Move Big Data Around

   With smaller data sets, we were always moving data around – extracting it from one data mart and entering it into other data stores. Big Data, which might more appropriately be called "Massive Data" or "Gigantic Data", is much more expensive and complicated to move, not to mention that data is always at risk during movement, so violating this principle also means violating the security commandment! Applications that provide insight into data for business intelligence should not move data around; instead, BI apps should access the data where it is. This enables a better degree of data discovery, which enables a higher degree of predictive intelligence.

4. You Should Not Pay by the Gigabyte

   When dealing with Big Data, some of the popular cost models of the past must be abandoned. Big Data can be very cost-effective, but not if your cost model includes having your service provider charge by the gigabyte, or by gigabytes indexed, or by number of users. Even if it seems like a good deal when you sign up, when organizations start collecting and working with Big Data, their data stores can grow from ten billion entries to hundreds of billions in only months, while the user base is also expanding exponentially. By staying away from these cost models, you ensure your organization's future scalability.

5. You Should Analyze thy Data in its Natural Form

   Although Big Data is often seen as unstructured, that's an oversimplification. While data sources are diverse, and incoming data unpredictable, modern BI tools can help make sense of your data without forcing it into tables that may incorrectly describe the data. The JSON file format is one way of allowing data to stay in its original format while applying structure in alternative ways – that is, by descriptors that allow data to be "semi-structured" or "multi-structured" – concepts that allow data to be worked with and processed without having to force it into a table, where it is in effect flattened, so that only one view can be had of it. This flattening and loss of dimensionality can eliminate much of the BI value and should be avoided.

6. You Should Implement Interactive Data Visualizations

   Understanding the difference in value between sharing static data and having interactive data visualizations is critical. Long-standing business habits encourage us to access data, create a static chart, graph, or table, and pass it around. Two years later, chances are that the same charts, graphs, and tables are still making the rounds, but by now they have lost their meaning, and are little more than pictures of a certain point in time. Build your BI apps to be interactive and updatable, so you get as close to a real-time visualization as possible. If the data you're representing is important, then it is only important if it is current and accurate.

   Having the data visualizations receiving a continuous data stream is an important factor in being able to recognize a piece of business intelligence and act on it. If the data stream is not continuous, your data scientists will find it harder to identify data gaps and apply correction algorithms and interpolation procedures. All of this overhead makes business intelligence harder to acquire and less reliable when you do.

7. You Should Build Apps, not Reports

   For decades, when we wanted to access data, we would ask for a report – from a data analyst, from an application, from our vendor. The application or analyst would go into a data store and run a query to pull a static report, and there was our data. But in Big Data, business intelligence is gained by compiling asynchronous data from multiple sources, and is most effective when new data automatically updates the visualized data. In addition, users want to be able to interact with and manipulate the data provided so they can visualize it in different ways to gain different types of business insights. Modern web applications provide a model for this kind of interactivity, and web frameworks should be adapted to provide similar benefits to BI applications.

8. You Should Use Intelligent Tools

   BI tools have grown in sophistication, and by choosing the right tool, you can increase your organization's business intelligence with updatable, automatically maintained data visualization mechanisms, intelligent modeling and caching operations, and advanced search capabilities.

   An important feature of intelligent BI tools is the capability of making data discoverable. While part of that is accomplished by working with data without forcing it into tables, the other half of the equation is BI tools that help discover the data that's pertinent to a business goal. BI teams don't always know what data is available or how to find it, so an enormous support for business-driven analytics is a robust discovery mechanism that can be put in place on top of a centralized data catalogue. A standard metadata scheme, rigorously applied to incoming data, will increase the discoverability of the data. Such metadata can include such values as time granularity, spatial granularity, usage statistics, source systems or application, frequency of updates, and so on. Documenting linkages between data sources is also recommended.

9. You Should Get Fast Results

   One benefit of working with data without transforming into tables is performance. Even when we're dealing with data on a massive scale, in this day and age we expect things to be fast. Old models, such as moving data into OLAP cubes, a kind of pre-computed cache, to get good performance, are not scalable to Big Data stores. Moving the data, as we've already discussed, is expensive and dangerous. In addition, with the size of the caches that you can expect to see with big data, a laptop or desktop computer can easily become overwhelmed. The problem is especially acute if new data is incoming during an operation. Sampling is not a good solution, because with the diversity of data, sampling is likely to miss a data type or something else important, thus preventing you from getting a good look at the big picture. Instead, you want flexible, interactive BI tools that work quickly, efficiently, and with all the data in place.

   An important reason that why real time operations and speed is critical? Especially when working with Big Data is that the one major goal of business intelligence is to be able to be agile – to move quickly to respond to market factors. The faster you are able to respond, the better you can take advantage of a business opportunity. The faster you can access and analyze your data, the faster you are able to respond.

10. You Should Try New Things

    The Big Data systems of today are geared toward providing a level of predictive analysis previously unavailable. Data processing tools make use of correlation, forecasting, and other techniques to bring sophisticated analytics into the hands of business users. The best BI applications recognize that not everyone in an organization who needs to work with data should have to become a data scientist or programmer. Instead, they allow business users to realize their goals by providing tools available within the framework. These tools allow different visualizations of data, simple methods to filter and image it and analysis paradigms that can be applied to an organization's particular needs.

    No organization gets everything right on the first try. Having a lab environment where data visualization and modeling can take place can be of enormous help in getting the most business intelligence out of your data. Your BI tools should be able to make re-visualize and manipulate views, which business owners and data scientists can collaborate on to ensure the best possible use of the organization's data store.

    Your IT and BI tools are all part of an ecosystem made up of many components that are constantly interacting. They collect data and feed it to BI systems to enable real-time decision-making as a primary function. Because of this, you need to ensure connectivity between all the components in your system. Your lab environment will also help ensure that any components in development can be tested to make sure they work well with the rest of the ecosystem before rollout.

11. You Should be an Active Participant in your Data Management

    Distancing yourself from your data and leaving all the work to your data scientists is not an effective option for dealing with the challenges of processing and analyzing Big Data. In order for the data to provide value through business intelligence, business people and not only data and IT people need to be involved in defining requirements and finding solutions. When business stakeholders and data scientists work together in a lab environment, solutions are much more aligned with requirements.

    Being an active participant also means establishing standards that are organization-wide. For instance, data taxonomies for business intelligence should be applicable to the whole company, and should not be developed department-by-department. This taxonomy, which describes data categorically and hierarchically, can establish organizational standards for product names, category names and parameters, metadata standards, and more.

    Having your IT and BI people as active participants ensures that software development standards are applied to data applications that meet business requirements as well. IT people are skilled at aspects of data management that turn out to be equally important to business-driven analytics, such as version control, continuous integration, standards for code documentation, tools for project management and team collaboration, standard frameworks, and so on.

    By engaging business stakeholders in areas that we used to treat as traditionally the domain of Information Technology, you are better able to drive your data analysis in the direction of business needs. Data is not gathered for collection's sake, but rather to accomplish a business goal: identify market trends, assess organizational reputation, identify pain points for customers, and find new markets and business models.

    Conclusion: