Recent developments in data protection law, and increased demands on creating responsive services, are combining to increase the burdens on data owners and processors. These rules will be backed up not just with teath, but the regulatory equivalent of tactical nuclear weapons. GDPR fines can be up to 4% of group turnover or more.
Data owners will be liabile for data loss even where the loss is due to the criminal actions of a third party, requiring greater data security. Analytic systems must therefore be able to analyse data even whilst it is still protected.
The purpose to which data may be put means that just because an organisation has a data item, it doesn’t mean that everyone in the organisation can see it, nor that it can be used for any purpose. Analytic sytems must have ways for meta-data to accompany data to describe who can see the results.
Data Life Cycle
The data life-cycle is becoming more complex. Traditionally data would be created, used, and deleted (with varying degrees of success). Now however, the data subject may wish to change, on the fly, the use to which it may be put. The organisation may wish to accommodate this, because it could give rise to further business. Or the data subject may wish to rescind permissions, and the organisation is obliged to comply. The meta-data to enable this will also have to be as complex and flexible as the data itself.
Data processing must be correct and shown to be accurate. Systems must be able to cope with error rates and confidence levels.
Decisions made on data may need to be explained and justified to management, if not to a regulator.
Velocity and Volume
The next generation of data analytic systems must be able to cope with these issues – whilst coping with ever greater data volumes and an enduring need for speed.