Many IoT projects involve a large number of sensors, which in turn capture a large amount of data that have to be managed, processed and analysed to come to useful conclusions.
These large datasets are, not surprisingly, called big data, which don’t necessarily only have to be used in conjunction with IoT projects but in many cases are.
When you start thinking about an IoT-based project and depending on its type and targets, at some point you will have to consider using big data analysis tools. Furthermore, if you are in a position to work with a large amount of data for analysis, the next thing you will need to consider is which roadmap you should follow.
Below are some guidelines so that you can easily navigate your way around drawing up an IoT-related big data project.
The first choice: which data
The first step is to separate the wheat from the chaff, to identify the data which will be useful compared to others that will be useless and therefore you won’t need to process, or may even have to avoid generating or gathering.
The best way of finding out which data you need to work with is to use case studies as examples, to imagine the project’s possibilities and what you are going to use it for. When doing this you can often uncover new usage options and so it can help you to define and/or improve your project.
Of course, you also need to think about data which have not yet been produced by the planned infrastructure and which might be relevant. This will enable you to plan for the systems which are to gather these data.
In this part you can take into account external data sources, i.e. any which will not be generated in your organisation’s own infrastructure and systems but which will be used in the analyses leading to conclusions and recommendations.
A perfect example of this external data use would be a car park project where you could tap sensor data drawn from a number of dealerships.
If you already know what data you are going to gather, it’s time to consider which structures you will use to collect and store it while bearing in mind which of these data are structured and which are not.
You will have to decide whether gathering these data and analysing them is worthwhile. This process carries costs for infrastructure, communications, equipment, software, analysts and so on, and if they go over the result achieved perhaps it’s time to rethink the global model or that part in particular.
Transformation possibilities
While the previous step shows you what you need to attain your goal, what you have to do in this stage is to use your imagination to grasp where and to what extent the results of the big data analysis can transform your organisation or the project itself.
Creating an operating model
Whether for a customer-facing business, in retail or wholesale, or for operations funded by public institutions, you will have to draw up a quantifiable and measurable business case or operating model so as to weigh up implementing the big data model or reframing it (or even cancelling it if need be).
In this stage you will have to take into account the systems required and whether the organisation can meet the project’s needs with its own resources, whether external resources have to be used (such as a public or hybrid cloud), whether the organisation’s staff can implement it or whether an external entity such as a consultant should be called in.
Finally: testing and implementation
Once you have the right model after analysing all the needs, variables, resources and systems, there will only be one step left before final implementation: real testing of the model.
The idea is to reproduce the model created on a small scale but in a working environment identical to the production one so that it is subjected to equivalent workloads. This will be the stage where you can spot design faults and fix them by fine-tuning based on the feedback from the testing and then scale up the design.
Once the model has been honed following the testing, it will be time to implement it on an organisation-wide scale.
There will be numerous companies specialising in big data at the IoT Solutions World Congress to be held in Barcelona from 29-31 October. They range from providers of services and products such as management and analysis software to consulting firms who can take over the entire cycle of this kind of project from inception to completion.