According to a survey of senior managers, data seems to be spiralling out of control and is having a major impact on organisational decision making. Is Big Data making it more difficult or is it a blessing in disguise?
Whichever way you look at it, the Big Data movement is getting bigger and bigger and is giving raise to runaway growth of the NoSQL community of software developers who work with semi-structured or unstructured data and non-relational data stores that fall into the following categories. (One way or another they employ Map/Reduce parallel programming framework to process and query the data.)
- Multidimensional map store - Each record maps a row name, a column name, and a time stamp to a value. (e.g. Google’s BigTable, HBASE and Cassandra).
- Key-value store - Each record consists of a key or unique identifier, mapped to one or more values. Redis falls into this category.
- Graph store - Each record consists of elements that together form a graph. Graphs depict relationships. E.g. network of transport routes that show shortest path to destination. Neo4j and InfoGrid are prominent in graph store.
- Document store - Each record consists of a document such as, Invoice in XML format. CouchDB and MongoDB come up frequently when discussing Document store.
One of the hallmarks of the NoSQL movement is that they all have non-relational distributed data stores that are designed for fault tolerance and based on the principles of BASE (Basically Available, Soft State, Eventually Consistent) and CAP (Consistency, Availability, Partition Tolerance)instead of the traditional ACID (Atomicity, Consistency, Isolation, Durability) properties of the relational database systems (RDBMS).
Another defining characteristic of the NoSQL movement is schema-less or flexible schema architecture. Above all, many of them lack the concept of ‘Transaction’ and data integrity! Also, the methodology for software design seems to be a bottom-up approach.
For me, who advanced my profession in the traditional structured approach to business system development using the principles of Information Engineering (IE), all this is quite unsettling! What is more, the data scientists seem to wield too much power in deciding the ‘schema’ on the fly as well as the business process and software functions all at the same time when writing a piece of code! Don’t even bother mentioning Master Data Management (MDM)!
Although the NoSQL community is divided in their opinion on the virtue of data modelling for NoSQL, they do provide some comfort by emphasising that NoSQL means ‘Not Only SQL’ and does not imply no more SQL! All that is well and good! But, how does the Big Data / NoSQL fit into RDBMS-world, the Entity-Relationship data modelling and development methodology? What is the implication for business planning and information system design?
Let’s take a brief look at the traditional information design approach and review where the Big Data / NoSQL may fit. Information engineering (IE) serves many purposes, including organisation planning, business process re-engineering, application development, information systems planning and systems re-engineering.
As a top-down methodology, IE uses facilitated modelling sessions with senior business managers that help to review the strategic business plans and to develop a strategic enterprise data, function and process models independent of each other but nevertheless in an integrated fashion. It results in reusable processes for rapid delivery into production as integrated databases and reusable systems.
A key aspect of information strategy planning is the design of a logical data model that represents data and their relationship to the business. It is based on entity-relationship mapping that conforms to the principles of relational model and normal form which helps to reduce redundancy and ensures integrity of the database. The logical data model serves as a blueprint for the enterprise data warehouse (EDW) which is implemented in a RDBMS that collects the enterprise data and makes it usable (timely, easy to manage, available, etc). Then the business can gain value from the data time and time again by asking many different business questions. Business managers typically supplement with data from external sources, such as market surveys, and integrate them into the data in the EDW. This allows them to enhance and enrich the data that is already familiar to the enterprise.
With the proliferation of online data and large-scale adoption of social media, business managers are beginning to find a wealth of data in market - sentiments about their products and competitors in the web servers and log files. Big Data / NoSQL can enable agile analytical capabilities on semi-structured data not seen in traditional information engineering which is great news! The managers no longer need to fully rely on the traditional random surveys. Instead, they now have the ability to use Big Data / NoSQL tools to more accurately explore and analyse such real-time market data, then integrate and operationalise them in the EDW while at the same time relying on the stability, maturity, scalability and availability of the RDBMS.
What will information engineering look like in the new age of information management? It will likely leverage the best of both worlds: RDBMS and NoSQL, as shown in the concept diagram below. Interestingly, Microsoft Researchers Erik Meijer and Gavin Bierman have proposed a model for co-existence of SQL and NoSQL which they call ‘Co-SQL’ that is based on Category Theory - which I am still learning! Erik Meijer and Gavin Bierman argue that Big Data / NoSQL is now in the same state of disarray as where relational movement was 30+ years before E.F. Codd was instrumental in standardising it by means of his relational theory. It is interesting times for business intelligence and analytics!
The post Is Big Data turning Information Engineering upside down? appeared first on International Blog.
