In-transit data analysis and distribution in a multi-cloud environment using cometcloud

Many applications require data to be captured and processed in real time, migrating all the data to a central site prior to analysis, a procedure that can create significant overhead. Examples include the variety of sensor network-based applications, where sensors interface with real world artifacts and must respond to physical phenomenon that cannot be predicted apriori. The amount of data likely to be generated by a sensor and processing requirements in such applications can not be pre-determined (as they are often dependent on the rate of change of the physical phenomenon being measured and potential occurrence of 'trigger events' which are non-deterministic). We propose the use of a multilayer Cloud infrastructure that distributes processing over both sensing nodes, multiple intermediate/gateways nodes and the more complex centralised data centre. Such layers need to work in coordination to ensure more reliable and efficient use of computing and network resources, preventing the need to move data to a central location when this is not necessary and creating data processing paths from data capture to analysis. We outline the basis for a decision function that evaluates: (i) where processing should be carried out, (ii) what processing should be undertaken centrally vs at an edge node, (iii) how processing can be distributed across multiple data centre locations to achieve QoS and cost targets. We present a prototype that has been implemented using the CometCloud system, deployed across three sites in the UK and the US and validate using an application which calculates energy flow in a Sports facility in Italy.