Real-time insights on a MASSIVE scale

Real-time insights on a MASSIVE scale

From delivering music on Spotify, to delivering news at the Daily Telegraph, Google Cloud Platform is already playing a major role in helping companies get their products into the hands of consumers. Recently, we’ve been helping a top 5 UK supermarket to realise the vast potential of GCP with real-time data processing.

A common story with an innovative twist!

To tell this story, we need to illustrate the business problems that our client was experiencing which led them to our door. They already had a legacy system which stored data collected from their tills; item numbers, timestamps, loyalty card details, promotional discounts, to name just a few. As you can imagine, this data provides a wealth of critical insights for a supermarket.

The supermarket's rapid growth over the last few years had resulted in them outgrowing this legacy system, which could not be adapted or scaled to meet their need; to access data as quickly as possible in order to drive increased sales. Data relating to sales events are logged as transactions occur, but the legacy system would not allow this data to be accessed until the following day - to keep pace with their own growth they now needed to surface this information in real-time.

Identifying this as a problem, the Head of Technology approached Appsbroker to see if a proof-of-concept could show evidence that Google Cloud Platform could enable real-time access to their data. To achieve this we quickly put together a solution which would push till log data into Google Cloud Storage every 15 minutes, working within existing polling times. We then created a service, called Sales Service, which would write this data into Google BigQuery, enabling analytics at either basket level or item level. This exercise proved that till log data could indeed be made accessible in real-time using GCP technology - this was the proof that was needed, and signified the thumbs up for the full project.

Moving from proof-of-concept to full solution

The full solution, which is currently being developed, is not a million miles from the original proof-of-concept, although it now utilises many more Google Cloud Platform technologies to ensure that the Sales Service is fast and robust:

  1. The till log data is uploaded to Google Cloud Storage in batch loads, in 15 minute intervals (as per their previous architecture)
  2. A bespoke Google Compute Engine application manages the orchestration, and sends outputs via Google Cloud Pub/Sub
  3. The Pub/Sub messages are consumed by Google Cloud Dataflow, which processes and enriches the data (from reference data stored in Google Bigtable) before it reaches Google BigQuery

The end-to-end process within Google Cloud Platform, as illustrated below, does not take more than a couple of minutes:

Big data - big benefits

From this data pipeline, Sales Service enables the supermarket to trigger alerts and build custom reports. The data is grouped into item and time period windows, and Google Dataflow can keep over 30,000 of those windows active at any given time. Currently our client will continue to send till log data every 15 minutes, but the system would be able to manage data at a per-basket level if they chose to get even closer to a real-time data pipeline.

Future proofing for a smarter future

Identifying, defining, and developing a solution like this is typical of the kind of projects that Appsbroker get involved with on behalf of our customers. For this particular engagement, our background in technical consultancy, combined with our Google Cloud technology expertise, has enabled us to surface key information from within the discovery phase, that we think will bring added value to our client in the future - for example, while the existing solution focuses solely on sales data extracted from tills, non-sales data such as operator behaviour, performance, and queue length are all potential extensions.

One of the greatest strengths of a widespread adoption of Google Cloud Platform across an entire business is the potential to integrate with other downstream systems and microservices - and we’ll be working with our client to realise that potential. While Spotify and the Telegraph group are using the power of GCP to put music and news in the hands of their consumers, our client can now leverage that same power to provide their whole marketplace to their growing customer base.

Jon King | Google Cloud Expert

Jon King | Google Cloud Expert

Introducing App Maker

Introducing App Maker

Google has announced a new feature to its enterprise users of G Suite, and Appsbroker is really pleased to be supporting them as a launch partner.

App Maker is a new component of G Suite, and offers customers access to a simple yet powerful application builder. It is the perfect extension to G Suite, and we think you’ll love it.

Smarter Working with G Suite

Smarter Working with G Suite

G Suite = Business transformation for all! Come and join the transformation challenge - you'll love it!