By Dr. Paul Terry
Big Data is best used as a tool to support mission-critical outcomes, but its value to enable curiosity-based research also triggers the “art of the possible”. Big Data allows users to store information without mapping it beforehand; it allows users to store data without limits on size or parameters on types; most importantly, Big Data allows users to ask numerous “what if” questions in a variety of scenarios. At PHEMI Health Systems, we’ve been using a very simple example with customers to illustrate the art of the possible.
My Canadian colleagues told me of a time when they used to take a roll of film to the local Blacks Photography store, wait eagerly for 5 days
to receive their envelope of 24 shiny photos and then use their acid-free tape to stick those photos permanently into the album, which they placed on the top shelf of the TV stand in the living room. Essentially, the photo structure was imposed at the outset and was permanently fixed. If they ever wanted to retrieve that one fabulous photo of baby Jimmy in the tub at 18 months or Aunt Hilde at the family reunion in 1996, they needed to find that specific album to visit that specific location on that specific page.
But what if they wanted to create an album featuring Jimmy from the age of two to six or Aunt Hilde on all of her various vacations? It would have taken immense fortitude. They would have had to spend hours going over each album to pull out photos and physically reconstruct new albums for each scenario. Not a workable solution – in terms of time, the removal of the “originals”, and the unnecessary creation of another “permanent” album that may again need to be deconstructed should the viewer want a different way of looking at the photos.
Fast forward to recent years with the advent of applications like Picasa and iPhoto that use facial recognition technology, tagging and other semantic-based tools. Budding photographers take as many photos as they like and the photos simply get stored. Picasa or iPhoto or some other tool runs the facial recognition and tagging, and suddenly a variety of filters or “queries” can be run on the photos. “Generate a view of Jimmy in the bathtub over the years” (not so cute at the age of 48, but the important thing is the possibility). “Create an album of Aunt Hilde’s vacations in tropical destinations.” The potential filters or views or “queries” are endless; there are multiple ways of looking at the photos and multiple relationships between the photos. No structure or rules were needed when collecting the photos – just the ability to store the photos (data), tools to provide identification (whether facial recognition, optical character recognition for text, or some other method of turning unstructured data into structured data) and some basic compute power to run the views (queries.)
Where the old photo album structure was imposed at the outset, in the new model, virtual albums can be computed on the fly and can be changed at any time.
Current SQL databases are similar to the old photo album; they work well when set up for the first time, but they fall over quickly as requirements evolve. Like the old photo albums, SQL data structure and schema is imposed at the outset, making it virtually impossible to obtain different views of the data if the request doesn’t fit the parameters of the original structure. It’s for this reason that a typical hospital today will have 200 or more SQL-based information systems, each purpose-built for it’s original intended use and then stuck in time with their limited query abilities.
Contrast this with Big Data where the structure is computed on-demand based on the researcher’s query. Big Data allows users to see their data in a way they never could before; it enables researchers to follow curiosity-based research as they retrieve, compute, and present data in limitless ways. Big Data enables the “art of the possible.”
Dr. Paul Terry is the President, CEO and “Big Data Guy” at PHEMI Health Systems. He serves on the board of directors for Providence Health Care, and for the past 11 years was also was the secretary/treasurer and technology adviser for the Michael Smith Foundation for Health Research – British Columbia’s health research support agency. Paul is a serial entrepreneur….