When a pioneer of modern industry founded a healthcare organization over 100 years ago, the idea that the organization would grow into one of the most pre-eminent non-profit healthcare organizations in the United States was likely not top-of-mind. The decades of industry-leading excellence in care allowed this regional health provider, with more than 1,200 physicians in over 40 specialties, cemented itself as a premier healthcare provider nationally. All 30,000 employees are dedicated to providing care, and that often includes finding innovative ways to deliver information and data, while minimizing costs and empowering their ambition of creating analytics around value-based care.
“Designing scalable data systems for the cloud is very different from traditional data warehouse projects,” continued CJ. “Although the tooling and the patterns are different, this doesn't mean that we have to rethink how data is engineered from the ground-up. Azure Data Services had plenty of entry points that felt familiar, or perhaps even less daunting than their current tooling. This offered the perfect opportunity for BlueGranite to help teach concurrently, as we were building, making the project collaborative, and fostering buy-in on all sides."
To begin, BlueGranite set out to understand the challenge at hand by better analyzing how the data intended for this platform was utilized by the organization. “Business analysts, researches, and a number of other parties all needed this data in a fast, easy-to-access manner,” recounts Eric Wozniak, VP of BlueGranite. “It was clear that their on-premises tooling was maintained and enhanced by talented engineers, but had reached a scale limit. Modernizing was critical to furthering the system-wide desire of fostering a healthy data estate."
With the challenge of creating truth from many disparate systems in mind, “the team was ready to see what new techniques and enhancements Azure had to offer,” recalls CJ. “It was apparent immediately how every new aspect of the Azure platform was opening doorways for faster development and tighter integration. It was thrilling to see how quickly their engineers latched onto core concepts and quickly exceeded the basic requirements of the project.” After initial architecture and investigatory discussions concluded, BlueGranite’s Catalyst Framework for Modern Data Platform was selected as the move-forward architecture.
“BlueGranite's Data Lake Hydrator took the process of moving dozens of datasets and terabytes of data from a few weeks to a few days,” CJ stated. “This foundational aspect of the Catalyst approach invigorated the team to attack the rest of the project. Having all of the data migrated to the cloud so quickly allowed for an easier, less stressful experience for building out the rest of the platform.”
BlueGranite identified 3 primary areas to help establish a scalable, strongly-governed data platform:
1 – Ingestion Framework: The number of data sources throughout the organization was immense, and it was important to have a framework that allowed new data sources to be added without a lot of additional development needed. The BlueGranite Data Lake Hydrator allowed for the speed of the Apache Spark-based Synapse Pipelines to dynamically accept new data sources with very little configuration and no new development. This was key to solving the interoperability issues that this organization was facing, just like many other healthcare organizations.
2 – Storage Design and Best Practices: The team worked to convert large, relational on-premises stores into a more modern approach. Data needed for business analytics and rapid intelligence for unit leaders and C-Suite individuals was stored within Azure Synapse dedicated pools, with its industry-leading massively-parallel processing engine, that allowed their resources to scale with the ebb and flow of individual user need. Other datasets, such as those for research or sharing with external networks, was stored in a well-curated and maintained Azure Data Lake. Now, users could get the data they needed in the format that worked best for their use case, without having to fracture the data platform into multiple units to find it.
3 – Compute and Service Layer Design: Understanding the end-use case of data is critical to the success of data platform projects. BlueGranite worked with the various client business stakeholders, engineers, and various analytic teams to understand their tooling and data needs, in order to design an architecture that allowed these groups the role-based access to the data that they needed, with the least amount of friction possible.
“Organizations like to think about data science and data analytics, the potential they bring, and how they fit into a provider’s organization. What they don’t think about is the fact that nearly 80% of every data project is acquiring and cleaning data,” explains CJ. “If we wanted to make sure that these analysts were able to deliver value, the amount of time verifying and cleansing data needed to be minimized by this architecture. We needed to design this platform with analytics in mind, because data without analytics is just a cost-center that drags down a balance sheet.”
“Every provider system wants a single source of truth. Creating that source is one-part architecture, one-part education,” explains CJ. “It is important that everyone references the same data, but it is also important that everyone understand how their new tools work.”
This modernization push introduced a lot of new tools and services, such as Azure Data Factory, Mapping Data Flows, Azure Synapse SQL Pools, and PySpark through Azure Synapse notebooks and Databricks Notebooks.
So many new tools meant a lot of education, and thankfully, using the Catalyst framework to boost the integration allowed for plenty of time for training. “It ended up being a part of the project that was most of the teams favorite,” CJ recalls, smilingly. “Everyone was super eager to see what else they could do with the new tools they had in Azure data services."
