When a pioneer of modern industry founded a healthcare organization over 100 years ago, the idea that the organization would grow into one of the most pre-eminent non-profit healthcare organizations in the United States was likely not top-of-mind. The decades of industry-leading excellence in care allowed this regional health provider, with more than 1,200 physicians in over 40 specialties, cemented itself as a premier healthcare provider nationally. All 30,000 employees are dedicated to providing care, and that often includes finding innovative ways to deliver information and data, while minimizing costs and empowering their ambition of creating analytics around value-based care.
Healthcare systems must always balance the delivery of care with operational efficiency. Clinical outcomes, revenue cycle management, progressive research, and administrative efficiency compete for attention. Creating a strategy for long-term success meant providing all the competing groups within such a robust provider with data that was reliable and timely. Fortunately, BlueGranite has industry experts that understand the importance and challenges surrounding the implementation of a modern healthcare data platform.
“Our client is fortunate to have a dearth of strong data scientists and data engineers,” explains CJ Knapp, BlueGranite Modern Data Platform Architect and 15-year Healthcare and Life Sciences veteran. “Despite their data talent, without a streamlined modern data platform, competing answers from disparate systems were causing inefficiency and headaches at nearly every organizational level.”
Integrating PAS/EHR data from Epic with diagnostic data, CMS data, genomics data, and the myriad of other sources through the organization was the key to success in the long-term. In order to tackle this challenge, the organization’s data engineering team decided to found a data-driven paradigm underpinned by a reliable, fast, and scalable data platform that would serve as their single source of truth.
“Designing scalable data systems for the cloud is very different from traditional data warehouse projects,” continued CJ. “Although the tooling and the patterns are different, this doesn't mean that we have to rethink how data is engineered from the ground-up. Azure Data Services had plenty of entry points that felt familiar, or perhaps even less daunting than their current tooling. This offered the perfect opportunity for BlueGranite to help teach concurrently, as we were building, making the project collaborative, and fostering buy-in on all sides."
To begin, BlueGranite set out to understand the challenge at hand by better analyzing how the data intended for this platform was utilized by the organization. “Business analysts, researches, and a number of other parties all needed this data in a fast, easy-to-access manner,” recounts Eric Wozniak, VP of BlueGranite. “It was clear that their on-premises tooling was maintained and enhanced by talented engineers, but had reached a scale limit. Modernizing was critical to furthering the system-wide desire of fostering a healthy data estate."
With the challenge of creating truth from many disparate systems in mind, “the team was ready to see what new techniques and enhancements Azure had to offer,” recalls CJ. “It was apparent immediately how every new aspect of the Azure platform was opening doorways for faster development and tighter integration. It was thrilling to see how quickly their engineers latched onto core concepts and quickly exceeded the basic requirements of the project.” After initial architecture and investigatory discussions concluded, BlueGranite’s Catalyst Framework for Modern Data Platform was selected as the move-forward architecture.
“BlueGranite's Data Lake Hydrator took the process of moving dozens of datasets and terabytes of data from a few weeks to a few days,” CJ stated. “This foundational aspect of the Catalyst approach invigorated the team to attack the rest of the project. Having all of the data migrated to the cloud so quickly allowed for an easier, less stressful experience for building out the rest of the platform.”
BlueGranite identified 3 primary areas to help establish a scalable, strongly-governed data platform:
1 – Ingestion Framework: The number of data sources throughout the organization was immense, and it was important to have a framework that allowed new data sources to be added without a lot of additional development needed. The BlueGranite Data Lake Hydrator allowed for the speed of the Apache Spark-based Synapse Pipelines to dynamically accept new data sources with very little configuration and no new development. This was key to solving the interoperability issues that this organization was facing, just like many other healthcare organizations.
2 – Storage Design and Best Practices: The team worked to convert large, relational on-premises stores into a more modern approach. Data needed for business analytics and rapid intelligence for unit leaders and C-Suite individuals was stored within Azure Synapse dedicated pools, with its industry-leading massively-parallel processing engine, that allowed their resources to scale with the ebb and flow of individual user need. Other datasets, such as those for research or sharing with external networks, was stored in a well-curated and maintained Azure Data Lake. Now, users could get the data they needed in the format that worked best for their use case, without having to fracture the data platform into multiple units to find it.
3 – Compute and Service Layer Design: Understanding the end-use case of data is critical to the success of data platform projects. BlueGranite worked with the various client business stakeholders, engineers, and various analytic teams to understand their tooling and data needs, in order to design an architecture that allowed these groups the role-based access to the data that they needed, with the least amount of friction possible.
“Organizations like to think about data science and data analytics, the potential they bring, and how they fit into a provider’s organization. What they don’t think about is the fact that nearly 80% of every data project is acquiring and cleaning data,” explains CJ. “If we wanted to make sure that these analysts were able to deliver value, the amount of time verifying and cleansing data needed to be minimized by this architecture. We needed to design this platform with analytics in mind, because data without analytics is just a cost-center that drags down a balance sheet.”
“Every provider system wants a single source of truth. Creating that source is one-part architecture, one-part education,” explains CJ. “It is important that everyone references the same data, but it is also important that everyone understand how their new tools work.”
This modernization push introduced a lot of new tools and services, such as Azure Data Factory, Mapping Data Flows, Azure Synapse SQL Pools, and PySpark through Azure Synapse notebooks and Databricks Notebooks.
So many new tools meant a lot of education, and thankfully, using the Catalyst framework to boost the integration allowed for plenty of time for training. “It ended up being a part of the project that was most of the teams favorite,” CJ recalls, smilingly. “Everyone was super eager to see what else they could do with the new tools they had in Azure data services."
Before long, BlueGranite helped this healthcare provider reach the objective of creating a reliable, veracious, and extensible modern data platform. Reaching the finish line required rearchitecting the organizational data through the lens of a cloud data platform, aligning these efforts with the needs of value-based care analysis, and finally aligning the needs of different department leaders.
The provider’s approach is proving to offer the extensibility needed, as more datasets than those originally in the scope of this project were uploaded by the end of the engagement. This, as well as the speed and volume at which data was able to be cleansed and ingested into the cloud platform, are the markers that every data-driven organization looks for in a successful project. By the end, engineers, who had never opened Azure Data Factory or a PySpark notebook, were writing complex transformations that were modular and portable.
Ideation, deployment, and monitoring of a well-governed solution ensures maximum return on investment. The litany of features across the suite of Azure Data Services allowed for an actionable foundation to be established. The foundation set here will allow the organization to start producing value-based care analytics quickly and with full trust that the data flowing into their models is both accurate and timely.
Interested in how BlueGranite can help your organization take a leap forward with a modern data platform? Contact us today.