Transforming clinical measures computation with cloud native BigData pipeline

The project identified a significant opportunity to revamp the clinical measure computation process for a leading healthcare provider:

• Transition to Cost-Effective Solutions: The existing MSSQL, stored procedures, and C# based toolchain was costly due to .NET licensing fees. There was a clear need to reduce Total Cost of Ownership (TCO) while maintaining or enhancing the system’s efficiency.
• Leveraging Open Source Technologies: By transitioning to open-source technologies, there was potential to significantly drive down costs.
• Improving Scalability: The existing system’s scalability was limited, necessitating a solution that could handle increasing data loads and computational demands more effectively.
• Modernizing the Infrastructure: Adopting a cloud-native Big Data pipeline was identified as a key strategy to modernize the infrastructure, aligning with current technological advancements.
• Enhanced Data Processing: The project aimed to improve the way clinical data was processed, aiming for faster, more accurate computations.
• Compliance and Security: In the healthcare sector, data compliance and security are paramount. The new system needed to meet these standards while being more cost-effective.
• Interoperability and Integration: The solution needed to ensure seamless integration with existing healthcare systems an

We analyzed the existing process and identified the technology, financial and business concerns that needed to be addressed. We then designed a cloud native BigData pipeline that processed over 100 million clinical records in a day. At it’s peak the pipeline processed Terabytes of clinical data every day. Here is an outline of the modules that were built for this transformation project:

• Source data storage and management: We used the Hadoop Distributed File System (HDFS) for handling and storing the large volumes of clinical raw data. Apache Hadoop was an excellent for big data storage and processing. The HDFS cluster was hosted on AWS EC2 nodes with SSDs and high network bandwidth. Eventually we added AWS S3 to this layer and controlled the cost of this module by storing data in various S3 storage tiers offered by AWS.
• Data Processing pipeline: We used Apache Spark for running the measure element and measure score map reduce pipelines. It was a great fit for our large-scale data processing use case. We used Apache Spark exclusively in batch mode. We augmented the computing pipeline with Apache Solr for fetching intermediate results or master table lookups. These tools helped in building an efficient data ingestion and processing pipeline.
• Database Management: This module was very crucial for the success of this entire product. We used Apache Cassandra for storing the intermediate results and the final measure scores. Apache Cassandra was a great fit for this use case because of its high availability, scalability and fault tolerance. We used Apache Cassandra exclusively in batch mode. We used Apache Solr for indexing the data stored in Apache Cassandra. This helped in building a fast and efficient data retrieval pipeline. Date, week, month, quarter and yearly data pivots were persisted in Apache Cassandra.
• Workflow Orchestration: We built a custom in house admin UI that could automate, schedule and orchestrate the Spark jobs.
• Containerization and Orchestration: We used built a custom build process to submit lightweight Spark job jar files via the workflow orchestration toolchain. We eventually moved to Docker containers and Kubernetes for containerization and orchestration.
• Monitoring and Logging: We used Prometheus, AWS Cloudwatch and Cloudwatch logs for monitoring the infrastructure and visualizing logs. The job level monitoring tools were built for support engineers to monitor the job execution and troubleshoot issues. This helped in building a robust monitoring and logging and trustable infrastructure.
• Security and Compliance: The pipeline was built to comply with the HIPAA regulations. We used AWS IAM roles to secure the data at rest and transit along with the infrastructure that accessed it. The entire pipeline was hosted on AWS Gov Cloud. We used AWS KMS for encrypting the data at rest and in transit. We used AWS Cloudwatch and Cloudwatch logs for monitoring the infrastructure and visualizing logs. The job level logs were stored in AWS S3 and were secured using AWS IAM roles to comply with the HIPAA regulations.

This setup was complex and powerful, suitable for large-scale, high-performance data processing and computation of clinical elements and measures.