Medical Data Processing with AWS S3 & BigQuery

• Amazon Simple Storage Service (S3) is used to store unstructured data, including DICOM files and CSV files.
• DICOM is a standard used for managing, storing, and transmitting medical imaging data such as X-rays, CT scans, MRIs, and ultrasounds.
• DICOM metadata contains structured data embedded in DICOM files, which includes patient demographics, imaging parameters, timestamps, and modality information.
• Google Cloud's BigQuery (BQ) is a scalable data warehouse used for querying structured datasets like anonymized DICOM metadata and linked reports.
• Deidentification removes Protected Health Information (PHI) from medical datasets to comply with HIPAA and GDPR.
• The Apache Beam Pipeline is a framework that extracts, transforms, and loads (ETL) data across multiple storage systems.
• Optical Character Recognition (OCR) extracts and sanitizes text from medical images and reports.
• The Full Deidentification Boundary ensures all identifiable PHI is removed before data is exposed to external users.

Hospital Data Ingestion

• Each hospital has an S3 Bucket that stores identified DICOMs (raw medical images with PHI) and identified CSV reports (structured patient data).
• Apache Beam processes and extracts DICOM metadata, storing it in a BigQuery Table (dicom_metadata).
• Identified Reports are stored in a separate BQ Table, and deidentified Thumbnails are stored in an S3 Bucket.

Data Processing and Linking

• The BigQuery final_table links deidentified DICOM metadata and deidentified reports.
• This is then loaded into BQ Table prod, a consolidated and indexed dataset for multiple hospitals.
• The Apache Beam Pipeline handles querying and aggregating deidentified metadata and processes original DICOMs using OCR to remove residual PHI.
• Cleaned, fully deidentified datasets are delivered to external users.

Searchable Deidentified Dataset

• Processed data is moved to the BQ Table public_table after deidentification.
• The public_table contains fully deidentified, structured medical data from all hospitals to enable advanced search and analytics.

Architecture Significance

• This architecture ensures secure handling and deidentification of PHI while maintaining data integrity.
• It provides a scalable, cloud-native pipeline for processing large-scale medical imaging datasets.
• AI/ML model training is facilitated on real-world medical data in compliance with HIPAA/GDPR.
• Fast querying and retrieval of multi-hospital data are enabled through Google BigQuery.