Automated Cloud Pipeline for Antarctic Environmental Sensor Data
At Geotab, I developed a fully automated pipeline to process and store weekly environmental sensor data from Antarctica in Google Cloud. Data was sent as zip files to a monitored Gmail inbox and handled using a custom Google Apps Script integrated with OAuth2, Google Drive, and Google Cloud Storage (GCS). Incoming files triggered Python-based Cloud Functions to extract, clean, and route the data through GCS buckets. Key services included Cloud Scheduler and BigQuery Data Transfer Service, with cleaned logs stored in partitioned and clustered BigQuery tables for efficient querying. SQL was used on BigQuery to extract key insights and analyze environmental conditions and long-term trends.