Course Introduction
Cloud Data Engineering with AWS Glue and Lake Formation Training Course provides a comprehensive framework for designing, implementing, governing, securing, and operating modern cloud data platforms using AWS Glue and AWS Lake Formation. The course examines how organizations can build scalable data lakes and analytics environments that bring together structured, semi-structured, and unstructured information from databases, applications, files, APIs, streaming systems, and cloud services. Participants develop an understanding of how data engineering, metadata management, governance, security, and analytical consumption can be integrated into a coherent AWS data architecture.
The course develops practical knowledge of AWS Glue architecture and its major capabilities for data discovery, cataloging, extraction, transformation, loading, workflow orchestration, and serverless data processing. Participants explore crawlers, the AWS Glue Data Catalog, jobs, connections, triggers, workflows, dynamic frames, Spark-based processing, job bookmarks, and reusable transformation patterns. Particular attention is given to building maintainable and scalable pipelines that can process growing data volumes while reducing operational complexity and supporting repeatable enterprise data engineering practices.
AWS Lake Formation capabilities are addressed as a central component of governed data lake engineering. Participants learn how to establish centralized data access controls, permissions, database and table governance, data locations, cross-account sharing, tag-based access controls, and secure analytical access. The training examines how governance requirements can be incorporated directly into data platform architecture rather than treated as a separate administrative function. Participants also explore how Lake Formation and the Glue Data Catalog can support data discovery, controlled sharing, lineage-oriented practices, and consistent governance across enterprise data environments.
A major emphasis is placed on reliable data ingestion, transformation, quality, and lifecycle management. Participants examine batch ingestion, incremental processing, schema discovery, schema evolution, partitioning, file formats, transformation pipelines, data validation, error handling, job bookmarks, and scalable processing patterns. The course addresses practical challenges such as source-system variability, duplicate records, changing schemas, late-arriving data, data freshness requirements, processing failures, and large-scale historical datasets. These practices help organizations build pipelines that are repeatable, resilient, efficient, and suitable for production data lake environments.
The training also addresses security, observability, performance, cost optimization, and operational excellence across AWS data engineering environments. Participants explore IAM integration, encryption, KMS, secure data locations, network controls, monitoring, CloudWatch, logging, alerting, job metrics, troubleshooting, and controlled access to analytical datasets. Modern delivery practices such as infrastructure as code, source control, CI/CD, automated testing, reusable configurations, and environment promotion are incorporated into the engineering lifecycle. Emerging issues including data mesh, data products, AI-ready data platforms, generative AI, metadata automation, data observability, lakehouse architectures, and intelligent data quality are also examined.
By the end of Cloud Data Engineering with AWS Glue and Lake Formation Training Course, participants will be able to design and operate governed, scalable, and secure AWS data lake solutions that support enterprise analytics and advanced data workloads. The course emphasizes practical architecture and production engineering, enabling participants to connect diverse sources, automate ingestion and transformation, manage metadata, implement fine-grained access controls, monitor pipelines, optimize processing, and establish sustainable operating practices. Participants will also be better prepared to evolve AWS data platforms as organizational analytics, governance, AI, and regulatory requirements change.
Duration
10 days
Who Should Attend
-
Cloud data engineers responsible for designing and operating AWS-based data pipelines and data lake environments.
-
AWS data architects designing scalable, governed, and secure enterprise data platforms.
-
Data engineers developing ingestion, transformation, cataloging, and analytical processing workloads on AWS.
-
Cloud architects responsible for modernizing traditional data warehouses and data integration platforms.
-
Data lake engineers implementing centralized storage, metadata, governance, and analytical access architectures.
-
ETL and ELT developers transitioning enterprise data integration workloads to AWS cloud services.
-
Data platform administrators managing data catalogs, permissions, processing environments, and operational controls.
-
Analytics engineers preparing governed datasets for business intelligence, reporting, and advanced analytics.
-
Data governance professionals responsible for access policies, metadata, ownership, classification, and compliance.
-
Security professionals supporting identity, encryption, access control, and protection of cloud data assets.
-
DevOps and platform engineers supporting automated deployment and operational management of AWS data platforms.
-
Database professionals integrating operational databases with AWS-based analytical data lakes and warehouses.
-
BI and reporting professionals requiring governed access to enterprise datasets stored in AWS data lakes.
-
Solution architects evaluating AWS Glue and Lake Formation for enterprise data integration and governance.
-
Data migration specialists moving large datasets from on-premises environments into AWS cloud platforms.
-
Technical project managers overseeing cloud data modernization, migration, and analytics transformation initiatives.
-
Machine learning and AI platform professionals preparing governed datasets for model development and advanced analytics.
-
Professionals seeking practical expertise in building, securing, governing, monitoring, and optimizing AWS data lake architectures.
Course Objectives
-
Develop a comprehensive understanding of AWS Glue and Lake Formation architecture and apply their capabilities to enterprise cloud data engineering requirements.
-
Design scalable AWS data lake architectures that integrate structured, semi-structured, and unstructured information from diverse enterprise sources.
-
Build reliable ingestion and transformation pipelines using AWS Glue jobs, crawlers, workflows, connections, and reusable processing patterns.
-
Configure and manage the AWS Glue Data Catalog to provide consistent metadata, schema discovery, dataset organization, and analytical data accessibility.
-
Apply Lake Formation governance capabilities to establish fine-grained permissions, secure data access, centralized policies, and controlled data sharing.
-
Implement efficient batch and incremental processing strategies using partitioning, job bookmarks, metadata, transformation logic, and optimized storage formats.
-
Design robust data transformation workflows using AWS Glue and Spark-based processing approaches for cleansing, enrichment, aggregation, and analytical preparation.
-
Establish secure data engineering practices using IAM, encryption, AWS KMS, controlled data locations, network security, and least-privilege access principles.
-
Implement data quality, validation, monitoring, logging, alerting, and troubleshooting practices that improve pipeline reliability and operational supportability.
-
Optimize AWS Glue workloads through appropriate worker selection, partitioning, parallelism, file formats, processing strategies, and cost-aware engineering decisions.
-
Integrate data engineering delivery with source control, CI/CD, infrastructure automation, testing, environment promotion, and production release management.
-
Evaluate emerging AWS data platform patterns involving data mesh, data products, lakehouse architectures, AI-ready data, generative AI, metadata automation, and data observability.
Comprehensive Course Outline
Module 1: Foundations of Cloud Data Engineering on AWS
-
Cloud data engineering principles, enterprise analytics requirements, modern data lake architectures, and the role of AWS managed services in scalable data platform engineering.
-
Differences between traditional ETL, cloud-native ELT, serverless processing, data lake engineering, lakehouse architectures, and distributed data processing approaches.
-
Enterprise data engineering lifecycle covering source discovery, ingestion, cataloging, transformation, validation, governance, storage, analytical consumption, monitoring, and optimization.
-
Architectural considerations for integrating AWS Glue and Lake Formation with Amazon S3, Athena, Redshift, EMR, databases, streaming platforms, applications, APIs, and external data services.
Module 2: AWS Glue Architecture and Core Components
-
AWS Glue architecture covering Data Catalog, crawlers, classifiers, jobs, connections, triggers, workflows, development environments, and serverless processing capabilities.
-
Understanding how AWS Glue components interact across discovery, metadata management, data movement, transformation, orchestration, monitoring, and production operations.
-
Designing reusable Glue environments through standardized naming, configuration, parameterization, environment separation, metadata management, and enterprise engineering conventions.
-
Selecting appropriate AWS Glue capabilities for batch processing, data discovery, transformation, cataloging, workflow automation, and integration with broader AWS data services.
Module 3: Amazon S3 Data Lake Storage and Data Organization
-
Designing enterprise data lakes on Amazon S3 using appropriate storage organization, prefixes, zones, naming conventions, lifecycle strategies, and data accessibility patterns.
-
Organizing raw, processed, curated, and analytical datasets to support controlled ingestion, transformation, governance, discovery, and downstream analytical consumption.
-
Evaluating storage formats including CSV, JSON, Avro, Parquet, and ORC for performance, compression, schema management, interoperability, and analytical workloads.
-
Applying partitioning, file sizing, compaction, lifecycle policies, and storage optimization techniques to support efficient and sustainable enterprise data lakes.
Module 4: AWS Glue Data Catalog and Metadata Engineering
-
AWS Glue Data Catalog architecture covering databases, tables, schemas, partitions, classifiers, crawlers, metadata organization, and integration with AWS analytical services.
-
Designing metadata management strategies that improve data discovery, dataset understanding, schema consistency, analytical access, and governance across large enterprise environments.
-
Managing schema discovery and evolution through crawler configuration, explicit schema definitions, metadata validation, partition registration, and controlled changes.
-
Establishing metadata standards for naming, ownership, descriptions, classifications, business context, technical metadata, lifecycle status, and data product management.
Module 5: Data Discovery, Crawlers and Schema Management
-
Designing AWS Glue crawlers for discovering schemas and partitions across Amazon S3 and supported data stores while controlling crawler scope and metadata behavior.
-
Understanding crawler classifications, schema detection, partition discovery, table creation, metadata updates, and operational considerations for large data lake environments.
-
Managing schema changes and source-system evolution while minimizing downstream disruption to transformation pipelines, analytical queries, and governed data consumers.
-
Developing practical approaches for balancing automated schema discovery with explicit metadata management when enterprise governance and data quality requirements demand greater control.
Module 6: AWS Glue Jobs and Transformation Engineering
-
AWS Glue job architecture, Spark-based processing, DynamicFrames, DataFrames, transformations, job parameters, worker configuration, and reusable processing approaches.
-
Designing transformation pipelines for cleansing, filtering, joining, aggregation, enrichment, standardization, deduplication, normalization, and analytical dataset preparation.
-
Applying Spark-oriented engineering principles to improve distributed processing performance, manage transformations effectively, and avoid common data engineering bottlenecks.
-
Developing modular Glue jobs that support reusable logic, parameterized execution, error handling, logging, validation, testing, and controlled production deployment.
Module 7: Incremental Processing, Job Bookmarks and Change Management
-
Incremental processing principles for reducing unnecessary data movement and transformation through timestamps, partitions, job bookmarks, control metadata, and change indicators.
-
AWS Glue job bookmarks and related techniques for tracking processed data and supporting recurring ingestion workloads across files, tables, and other supported data sources.
-
Designing incremental pipelines for inserts, updates, deletes, late-arriving records, historical corrections, and changing source-system conditions.
-
Building restartable and recoverable processing patterns that maintain consistency while supporting large-scale recurring workloads and evolving enterprise data requirements.
Module 8: Data Lake Transformation, Partitioning and Performance Engineering
-
Designing partitioning strategies based on workload characteristics, query patterns, data volume, time dimensions, business attributes, and analytical access requirements.
-
Optimizing Parquet and ORC datasets through compression, column selection, file sizing, partition management, compaction, and efficient storage organization.
-
Improving Glue job performance through worker selection, parallel processing, transformation design, data partitioning, shuffle management, and appropriate Spark configurations.
-
Evaluating processing bottlenecks across storage, network, compute, transformations, metadata operations, and downstream analytical services to guide targeted optimization.
Module 9: AWS Lake Formation Architecture and Data Governance
-
AWS Lake Formation architecture, data lake registration, permissions, data locations, databases, tables, governed access, and integration with the AWS Glue Data Catalog.
-
Establishing centralized governance models that provide controlled access to data assets while supporting analytical flexibility across departments, teams, and organizational boundaries.
-
Designing database- and table-level access controls alongside more granular governance requirements for enterprise datasets containing sensitive or regulated information.
-
Developing sustainable governance operating models covering data ownership, stewardship, access approvals, policy administration, auditing, documentation, and continuous control improvement.
Module 10: Fine-Grained Access Control and Secure Data Sharing
-
Implementing Lake Formation permissions and fine-grained access patterns to control who can discover, query, modify, and administer specific data resources.
-
Designing secure cross-account and organizational data sharing approaches that enable collaboration while maintaining appropriate ownership, access boundaries, and governance responsibilities.
-
Integrating Lake Formation with IAM, AWS KMS, Amazon S3, Athena, Redshift, and other AWS services to establish layered data security and controlled analytical access.
-
Managing access reviews, permission changes, policy conflicts, least-privilege principles, audit requirements, and operational procedures for maintaining secure data environments.
Module 11: Data Quality, Validation and Reliable Pipeline Engineering
-
Data quality engineering practices covering completeness, accuracy, consistency, uniqueness, validity, timeliness, schema conformity, and business-rule validation across data lake pipelines.
-
Designing validation stages within Glue workflows to identify malformed records, unexpected schemas, duplicates, missing values, invalid business conditions, and source-system anomalies.
-
Implementing error handling, quarantine patterns, retry strategies, exception logging, reconciliation processes, and controlled recovery procedures for failed data processing.
-
Establishing data quality monitoring and operational metrics that provide early visibility into pipeline degradation, source changes, data freshness problems, and analytical reliability risks.
Module 12: Security, Compliance and Enterprise Data Protection
-
AWS identity and access management principles for Glue and Lake Formation environments, including roles, policies, service permissions, least privilege, and controlled administrative access.
-
Protecting sensitive information using encryption, AWS KMS, secure S3 configurations, network controls, private connectivity, credential management, and secure service integration.
-
Applying governance and compliance practices for regulated, confidential, personal, financial, healthcare, and otherwise sensitive enterprise datasets stored within cloud data lakes.
-
Designing auditable security operating models covering access reviews, policy changes, administrative activities, data usage, incident response, and continuous compliance monitoring.
Module 13: Orchestration, Monitoring and Production Operations
-
AWS Glue workflows, triggers, dependencies, schedules, conditional execution, notifications, and orchestration patterns for coordinating complex enterprise data processing.
-
Monitoring Glue jobs through execution history, metrics, logs, CloudWatch integration, operational dashboards, alerts, and pipeline-level health indicators.
-
Troubleshooting common production issues involving schema changes, permissions, failed transformations, source connectivity, Spark processing, data quality, resource constraints, and job configuration.
-
Establishing production operating procedures covering incident management, root-cause analysis, recovery, reruns, operational documentation, service-level expectations, and continuous reliability improvement.
Module 14: CI/CD, Automation and Enterprise Deployment
-
Source control and collaborative development practices for AWS Glue scripts, configurations, infrastructure definitions, data catalog components, and Lake Formation governance policies.
-
Continuous integration and deployment approaches for promoting data engineering workloads across development, testing, staging, and production environments.
-
Infrastructure automation using AWS-native deployment capabilities and infrastructure-as-code principles for repeatable creation and configuration of data platform resources.
-
Implementing automated testing, validation, release approvals, rollback planning, environment parameterization, and controlled deployment processes for production data engineering.
Module 15: Data Mesh, AI and Emerging AWS Data Engineering Architectures
-
Data mesh principles, domain-oriented data ownership, data products, federated governance, self-service analytical platforms, and the role of Glue and Lake Formation in decentralized enterprise data ecosystems.
-
Designing AI-ready data platforms that support machine learning, generative AI, retrieval-augmented applications, feature engineering, model development, and governed access to high-quality enterprise data.
-
Emerging patterns involving data observability, automated metadata management, intelligent data quality, synthetic data, event-driven ingestion, real-time processing, and increasingly autonomous data operations.
-
Evaluating lakehouse architectures, open table formats, multi-engine analytics, serverless data platforms, cross-account data sharing, and evolving AWS-native approaches to enterprise data management.
Module 16: Enterprise AWS Data Platform Strategy and Future Readiness
-
Developing an enterprise strategy for AWS Glue and Lake Formation aligned with organizational data priorities, analytics requirements, governance models, security expectations, and cloud transformation objectives.
-
Establishing operating models covering platform ownership, data stewardship, support responsibilities, access administration, monitoring, capacity planning, cost management, and continuous improvement.
-
Creating modernization roadmaps for consolidating fragmented data environments into governed cloud data lakes while accommodating migration priorities, legacy systems, and changing business requirements.
-
Building future-ready architectures capable of supporting expanding data volumes, advanced analytics, AI adoption, regulatory changes, new data products, multi-account environments, and evolving cloud engineering practices.
Training Approach
This course will be delivered by our skilled trainers who have vast knowledge and experience as expert professionals in the fields. The course is taught in English and through a mix of theory, practical activities, group discussion and case studies. Course manuals and additional training materials will be provided to the participants upon completion of the training.
Tailor-Made Course
This course can also be tailor-made to meet organization requirement. For further inquiries, please contact us on: Email: training@upskilldevelopment.com Tel: +254 721 331 808
Training Venue
The training will be held at our Upskill Training Centre. We also offer training for a group (at a discount of 10% to 50%) at requested location all over the world. The Onsite course fee covers the course tuition, training materials, two break refreshments, buffet lunch, airport transfers, Upskill gift package, and guided tour.
Visa application, travel expenses, dinners, accommodation, insurance, and other personal expenses are catered by the participant
Certification
Participants will be issued with Upskill certificate upon completion of this course.
Airport Pickup and Accommodation
Airport pickup and accommodation is arranged upon request. For booking contact our Training Coordinator through Email: training@upskilldevelopment.com, +254 721 331 808
Terms of Payment:
Unless otherwise agreed between the two parties’ payment of the course fee should be done 3 working days before commencement of the training so as to enable us to prepare better.
