Data science course with AI data platforms
Cloud Data
Engineering & AI
Build modern data pipelines, lakehouses and AI data platforms. Learn
SQL, Python, AWS, Spark, Databricks, Airflow, dbt and Kafka the way data
engineering teams use them, then extend your pipelines to feed GenAI
and RAG applications.
16 Weeks
Course duration
120+ Hours
Guided learning
3 Projects
Plus a capstone
Entry Level
No coding needed
Live online, instructor-led, with hands-on labs.
Next batch: 02 February 2027        Weekend batch: Saturday & Sundays        Seats per batch: 25
Reserve a seat
The stack this course is built on
The technologies data engineering job descriptions ask for most. You build with each one, and
they come together in one platform by the end of the course.
Why cloud data engineering
with AI, and why now
SQL, Python, Spark and cloud remain the core of data
engineering hiring. What has changed is that companies
now need engineers who can run lakehouses and feed AI
applications with clean, governed data. Switch between the
two to see the difference.
What you learn
A list of cloud services and tools, each taught on its own.
How you work
Click through consoles and run notebooks by hand.
Projects
Many small exercises on unrelated datasets.
Quality of your models
Pipelines that run, with nobody checking what comes out.
Interview readiness
Tool knowledge, little practice explaining design choices.
How every module is taught
Each session is a short concept, a live build and a lab you finish yourself, on the same
business throughout. AI tools are used the way working data engineers use them.
Skill first
You write the SQL, the PySpark job and the DAG yourself before any AI shortcut. You can’t review AI-written pipeline code in a skill you don’t have.
Then speed it up
You use AI assistants to draft queries and transformations, explain unfamiliar code, debug failed jobs and generate documentation and tests.
Always verify
Every pipeline is checked: row counts reconciled, data quality tests passing, costs and run times measured. Teams rely on data that can be trusted.
Curriculum
Thirteen modules and a final capstone, in the order a modern data platform is built. Highlighted
modules are where AI comes in. Open any module to see topics, tools and what you build.
SQL for data engineering
The skill every data engineering interview tests first, taken from fundamentals
to production-grade queries.
What you'll learn
- Fundamentals refresher: SELECT, filtering, aggregation and GROUP BY
- Joins, subqueries and CTEs
- Window functions: ranking, running totals, LAG and LEAD
- Analytical queries and deduplication patterns
- Query plans, indexes and query optimisation
- Stored procedures, views and transactions
- SQL interview problems, timed
Tools and AI you use
- PostgreSQL
- AI assistants to draft and explain queries, checked against expected results
Hands-on Write the SQL behind Zestro’s order, seller and customer reporting, then make the three slowest queries at least five times
 faster.
Python for data engineering
Python as data engineers use it: reliable scripts that move and shape data, not
 notebooks.
What you'll learn
- Python fundamentals, functions and OOP basics
- File handling: CSV, JSON and Parquet
- Calling REST APIs and handling pagination
- Pandas and NumPy for data shaping
- Exception handling, logging and configuration
- Writing a reusable, tested ETL script
Tools and AI you use
- Python, VS Code, pytest
- AI assistants for code review and test generation
Hands-onBuild a Python ETL script that pulls Zestro orders from an API, validates and cleans them, logs failures and writes Parquet
 files.
Data engineering foundations and data modelling
How data platforms are designed, before choosing any tool.
What you'll learn
- ETL versus ELT
- Batch versus streaming
- Data pipeline architecture
- Data lakes, data warehouses and lakehouses
- Dimensional modelling: star and snowflake schemas
- Slowly changing dimensions, types 1 and 2
- File formats and partitioning: Parquet, Avro and Delta
Tools and AI you use
- Draw.io for architecture diagrams
- AI to critique a proposed model, then defend your choices
Hands-onDesign Zestro’s analytical data model and a first platform architecture, including how customer and seller changes
are tracked over time.
AWS for data engineering
One cloud, taught deeply. Every service is used in a pipeline, not just toured
in the console.
What you'll learn
- IAM: users, roles and least-privilege policies
- S3: buckets, prefixes, storage classes and lifecycle rules
- EC2 and RDS basics, and VPC essentials
- Lambda for event-driven ingestion
- Glue: crawlers, catalog and Spark ETL jobs
- Athena and Redshift for querying and warehousing
- CloudWatch, budgets and cost control
- Azure and GCP equivalents, mapped service by service
Tools and AI you use
- AWS console and CLI
- AI to draft IAM policies, checked for over-broad permissions
Hands-on Build Project 1: ingest Zestro API and CSV data with Python into S3, transform with Glue, load into Redshift and
report in Power BI.Â
Apache Spark and PySpark
The largest module on the course. Distributed processing from first principles to tuned
 production jobs.
What you'll learn
- Spark architecture: driver, executors, jobs, stages and tasks
- DataFrames, Spark SQL, transformations and actions
- Joins, broadcast joins and handling data skew
- Partitioning, shuffling and caching
- Reading the Spark UI and tuning performance
- UDFs and when to avoid them
- PySpark ETL pipelines, structured and tested
Tools and AI you use
- PySpark
- Spark UI
- AI to explain query plans and slow stages, confirmed in the Spark UI
Hands-on Process two years of Zestro clickstream and order data with PySpark, find the job’s bottleneck in the Spark UI
and cut its run time.
Databricks and the lakehouse
Where modern data platforms are heading: Spark, Delta Lake and governance
 in one place.
What you'll learn
- Databricks architecture, workspaces and clusters
- Lakehouse and medallion architecture: bronze, silver and gold
- Delta tables: ACID, schema enforcement and evolution
- Time travel, MERGE and OPTIMIZE
- Auto Loader for incremental ingestion
- Unity Catalog for governance and access
- Databricks Workflows and basic streaming
Tools and AI you use
- Databricks
- Delta Lake
- Unity Catalog
- Databricks Assistant, checked by you
Hands-onBuild Project 2: a medallion lakehouse for Zestro with Auto Loader into bronze, cleaned silver tables, business-ready gold tables
and a dashboard.
Pipeline orchestration with Apache Airflow
Pipelines that run on schedule, recover from failure and tell you when
something breaks.
What you'll learn
- DAGs, tasks, operators and dependencies
- Scheduling, sensors and data-aware scheduling
- Retries, timeouts, backfills and catch-up
- Variables, connections and secrets
- Variables, connections and secrets
- Orchestrating AWS and Databricks jobs
- Production DAG design patterns
Tools and AI you use
- Apache Airflow in Docker
- AI to draft DAGs, then tested for failure and retry behaviour
Hands-onOrchestrate Zestro’s daily pipeline in Airflow, from ingestion through Databricks to warehouse, with retries, alerts andÂ
a backfill of last month.
dbt and analytics engineering
The modern way to transform, test and document warehouse data.
What you'll learn
- dbt fundamentals: models, sources and refs
- Tests and documentation
- Jinja and macros
- Incremental models
- Snapshots for slowly changing dimensions
- Running dbt from Airflow
Tools and AI you use
- dbt Core
- AI to generate tests and documentation, reviewed for accuracy
Hands-onRebuild Zestro’s gold layer as tested, documented dbt models, with a snapshot tracking seller changes, scheduledÂ
from Airflow.
Data streaming with Apache Kafka
Real-time pipelines for when daily batch isn’t fast enough. Practical, not Kafka
 administration.
What you'll learn
- Streaming versus batch: when each is worth it
- Kafka architecture: brokers, topics and partitions
- Producers, consumers and consumer groups
- Offsets and delivery guarantees
- Kafka to Spark Structured Streaming
- Writing streams to Delta tables
- Real-time pipeline architecture
Tools and AI you use
- Apache Kafka in Docker
- Spark Structured Streaming
Hands-on Stream Zestro orders through Kafka into Spark Structured Streaming and a live Delta table that updates a near real-time
 sales view.
Data quality and governance
What makes data trustworthy, and what makes a data engineer look professional.
What you'll learn
- Data profiling and validation
- Data quality checks with Great Expectations and dbt tests
- Schema management and contracts
- Data lineage
- PII detection, masking and data security
- Access control and governance basics
Tools and AI you use
- Great Expectations
- Unity Catalog lineage
- AI to suggest quality rules from a data profile, then tested
Hands-on Add quality gates to Zestro’s pipelines that stop bad data reaching gold tables, and mask customer PII for analyst access.
DevOps for data engineering
The engineering practices many data courses skip, and many interviewers ask about.
What you'll learn
- Git and GitHub workflows for data teams
- Docker for pipelines and local environments
- Environment and secrets management
- Testing data pipelines
- CI/CD with GitHub Actions
- Infrastructure as code with Terraform basics
Tools and AI you use
- Git, GitHub Actions, Docker, Terraform
- AI to draft CI workflows and Terraform, reviewed before running
Hands-on Put Zestro’s pipelines under CI/CD: tests and dbt checks on every pull request, containerised jobs, and S3 resources
 defined in Terraform.
AI for data engineers
Not generic ChatGPT. AI applied to data engineering work, and the data pipelines that AI applications depend on.
What you'll learn
- GenAI fundamentals for data engineers, and LLM APIs
- AI-assisted SQL, PySpark, pipeline development and debugging
- Document ingestion pipelines: parsing, chunking and metadata
- Embedding pipelines and vector databases
- RAG data pipelines and keeping them fresh
- AI-generated metadata, documentation and data quality assistants
- Building a simple data engineering agent
Tools and AI you use
- LLM APIs
- pgvector
- Databricks
- LangChain for ingestion and retrieval
Hands-onBuild Project 3: stream Zestro orders and support documents through Kafka and Spark into Delta, then feed an embedding
pipeline and RAG layer that answers order and policy questions.
Data engineering system design
The interview round that separates candidates, practised on realistic scenarios.
What you'll learn
- Designing for scale: a 10 TB per day e-commerce pipeline
- Fault tolerance, idempotency and reprocessing
- Partitioning and performance
- Cost optimisation
- Batch versus streaming trade-offs
- Security and monitoring by design
Tools and AI you use
- Whiteboard and diagramming tools
- AI as a mock interviewer, with peer and trainer review
Hands-on Design Zestro’s full platform for 10 TB a day, source to Kafka, Spark, lakehouse, warehouse and BI, and defend
it in a mock interview.
Final capstone: enterprise cloud data platform
Bring the three projects together into one enterprise data platform, designed, built
and presented by you.
What you'll do
- Write an architecture and design document
- Batch and streaming ingestion on AWS and Kafka
- Medallion lakehouse on Databricks with Delta Lake
- Orchestration with Airflow and transformation with dbt
- Quality gates, governance and CI/CD
- An AI-ready layer with embeddings and RAG
- A 20-minute architecture review with a panel
How it's assessed
- You choose the design and justify it
- The panel reviews your architecture, data quality results and costs
You leave with An enterprise cloud data platform on GitHub, an architecture document and a design review you can walk an interviewer
through.
Who it's for, and how it
runs
Built for people aiming at data engineer and cloud data engineer roles, whether starting out or moving across from analytics, software or IT. Some programming exposure helps; SQL and Python are taught from the ground up.
This course suits you if you are
A graduate or final-year student
From engineering, computer science or IT, aiming for a first data engineering role.
A data or BI
analyst
Comfortable with SQL and reports, and ready to build the pipelines behind them.
A software or backend developer
Who wants to move into data platforms, Spark and the cloud.
An ETL, DBA or support engineer
Working with legacy tools and moving to a modern cloud and lakehouse stack.
An engineer heading
into AI
Who wants to build the data side of GenAI: ingestion, embeddings and RAG pipelines.
A working
professional
Able to commit to 16 weeks of live sessions, with time between sessions to finish labs and projects.
One business, three projects
and a capstone
Every lab and project uses Zestro, a fictional e-commerce marketplace with millions of orders,
clickstream events, seller catalogues and support documents. Each project builds on the last,
so by the end you have designed and built one enterprise data platform that answers a real business need:
Can every team get trusted, near real-
time data, ready for analytics and AI?
Tools you will use
All industry standard. Highlighted tools are the core stack the course is built around.
Roles this course prepares you for
The skills map directly to the data engineering roles companies hire for most often.
Data Engineer
Build and run batch and streaming pipelines that teams across the business rely on.
Cloud Data Engineer (AWS)
Design storage, processing and warehousing on AWS, with security and cost in mind.
Big Data or Spark Engineer
Process large data volumes with PySpark, and tune jobs for speed and cost.
Databricks or Lakehouse Engineer
Build medallion lakehouses on Delta Lake with governed, reliable tables.
Analytics Engineer
Model, test and document warehouse data with dbt for analysts and BI teams.
AI Data Engineer (early career)
Build the ingestion, embedding and quality pipelines behind GenAI applications.
Frequently Asked Questions
Some programming exposure helps, but SQL and Python are taught from the ground up in the first three weeks, up to the level the rest of the course needs. [Add any pre-course readiness check or primer you offer here.
Going deep on one cloud makes you employable faster than skimming three. AWS is widely used in Indian data engineering hiring, and the course maps each AWS service to its Azure and GCP equivalent so you can move across later.
[Confirm before publishing: whether cloud and Databricks lab access is provided, or learners use their own AWS free tier and Databricks Free Edition accounts, and the typical cost across the course.] Labs are designed to keep usage small, and you’ll learn to shut resources down and set budget alerts from week 5.
Any recent laptop that can run Docker, Python and VS Code. 16 GB RAM is recommended, as Airflow and Kafka run locally in containers.
[Confirm before publishing: whether the course is mapped to the AWS Certified Data Engineer Associate and Databricks Certified Data Engineer Associate exams, and whether vouchers are included.]
No. It’s AI applied to data engineering: AI-assisted SQL, PySpark and debugging, then building document ingestion, embedding and RAG data pipelines, and a small data engineering agent.
A GitHub portfolio with three projects, a final enterprise data platform capstone with its architecture design, and system design practice you can walk through in interviews. [Add any placement or career support you offer here.]
[Recordings are shared after every session, and you can bring questions to the next doubt-clearing slot.]