Spark Sql And Spark 3 Using Scala Hands-On With Labs

A comprehensive course on Spark SQL as well as Data Frame APIs using Scala with complementary lab access
What you'll learn
All the HDFS Commands that are relevant to validate files and folders in HDFS.
Enough Scala to work Data Engineering Projects using Scala as Programming Language
Spark Dataframe APIs to solve the problems using Dataframe style APIs.
Basic Transformations such as Projection, Filtering, Total as well as Aggregations by Keys using Spark Dataframe APIs
Inner as well as outer joins using Spark Data Frame APIs
Ability to use Spark SQL to solve the problems using SQL style syntax.
Basic Transformations such as Projection, Filtering, Total as well as Aggregations by Keys using Spark SQL
Inner as well as outer joins using Spark SQL
Basic DDL to create and manage tables using Spark SQL
Basic DML or CRUD Operations using Spark SQL
Create and Manage Partitioned Tables using Spark SQL
Manipulating Data using Spark SQL Functions
Advanced Analytical or Windowing Functions to perform aggregations and ranking using Spark SQL
Requirements
Basic programming skills
Self support lab (Instructions provided) or ITVersity lab at additional cost for appropriate environment.
Minimum memory required based on the environment you are using with 64 bit operating system
4 GB RAM with access to proper clusters or 16 GB RAM with virtual machines such as Cloudera QuickStart VM
Description
Overview
Section 1: Introduction
Lecture 1 CCA 175 Spark and Hadoop Developer - Curriculum
Section 2: Setting up Environment using AWS Cloud9
Lecture 2 Getting Started with Cloud9
Lecture 3 Creating Cloud9 Environment
Lecture 4 Warming up with Cloud9 IDE
Lecture 5 Overview of EC2 related to Cloud9
Lecture 6 Opening ports for Cloud9 Instance
Lecture 7 Associating Elastic IPs to Cloud9 Instance
Lecture 8 Increase EBS Volume Size of Cloud9 Instance
Lecture 9 Setup Jupyter Lab on Cloud9
Lecture 10[Commands] Setup Jupyter Lab on Cloud9
Section 3: Setting up Environment - Overview of GCP and Provision Ubuntu VM
Lecture 11 Signing up for GCP
Lecture 12 Overview of GCP Web Console
Lecture 13 Overview of GCP Pricing
Lecture 14 Provision Ubuntu VM from GCP
Lecture 15 Setup Docker
Lecture 16 Why we are setting up Python and Jupyter Lab for Scala related course?
Lecture 17 Validating Python
Lecture 18 Setup Jupyter Lab
Section 4: Setup Hadoop on Single Node Cluster
Lecture 19 Introduction to Single Node Hadoop Cluster
Lecture 20 Setup Prerequisties
Lecture 21[Commands] - Setup Prerequisites
Lecture 22 Setup Password less login
Lecture 23[Commands] - Setup Password less login
Lecture 24 Download and Install Hadoop
Lecture 25[Commands] - Download and Install Hadoop
Lecture 26 Configure Hadoop HDFS
Lecture 27[Commands] - Configure Hadoop HDFS
Lecture 28 Start and Validate HDFS
Lecture 29[Commands] - Start and Validate HDFS
Lecture 30 Configure Hadoop YARN
Lecture 31[Commands] - Configure Hadoop YARN
Lecture 32 Start and Validate YARN
Lecture 33[Commands] - Start and Validate YARN
Lecture 34 Managing Single Node Hadoop
Lecture 35[Commands] - Managing Single Node Hadoop
Section 5: Setup Hive and Spark on Single Node Cluster
Lecture 36 Setup Data Sets for Practice
Lecture 37[Commands] - Setup Data Sets for Practice
Lecture 38 Download and Install Hive
Lecture 39[Commands] - Download and Install Hive
Lecture 40 Setup Database for Hive Metastore
Lecture 41[Commands] - Setup Database for Hive Metastore
Lecture 42 Configure and Setup Hive Metastore
Lecture 43[Commands] - Configure and Setup Hive Metastore
Lecture 44 Launch and Validate Hive
Lecture 45[Commands] - Launch and Validate Hive
Lecture 46 Scripts to Manage Single Node Cluster
Lecture 47[Commands] - Scripts to Manage Single Node Cluster
Lecture 48 Download and Install Spark 2
Lecture 49[Commands] - Download and Install Spark 2
Lecture 50 Configure Spark 2
Lecture 51[Commands] - Configure Spark 2
Lecture 52 Validate Spark 2 using CLIs
Lecture 53[Commands] - Validate Spark 2 using CLIs
Lecture 54 Validate Jupyter Lab Setup
Lecture 55[Commands] - Validate Jupyter Lab Setup
Lecture 56 Intergrate Spark 2 with Jupyter Lab
Lecture 57[Commands] - Intergrate Spark 2 with Jupyter Lab
Lecture 58 Download and Install Spark 3
Lecture 59[Commands] - Download and Install Spark 3
Lecture 60 Configure Spark 3
Lecture 61[Commands] - Configure Spark 3
Lecture 62 Validate Spark 3 using CLIs
Lecture 63[Commands] - Validate Spark 3 using CLIs
Lecture 64 Intergrate Spark 3 with Jupyter Lab
Lecture 65[Commands] - Intergrate Spark 3 with Jupyter Lab
Section 6: Scala Fundamentals
Lecture 66 Introduction and Setting up of Scala
Lecture 67 Setup Scala on Windows
Lecture 68 Basic Programming Constructs
Lecture 69 Functions
Lecture 70 Object Oriented Concepts - Classes
Lecture 71 Object Oriented Concepts - Objects
Lecture 72 Object Oriented Concepts - Case Classes
Lecture 73 Collections - Seq, Set and Map
Lecture 74 Basic Map Reduce Operations
Lecture 75 Setting up Data Sets for Basic I/O Operations
Lecture 76 Basic I/O Operations and using Scala Collections APIs
Lecture 77 Tuples
Lecture 78 Development Cycle - Create Program File
Lecture 79 Development Cycle - Compile source code to jar using SBT
Lecture 80 Development Cycle - Setup SBT on Windows
Lecture 81 Development Cycle - Compile changes and run jar with arguments
Lecture 82 Development Cycle - Setup IntelliJ with Scala
Lecture 83 Development Cycle - Develop Scala application using SBT in IntelliJ
Section 7: Overview of Hadoop HDFS Commands
Lecture 84 Getting help or usage of HDFS Commands
Lecture 85 Listing HDFS Files
Lecture 86 Managing HDFS Directories
Lecture 87 Copying files from local to HDFS
Lecture 88 Copying files from HDFS to local
Lecture 89 Getting File Metadata
Lecture 90 Previewing Data in HDFS File
Lecture 91 HDFS Block Size
Lecture 92 HDFS Replication Factor
Lecture 93 Getting HDFS Storage Usage
Lecture 94 Using HDFS Stat Commands
Lecture 95 HDFS File Permissions
Lecture 96 Overriding Properties
Section 8: Apache Spark 2 using Scala - Data Processing - Overview
Lecture 97 Introduction for the module
Lecture 98 Starting Spark Context using spark-shell
Lecture 99 Overview of Spark read APIs
Lecture 100 Previewing Schema and Data using Spark APIs
Lecture 101 Overview of Spark Data Frame APIs
Lecture 102 Overview of Functions to Manipulate Data in Spark Data Frames
Lecture 103 Overview of Spark Write APIs
Section 9: Apache Spark 2 using Scala - Processing Column Data using Pre-defined Functions
Lecture 104 Introduction to Pre-defined Functions
Lecture 105 Creating Spark Session Object in Notebook
Lecture 106 Create Dummy Data Frames for Practice
Lecture 107 Categories of Functions on Spark DAta Frame Columns
Lecture 108 Using Spark Special Functions - col
Lecture 109 Using Spark Special Functions - lit
Lecture 110 Manipulating String Columns using Spark Functions - Case Conversion and Length
Lecture 111 Manipulating String Columns using Spark Functions - substring
Lecture 112 Manipulating String Columns using Spark Functions - split
Lecture 113 Manipulating String Columns using Spark Functions - Concatenating Strings
Lecture 114 Manipulating String Columns using Spark Functions - Padding Strings
Lecture 115 Manipulating String Columns using Spark Functions - Trimming unwanted characters
Lecture 116 Date and Time Functions in Spark - Overview
Lecture 117 Date and Time Functions in Spark - Date Arithmetic
Lecture 118 Date and Time Functions in Spark - Using trunc and date_trunc
Lecture 119 Date and Time Functions in Spark - Using date_format and other functions
Lecture 120 Date and Time Functions in Spark - dealing with unix timestamp
Lecture 121 Pre-defined Functions in Spark - Conclusion
Section 10: Apache Spark 2 using Scala - Basic Transformations using Data Frames
Lecture 122 Introduction to Basic Transformations using Data Frame APIs
Lecture 123 Starting Spark Context
Lecture 124 Overview of Filtering using Spark Data Frame APIs
Lecture 125 Filtering Data from Spark Data Frames - Reading Data and Understanding Schema
Lecture 126 Filtering Data from Spark Data Frames - Task 1 - Equal Operator
Lecture 127 Filtering Data from Spark Data Frames - Task 2 - Comparison Operators
Lecture 128 Filtering Data from Spark Data Frames - Task 3 - Boolean AND
Lecture 129 Filtering Data from Spark Data Frames - Task 4 - IN Operator
Lecture 130 Filtering Data from Spark Data Frames - Task 5 - Between and Like
Lecture 131 Filtering Data from Spark Data Frames - Task 6 - Using functions in Filter
Lecture 132 Overview of Aggregations using Spark Data Frame APIs
Lecture 133 Overview of Sorting using Spark Data Frame APIs
Lecture 134 Solution - Get Delayed Counts using Spark Data Frame APIs - Part 1
Lecture 135 Solution - Get Delayed Counts using Spark Data Frame APIs - Part 2
Lecture 136 Solution - Getting Delayed Counts By Date using Spark Data Frame APIs
Section 11: Apache Spark 2 using Scala - Joining Data Sets
Lecture 137 Prepare and Validate Data Sets
Lecture 138 Starting Spark Session or Spark Context
Lecture 139 Analyze Data Sets for Joins using Spark Data Frame APIs
Lecture 140 Eliminate Duplicate records from Data Frame using Spark Data Frame APIs
Lecture 141 Recap of Basic Transformations using Spark Data Frame APIs
Lecture 142 Joining Data Sets using Spark Data Frame APIs - Problem Statements
Lecture 143 Overview of Joins using Spark Data Frame APIs
Lecture 144 Inner Join using Spark Data Fr - Get number of flights departed from US airports
Lecture 145 Inner Join using Spark Data Fram - Get number of flights departed from US States
Lecture 146 Outer Join using Spark Data Frame APIs - Get Aiports - Never Used
Section 12: Apache Spark using SQL - Getting Started
Lecture 147 Getting Started with Spark SQL - Overview
Lecture 148 Overview of Spark Documentation
Lecture 149 Launching and using Spark SQL CLI
Lecture 150 Overview of Spark SQL Properties
Lecture 151 Running OS Commands using Spark SQL
Lecture 152 Understanding Spark Metastore Warehouse Directory
Lecture 153 Managing Spark Metastore Databases
Lecture 154 Managing Spark Metastore Tables
Lecture 155 Retrieve Metadata of Spark Metastore Tables
Lecture 156 Role of Spark Metastore or Hive Metastore
Lecture 157 Exercise - Getting Started with Spark SQL
Section 13: Apache Spark using SQL - Basic Transformations
Lecture 158 Basic Transformation using Spark SQL - Introduction
Lecture 159 Spark SQL - Overview
Lecture 160 Define Problem Statement for Basic Transformations using Spark SQL
Lecture 161 Prepare or Create Tables using Spark SQL
Lecture 162 Projecting or Selecting Data using Spark SQL
Lecture 163 Filtering Data using Spark SQL
Lecture 164 Joining Tables using Spark SQL - Inner
Lecture 165 Joining Tables using Spark SQL - Outer
Lecture 166 Aggregating Data using Spark SQL
Lecture 167 Sorting Data using Spark SQL
Lecture 168 Conclusion - Final Solution using Spark SQL
Section 14: Apache Spark using SQL - Basic DDL and DML
Lecture 169 Introduction to Basic DDL and DML using Spark SQL
Lecture 170 Create Spark Metastore Tables using Spark SQL
Lecture 171 Overview of Data Types for Spark Metastore Table Columns
Lecture 172 Adding Comments to Spark Metastore Tables using Spark SQL
Lecture 173 Loading Data Into Spark Metastore Tables using Spark SQL - Local
Lecture 174 Loading Data Into Spark Metastore Tables using Spark SQL - HDFS
Lecture 175 Loading Data into Spark Metastore Tables using Spark SQL - Append and Overwrite
Lecture 176 Creating External Tables in Spark Metastore using Spark SQL
Lecture 177 Managed Spark Metastore Tables vs External Spark Metastore Tables
Lecture 178 Overview of Spark Metastore Table File Formats
Lecture 179 Drop Spark Metastore Tables and Databases
Lecture 180 Truncating Spark Metastore Tables
Lecture 181 Exercise - Managed Spark Metastore Tables
Section 15: Apache Spark using SQL - DML and Partitioning
Lecture 182 Introduction to DML and Partitioning of Spark Metastore Tables using Spark SQL
Lecture 183 Introduction to Partitioning of Spark Metastore Tables using Spark SQL
Lecture 184 Creating Spark Metastore Tables using Parquet File Format
Lecture 185 Load vs. Insert into Spark Metastore Tables using Spark SQL
Lecture 186 Inserting Data using Stage Spark Metastore Table using Spark SQL
Lecture 187 Creating Partitioned Spark Metastore Tables using Spark SQL
Lecture 188 Adding Partitions to Spark Metastore Tables using Spark SQL
Lecture 189 Loading Data into Partitioned Spark Metastore Tables using Spark SQL
Lecture 190 Inserting Data into Partitions of Spark Metastore Tables using Spark SQL
Lecture 191 Using Dynamic Partition Mode to insert data into Spark Metastore Tables
Lecture 192 Exercise - Partitioned Spark Metastore Tables using Spark SQL
Section 16: Apache Spark using SQL - Pre-defined Functions
Lecture 193 Introduction - Overview of Spark SQL Functions
Lecture 194 Overview of Pre-defined Functions using Spark SQL
Lecture 195 Validating Functions using Spark SQL
Lecture 196 String Manipulation Functions using Spark SQL
Lecture 197 Date Manipulation Functions using Spark SQL
Lecture 198 Overview of Numeric Functions using Spark SQL
Lecture 199 Data Type Conversion using Spark SQL
Lecture 200 Dealing with Nulls using Spark SQL
Lecture 201 Using CASE and WHEN using Spark SQL
Lecture 202 Query Example - Word Count using Spark SQL
Section 17: Apache Spark using SQL - Pre-defined Functions - Exercises
Lecture 203 Prepare Users Table using Spark SQL
Lecture 204 Exercise 1 - Get number of users created per year
Lecture 205 Exercise 2 - Get the day name of the birth days of users
Lecture 206 Exercise 3 - Get the names and email ids of users added in the year 2019
Lecture 207 Exercise 4 - Get the number of users by gender
Lecture 208 Exercise 5 - Get last 4 digits of unique ids
Lecture 209 Exercise 6 - Get the count of users based up on country code
Section 18: Apache Spark using SQL - Windowing Functions
Lecture 210 Introduction to Windowing Functions using Spark SQL
Lecture 211 Prepare HR Database in Spark Metastore using Spark SQL
Lecture 212 Overview of Windowing Functions using Spark SQL
Lecture 213 Aggregations using Windowing Functions using Spark SQL
Lecture 214 LEAD or LAG Functions using Spark SQL
Lecture 215 Getting first and last values using Spark SQL
Lecture 216 Ranking using Windowing Functions in Spark SQL
Lecture 217 Order of execution of Spark SQL Queries
Lecture 218 Overview of Subqueries using Spark SQL
Lecture 219 Filtering Window Function Results using Spark SQL
Section 19: Sample scenarios with solutions
Lecture 220 Introduction to Sample Scenarios and Solutions
Lecture 221 Problem Statements - General Guidelines
Lecture 222 Initializing the job - General Guidelines
Lecture 223 Getting crime count per type per month - Understanding Data
Lecture 224 Getting crime count per type per month - Implementing the logic - Core API
Lecture 225 Getting crime count per type per month - Implementing the logic - Data Frames
Lecture 226 Getting crime count per type per month - Validating Output
Lecture 227 Get inactive customers - using Core Spark API (leftOuterJoin)
Lecture 228 Get inactive customers - using Data Frames and SQL
Lecture 229 Get top 3 crimes in RESIDENCE - using Core Spark API
Lecture 230 Get top 3 crimes in RESIDENCE - using Data Frame and SQL
Lecture 231 Convert NYSE data from text file format to parquet file format
Lecture 232 Get word count - with custom control arguments, num keys and file format
Any IT aspirant/professional willing to learn Data Engineering using Apache Spark,Python Developers who want to learn Spark using Scala to add additional skill to be a Data Engineer,Java or Scala Developers to learn Spark using Scala to add Data Engineering Skills to their profile


Homepage

https://www.udemy.com/course/cca-175-spark-and-hadoop-developer-certification-scala/