To conclude this introduction to Spark, a sample Scala application – wordcount over tweets is provided, it is developed in the Scala API. The undelying technology that gives spark its capability. Data Processing APIs that exists in Spark. A knwoledge of how to work with the Data Processing APIs.
Scala
Scala stands for Scalable Language. It is a multi-paradigm programming language. Scala language includes features of functional programming and object-oriented programming. It is a statically typed language. Its source code is compiled into bytecode and executed by Java virtual machine (JVM).
Spark
Apache spark this exclusive open-source, distributed processing system used for big data workloads. It utilizes in-memory caching and optimized query execution for fast queries against data of any size. Simply Put, spark the A Fast and General engines for large-scale data processing.
Scala Programming is based on Java, so if you are aware of Java syntax, then it’s pretty easy to learn Scala. Further if you do not have expertise in Java but if you know any other programming language like C, C++ or Python then it will also help in grasping Scala concepts very quickly.
Agenda of Scala with Spark
- Overview
- Why Learn Scala?
- What You Will Learn
- Installation
- Scala REPL
- Scala IntelliJ IDEA
- Summary
Functional Programming Concepts
- Overview
- Immutability
- Expressions vs. Statements
- Functions
- Pure and Impure Functions
- Referential Transparency
- Higher-Order Functions
- Summary
- Overview
- REPL
- Variables and Values
- Conditionals
- Loops
- Summary
- Overview
- Structure of a Function
- Anonymous Functions
- Higher-Order Functions
- Summary
- Overview
- Structure of Classes
- Companion Objects
- Creating Objects Using Apply
- Case Classes
- Launching Standalone Programs Extending App
- Summary
Null Checks and Error Handling
- Overview
- Why Null Checks Are Bad
- Option as a Better Alternative
- Error Handling in Imperative Languages
- Error Handling with Try
- Error Handling with Either
- Summary
- Overview
- What Is It?
- Match on Constants
- Match on Case Classes
- Match on Sequences
- Match on Type Only
- Guarding Your Match
- Summary
- Overview
- Benefits of Scala Collections Library
- Mutable and Immutable
- Class Hierarchy Diagram
- List with Examples
- Set with Examples
- Map with Examples
- Methods on Numeric Collections
- Filtering, Size, and Conversion Operations on Collections
- Transforming a Collection Using Map
- Transforming a Collection Using FlatMap
- Working with Option and FlatMap
- Summary
- Overview
- Concurrency and Parallelism
- Future and ExecutionContext
- Futures Transformation
- Filtering and Collecting Futures
- Other Ways to Model Asynchronous Operation
- Dealing with Future Failures
- Summary
Understanding Object-oriented Scala
- Overview
- Understanding Classes and Objects
- Creating Classes and Objects
- Making Objects Immutable
- Understanding Singleton Objects
- Understanding Functional Objects
- Understanding Abstract Classes, Inheritance, and Composition
- session_start()
- Project Demo
- Overview
- Introduction to Scala Types
- Embedding Scala Expressions in String Literals
- Using Methods as Operators
- Understanding Scala Class Hierarchy
- Project Demo
- Summary
Working with Functions and Closures
- Overview
- Understanding Local Functions
- Understanding Function Literals and Function Values
- Understanding Partially Applied Functions
- Understanding Closures
- Applying Repeated Arguments to Functions
- Understanding Named Arguments and Default Parameter Values
- Understanding Tail Recursion
- Project Demo
- Summary
Creating Your Own Control Structures
- Overview
- Using Function Value to Remove Code Duplication
- Understanding Currying
- Creating Control Abstraction
- Understanding by Named Parameters
- Project Demo
- Summary
- Overview
- Understading Traits
- Developing Rich Interfaces with Traits
- Developing Stackable Modifications with Traits
- Understanding When to Use Traits or Not
- Project Demo
- Summary
- Overview
- Understanding How Reduce Works
- Understanding How Fold Operations Work
- Project Demo
- Summary
- Overview
- Understanding How Implicit Conversion Works
- Understanding the Rules for Implicits
- Working with Application of Implicits
- Project Demo
- Summary
- Why Spark?
- Hadoop Explosion to Spark Unification
- Spark’s Background
- Installation
- Spark Programming Languages
- Hello Big Data!
- Logistics
- Resources
- Summary
- Intro
- Spark Appification
- What Is an RDD?
- Loading Data
- Lambdas
- Transforming Data
- More Transformations
- Actions and the Associative Property
- Acting on Data
- Persistence
- Resources
- Summary
- Intro
- Implicit Conversions
- Key-Value Methods
- Caching Data
- Accumulating Data
- Java in Spark
- Resources
- Summary
Distribution and Instrumentation
- Intro
- Spark Submit
- Cluster Management
- Standalone Cluster Scripts
- AWS Setup
- Spark on Yarn in EMR
- Spark UI
- Resources
- Summary
- Intro
- Spark SQL
- Spark SQL Demo
- Spark SQL Demo – The SQL Side
- Streaming
- Streaming Demo
- Machine Learning
- Machine Learning Demo
- GraphX
- GraphX Demo
- Resources
- Summary
- Intro
- Closures
- Broadcasting
- Optimizing Partitioning
- Spark’s Future
- Resources
- Summary
Why Scala skill is essential for every Software Engineer?
- Scala collections library, to get inside and power provided by the language to us as programmers to use.
- Concurrency to help you gain other and better ways to write asynchronous code which behaves in a predictable manner.
If you are looking for master in Scala with Spark, I’ll suggest you DevOpsSchool.com it is a good institute of India
