{"id":529,"date":"2021-07-15T07:02:36","date_gmt":"2021-07-15T07:02:36","guid":{"rendered":"http:\/\/www.wizbrand.com\/tutorials\/?p=529"},"modified":"2022-04-26T02:28:20","modified_gmt":"2022-04-26T02:28:20","slug":"introduction-of-scala-with-spark","status":"publish","type":"post","link":"https:\/\/www.wizbrand.com\/tutorials\/introduction-of-scala-with-spark\/","title":{"rendered":"Introduction of Scala with Spark"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">To conclude this introduction to Spark, a sample Scala application &#8211; wordcount over tweets is provided, it is developed in the Scala API. The undelying technology that gives spark its capability. Data Processing APIs that exists in Spark. A knwoledge of how to work with the Data Processing APIs.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Scala<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Scala stands for Scalable Language. It is a multi-paradigm programming language. Scala language includes features of functional programming and object-oriented programming. It is a statically typed language. Its source code is compiled into bytecode and executed by Java virtual machine (JVM).<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Spark<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Apache spark this exclusive open-source, distributed processing system used for big data workloads. It utilizes in-memory caching and optimized query execution for fast queries against data of any size. Simply Put, spark the A Fast and General engines for large-scale data processing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Pre-requisites<\/strong><\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Scala Programming is based on Java, so if you are aware of Java syntax, then it&#8217;s pretty easy to learn Scala. Further if you do not have expertise in Java but if you know any other programming language like C, C++ or Python then it will also help in grasping Scala concepts very quickly.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Agenda of\u00a0Scala with Spark<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Introduction &#8211;\u00a0<em>Scala<\/em><\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>Why Learn Scala?<\/li><li>What You Will Learn<\/li><li>Installation<\/li><li>Scala REPL<\/li><li>Scala IntelliJ IDEA<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Functional Programming Concepts<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>Immutability<\/li><li>Expressions vs. Statements<\/li><li>Functions<\/li><li>Pure and Impure Functions<\/li><li>Referential Transparency<\/li><li>Higher-Order Functions<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>First Interaction with Scala<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>REPL<\/li><li>Variables and Values<\/li><li>Conditionals<\/li><li>Loops<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Functions<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>Structure of a Function<\/li><li>Anonymous Functions<\/li><li>Higher-Order Functions<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Classes<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>Structure of Classes<\/li><li>Companion Objects<\/li><li>Creating Objects Using Apply<\/li><li>Case Classes<\/li><li>Launching Standalone Programs Extending App<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Null Checks and Error Handling<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>Why Null Checks Are Bad<\/li><li>Option as a Better Alternative<\/li><li>Error Handling in Imperative Languages<\/li><li>Error Handling with Try<\/li><li>Error Handling with Either<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Pattern Matching<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>What Is It?<\/li><li>Match on Constants<\/li><li>Match on Case Classes<\/li><li>Match on Sequences<\/li><li>Match on Type Only<\/li><li>Guarding Your Match<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Collections<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>Benefits of Scala Collections Library<\/li><li>Mutable and Immutable<\/li><li>Class Hierarchy Diagram<\/li><li>List with Examples<\/li><li>Set with Examples<\/li><li>Map with Examples<\/li><li>Methods on Numeric Collections<\/li><li>Filtering, Size, and Conversion Operations on Collections<\/li><li>Transforming a Collection Using Map<\/li><li>Transforming a Collection Using FlatMap<\/li><li>Working with Option and FlatMap<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Concurrency<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>Concurrency and Parallelism<\/li><li>Future and ExecutionContext<\/li><li>Futures Transformation<\/li><li>Filtering and Collecting Futures<\/li><li>Other Ways to Model Asynchronous Operation<\/li><li>Dealing with Future Failures<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Understanding Object-oriented Scala<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>Understanding Classes and Objects<\/li><li>Creating Classes and Objects<\/li><li>Making Objects Immutable<\/li><li>Understanding Singleton Objects<\/li><li>Understanding Functional Objects<\/li><li>Understanding Abstract Classes, Inheritance, and Composition<\/li><li>session_start()<\/li><li>Project Demo<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Working with Basic Types<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>Introduction to Scala Types<\/li><li>Embedding Scala Expressions in String Literals<\/li><li>Using Methods as Operators<\/li><li>Understanding Scala Class Hierarchy<\/li><li>Project Demo<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Working with Functions and Closures<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>Understanding Local Functions<\/li><li>Understanding Function Literals and Function Values<\/li><li>Understanding Partially Applied Functions<\/li><li>Understanding Closures<\/li><li>Applying Repeated Arguments to Functions<\/li><li>Understanding Named Arguments and Default Parameter Values<\/li><li>Understanding Tail Recursion<\/li><li>Project Demo<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Creating Your Own Control Structures<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>Using Function Value to Remove Code Duplication<\/li><li>Understanding Currying<\/li><li>Creating Control Abstraction<\/li><li>Understanding by Named Parameters<\/li><li>Project Demo<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Working with Traits<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>Understading Traits<\/li><li>Developing Rich Interfaces with Traits<\/li><li>Developing Stackable Modifications with Traits<\/li><li>Understanding When to Use Traits or Not<\/li><li>Project Demo<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Understanding Collections<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>Understanding How Reduce Works<\/li><li>Understanding How Fold Operations Work<\/li><li>Project Demo<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Working with Implicits<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Overview<\/li><li>Understanding How Implicit Conversion Works<\/li><li>Understanding the Rules for Implicits<\/li><li>Working with Application of Implicits<\/li><li>Project Demo<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Introduction &#8211;\u00a0<em>Spark<\/em><\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Why Spark?<\/li><li>Hadoop Explosion to Spark Unification<\/li><li>Spark&#8217;s Background<\/li><li>Installation<\/li><li>Spark Programming Languages<\/li><li>Hello Big Data!<\/li><li>Logistics<\/li><li>Resources<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Spark Core: Part 1<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Intro<\/li><li>Spark Appification<\/li><li>What Is an RDD?<\/li><li>Loading Data<\/li><li>Lambdas<\/li><li>Transforming Data<\/li><li>More Transformations<\/li><li>Actions and the Associative Property<\/li><li>Acting on Data<\/li><li>Persistence<\/li><li>Resources<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Spark Core: Part 2<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Intro<\/li><li>Implicit Conversions<\/li><li>Key-Value Methods<\/li><li>Caching Data<\/li><li>Accumulating Data<\/li><li>Java in Spark<\/li><li>Resources<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Distribution and Instrumentation<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Intro<\/li><li>Spark Submit<\/li><li>Cluster Management<\/li><li>Standalone Cluster Scripts<\/li><li>AWS Setup<\/li><li>Spark on Yarn in EMR<\/li><li>Spark UI<\/li><li>Resources<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Spark Libraries<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Intro<\/li><li>Spark SQL<\/li><li>Spark SQL Demo<\/li><li>Spark SQL Demo &#8211; The SQL Side<\/li><li>Streaming<\/li><li>Streaming Demo<\/li><li>Machine Learning<\/li><li>Machine Learning Demo<\/li><li>GraphX<\/li><li>GraphX Demo<\/li><li>Resources<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Optimizations and the Future<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Intro<\/li><li>Closures<\/li><li>Broadcasting<\/li><li>Optimizing Partitioning<\/li><li>Spark&#8217;s Future<\/li><li>Resources<\/li><li>Summary<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><a><strong>Why Scala skill is essential for every Software Engineer?<\/strong><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Scala collections library, to get inside and power provided by the language to us as programmers to use.<\/li><li>Concurrency to help you gain other and better ways to write asynchronous code which behaves in a predictable manner.<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you are looking for master in Scala with Spark, I&#8217;ll suggest you <a href=\"https:\/\/www.devopsschool.com\/certification\/master-in-scala-with-spark.html\">DevOpsSchool<\/a>.com it is a good institute of India<\/p>\n","protected":false},"excerpt":{"rendered":"<p>To conclude this introduction to Spark, a sample Scala application &#8211; wordcount over tweets is provided, it is developed in [&hellip;]<\/p>\n","protected":false},"author":19,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[254],"class_list":["post-529","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-scala-spark-software-engineer-sql"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.wizbrand.com\/tutorials\/wp-json\/wp\/v2\/posts\/529","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wizbrand.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wizbrand.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wizbrand.com\/tutorials\/wp-json\/wp\/v2\/users\/19"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wizbrand.com\/tutorials\/wp-json\/wp\/v2\/comments?post=529"}],"version-history":[{"count":1,"href":"https:\/\/www.wizbrand.com\/tutorials\/wp-json\/wp\/v2\/posts\/529\/revisions"}],"predecessor-version":[{"id":530,"href":"https:\/\/www.wizbrand.com\/tutorials\/wp-json\/wp\/v2\/posts\/529\/revisions\/530"}],"wp:attachment":[{"href":"https:\/\/www.wizbrand.com\/tutorials\/wp-json\/wp\/v2\/media?parent=529"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wizbrand.com\/tutorials\/wp-json\/wp\/v2\/categories?post=529"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wizbrand.com\/tutorials\/wp-json\/wp\/v2\/tags?post=529"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}