This is a based on playframwork for submit spark app

Docker with Airflow + Postgres + Spark cluster + JDK (spark-submit support) + Jupyter Notebooks

Using python3.6 alpine base image adds java,pandas, numpy,pyspark and spark as rundeps. This image can be used as container image when you run spark-submit on k8.

Processing IT Recruitment Data in HDFS Cluster, Spark, Elasticsearch and Kibana, deployed by Docker compose

sbt plugin for spark-submit

Steps to deploy a Spark app to Kubernetes cluster using spark-submit or a pod template

Kronos extensions for storage, queue and handlers.

scripts, configurations, resources, and others

Run a simple spark word count job via 1) scala sbt spark 2) docker

An easy to use script that automatically adds files to the spark-submit command.

A generic, multimodule framework for building scalable ETL (Extract, Transform, Load) processes using Apache Spark and Java. This framework is designed to be cloud-agnostic, maintainable, testable, and near-production-ready.

GDELT Fu-Fu

The primary objective of this study is to explore the feasibility of using machine learning algorithms to classify health insurance plans based on their coverage for routine dental services. To achieve this, I used six different classification algorithms: LR, DT, RF, GBT, SVM, FM(Tech: PySpark, SQL, Databricks, Zeppelin books, Hadoop, Spark-Submit)

PySpark-based ETL pipeline that extracts transaction data from a MySQL database, cleans and transforms it, aggregates monthly sales per customer, and writes the processed data to an S3 bucket in Parquet format.

Running Python Engg file with Spark-submit

Apache Spark and Apache Kafka integration and Spark Analytics over DataFrame.

Movie Recommendation using Apache Spark MLlib

Spatial Hotspot Analysis on Geo-Spatial Data using Apache Spark and Scala

A lightweight and ready-to-use Apache Spark environment for Windows development

A machine learning pipeline using PySpark to predict flight arrival delays based on departure delays, timing, and aircraft metadata from the 2007 US DOT dataset.