A curated list of awesome Apache Spark packages and resources.
Apache Spark is an open-source cluster-computing framework. Originally developed at the University of California, Berkeley’s AMPLab, the Spark codebase was later donated to the Apache Software Foundation, which has maintained it since. Spark provides an interface for programming entire clusters with implicit data parallelism and fault-tolerance (Wikipedia 2017).
Users of Apache Spark may choose between different the Python, R, Scala and Java programming languages to interface with the Apache Spark APIs.
- Clojure DSL.
- C# bindings (Deprecated in favor of .NET for Apache Spark).
- .NET bindings.
- An alternative R backend, using
dplyr.
- Haskell on Apache Spark.
- A scala kernel for Jupyter.
- Web-based notebook that enables interactive data analytics with
plugable backends, integrated plotting, and extensive Spark support
out-of-the-box.
- Polynote: an IDE-inspired polyglot notebook. It supports mixing
multiple languages in one notebook, and sharing data between them
seamlessly. It encourages reproducible notebooks with its immutable data
model. Orginating from
Netflix.
- Scalable and stable Scala and Spark focused notebook bridging the gap
between JVM and Data Scientists (incl. extendable, typesafe and reactive
charts).
- Jupyter magics and kernels for
working with remote Spark clusters, for interactively working with
remote Spark clusters through
Livy, in Jupyter
notebooks.
- Support for efficient queries on compressed data.
- A library that brings useful functions from modern database management
systems to Apache Spark.
- A Scala library with essential Spark functions and extensions to make
you more productive.
- A native PySpark implementation of spark-daria.
- A library of general purpose functions and UDF’s.
SparkSQL has
serveral built-in Data Sources
for files. These include csv, json,
parquet, orc, and avro. It also
supports JDBC databases as well as Apache Hive. Additional data sources
can be added by including the packages listed below, or writing your own.
- CSV reader and writer (obsolete since Spark 2.0
[SPARK-12833]).
- Apache Avro reader and writer
(obselete since Spark 2.4
[SPARK-24768]).
- XML parser and writer.
- MongoDB reader and writer.
- Cassandra support including data source and API and support for
arbitrary queries.
- Riak TS & Riak KV connector.
- Official MongoDB connector.
- Official OrientDB connector.
- Storage layer with ACID transactions.
- Geospatial analytics using Spark.
- Cluster computing system for processing large-scale spatial data.
- Scala / Java / Python library for interacting with time series data on
Apache Spark.
- A time series library for Apache Spark.
- Graph analytics platform on top of Neo4j and GraphX.
- Data frame based graph API.
- Bolt protocol based, Neo4j Connector with RDD, DataFrame and GraphX /
GraphFrames support.
- Library extending GraphX features with multiple functionalities useful
in graph analytics (measures, generators, link prediction etc.).
Scala and Spark API to benchmark and analyse clustering algorithms on
any vectorization you can generate
- An Implementation of the DBSCAN clustering algorithm on top of Apache
Spark by irvingc and based on
the paper from He, Yaobin, et al. MR-DBSCAN: a scalable MapReduce-based DBSCAN algorithm for heavily
skewed data.
- Declarative machine learning framework on top of Spark.
- Scikit-learn integration with distributed model training.
- PMML transformer library for Spark ML.
- Distributed deep learning framework with PySpark and Keras.
- A system to manage machine learning models for
- Set of tools designed to analyse genomics data.
- Genetic analysis framework.
spark.ml and
scikit-learn
.
- H2O interoperability layer.
- Distributed Deep Learning library.
- Execution engine and serialization format which supports deployment of
o.a.s.ml models without dependency on
SparkSession.
- REST server with extensive language support (Python, R, Scala),
ability to maintain interactive sessions and object sharing.
- Simple Spark as a Service which supports objects sharing using so
called named objects. JVM only.
- Service for exposing Spark analytical jobs and machine learning models
as realtime, batch or reactive web services.
- IPython protocol based middleware for interactive applications.
- Improved implementation of Thrift JDBC/ODBC Server.
- Cross-platform monitoring tool (Spark UI / Spark History Server
replacement).
- Collection of tools varying from ML extensions to additional RDD
methods.
- Helpers & syntactic sugar for PySpark.
- Static type annotations for PySpark (obsolete since Spark 3.1. See
SPARK-32681).
- A command-line tool for launching Spark clusters on EC2.
- Data Cleansing and Exploration utilities with the goal of simplifying
data cleaning.
- DataFrame wrapper for
Stanford CoreNLP.
- Natural language processing library built on top of Apache Spark ML.
- Collection of the streaming connectors excluded from Spark 2.0 (Akka,
MQTT, Twitter. ZeroMQ).
- Unified data processing engine supporting both batch and streaming
applications. Apache Spark is one of the supported execution
environments.
- Interface for querying larger than memory datasets using Pandas-like
syntax. It supports both Spark DataFrames and
RDDs.
- Pandas DataFrame API on top of Apache Spark.
- Deequ is a library built on top of Apache Spark for defining “unit
tests for data”, which measure data quality in large datasets.
- Collection of base test classes.
- A lightweight and fast testing framework.
- Open-source toolkit for analyzing web archives.
- Workflow management system with
Spark backend.
Wikipedia. 2017. “Apache Spark — Wikipedia, the Free Encyclopedia.” https://en.wikipedia.org/w/index.php?title=Apache_Spark&oldid=781182753.
This work (Awesome Spark, by
https://github.com/awesome-spark/awesome-spark), identified by
Maciej Szymkiewicz, is free of known copyright restrictions.
Apache Spark, Spark, Apache, and the Spark logo are trademarks of The Apache Software Foundation. This compilation is not endorsed by The Apache Software Foundation.
Inspired by sindresorhus/awesome.