LSD's Platform as a Service Software Stack
Core Services
This platform provides a Apache Hadoop Stack. The two main components of this stack are HDFS and YARN.
Hadoop Distributed File System(HDFS)(version 3.1.1)
HDFS is a distributed file system spanning a cluster of machines. Each machine contributes to the filesystem storage capacity by storing some of the data locally. Each file stored on HDFS is split into blocks of size typically (but configurable) 128MB or 256MB. Those blocks are spread throughout the cluster and replicated 3 times each, providing fault-tolerance and read scalability.
YARN(version 3.1.1)
While HDFS provides user with access to disk capacity, YARN manages RAM and CPU resources. Applications running on a YARN-managed hadoop cluster access computing machine by asking resources to YARN. Hadoop MapReduce and Apache Spark are typical application running on top of YARN.
A key characteristics of this Apache Hadoop is that Yarn can provide access to computing ressources on the hardware where data is stored, allowing the process to treat data locally. The idea is thus to send the computation to the data rather than the other way around.
Hadoop MapReduce(version 3.1.1)
MapReduce is a framework running on top of YARN built to process large data in a distributed manner.
Additional Services and Libraries
Apache Spark(version 2.3.1 and 3.1.1)
Like MapReduce, Spark is a framework designed to distributively process large amout of data. It features, among other things, perfomances improvement compared to MapReduce, the availibity to process mini-batch streaming and a machine learning library.
Apache HBase(version 2.0.0)
HBase is non-relational distributed data store using HDFS as a storage back-end.
Apache Solr(version 7.4.0)
Apache Solr is a distributed index, based on the Lucene indexing library, featuring on demand full-text search capabilities as well as faceting (aggregations) similar to ElasticSearch (which is also based on lucene). It comes with the visualization tool Banana which is a port of ElasticSearch's Kibana, for creating dashboard to visualise data indexed in Solr.
Intel BigDL(version 2.2.0)
BigDL is a library built on top of Apache Spark to run Deep Learning applications on Hadoop infrastructure.
Apache Kafka(version 1.1.1)
Apache Kafka is a distributed stream-processing system used to scalabily publish and consume records in realtime streams.
Apache Hive(version 3.1.0)
The Apache Hive ™ data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. Structure can be projected onto data already in storage. A command line tool and JDBC driver are provided to connect users to Hive. — Apache Hive Website
Useful links
These ares the WebUIs of services running on the PaaS.
| Service | WebUIs |
|---|---|
| HDFS | here or here: Filesytem WebUI |
| YARN | here or here: ResourceManager WebUi (see applications, their logs...) |
| SparkHistory | here: History of Spark applications and their tasks |
| Solr Banana | any of here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here: SolrBanana User Interface |
| HBASE UI | here or here: HBase WebUI |
| HBASE REST API | here or here HBASE REST API |
| KNOX PROXY HOMEPAGE | here: Homepage of the proxy to the other WebUIs |
The knox proxy asks for the user kerberos password (the one used with kinit) to authenticate to the proxied webui accessed through it.