LSD's Platform as a Service Software Stack

Core Services

This platform provides a Apache Hadoop Stack. The two main components of this stack are HDFS and YARN.

Hadoop Distributed File System(HDFS)(version 3.1.1)

HDFS is a distributed file system spanning a cluster of machines. Each machine contributes to the filesystem storage capacity by storing some of the data locally. Each file stored on HDFS is split into blocks of size typically (but configurable) 128MB or 256MB. Those blocks are spread throughout the cluster and replicated 3 times each, providing fault-tolerance and read scalability.

YARN(version 3.1.1)

While HDFS provides user with access to disk capacity, YARN manages RAM and CPU resources. Applications running on a YARN-managed hadoop cluster access computing machine by asking resources to YARN. Hadoop MapReduce and Apache Spark are typical application running on top of YARN.
A key characteristics of this Apache Hadoop is that Yarn can provide access to computing ressources on the hardware where data is stored, allowing the process to treat data locally. The idea is thus to send the computation to the data rather than the other way around.

Hadoop MapReduce(version 3.1.1)

MapReduce is a framework running on top of YARN built to process large data in a distributed manner.

Additional Services and Libraries

Apache Spark(version 2.3.1 and 3.1.1)

Like MapReduce, Spark is a framework designed to distributively process large amout of data. It features, among other things, perfomances improvement compared to MapReduce, the availibity to process mini-batch streaming and a machine learning library.

Apache HBase(version 2.0.0)

HBase is non-relational distributed data store using HDFS as a storage back-end.

Apache Solr(version 7.4.0)

Apache Solr is a distributed index, based on the Lucene indexing library, featuring on demand full-text search capabilities as well as faceting (aggregations) similar to ElasticSearch (which is also based on lucene). It comes with the visualization tool Banana which is a port of ElasticSearch's Kibana, for creating dashboard to visualise data indexed in Solr.

Intel BigDL(version 2.2.0)

BigDL is a library built on top of Apache Spark to run Deep Learning applications on Hadoop infrastructure.

Apache Kafka(version 1.1.1)

Apache Kafka is a distributed stream-processing system used to scalabily publish and consume records in realtime streams.

Apache Hive(version 3.1.0)

The Apache Hive ™ data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. Structure can be projected onto data already in storage. A command line tool and JDBC driver are provided to connect users to Hive. — Apache Hive Website

These ares the WebUIs of services running on the PaaS.

Service WebUIs
HDFS here or here: Filesytem WebUI
YARN here or here: ResourceManager WebUi (see applications, their logs...)
SparkHistory here: History of Spark applications and their tasks
Solr Banana any of here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here or here: SolrBanana User Interface
HBASE UI here or here: HBase WebUI
HBASE REST API here or here HBASE REST API
KNOX PROXY HOMEPAGE here: Homepage of the proxy to the other WebUIs

These WebUIs are accessible using the gateway server's firefox browser using ssh -X option for X11 forwarding or your machine's browser throught a configuration described here.

The knox proxy asks for the user kerberos password (the one used with kinit) to authenticate to the proxied webui accessed through it (no browser configuration is needed to access the webui through that proxy).