Rapidweaver Uploads Some Files Then Stops Communicating With Ftp Server

Distributed-file-system-server-with-RAFT

License: MIT

INTRODUCTION

The project Snapchat is aimed at enabling people to connect, share, explore, and interact exclusively through pictures. Snapchat allows non‐terminate and guaranteed admission to the user. The system is enforced to assemble, store and stream pictures by building a distributed file system that provides scalable client operations. The fundamental task is to allow the clients to communicate with each other by transferring images to each other. This is achieved through enabling the client with the features of upload and download of the images to and from the server.

GOAL OF SNAPCHAT

Snapchat is designed to provide a highly scalable system that supports massively parallel processing of customer requests and highly connected data sets. The system implements the Raft algorithm that substantially elects a leader to perform tasks such every bit delegating tasks to other servers in the cluster and perform log replication. The pattern ensures balanced distribution of work.

COMMUNICATION BACKBONE

Snapchat requires flexible network to support different client operations that are cooperative in nature. This arrangement provides the quickest possible communication inside an unrestricted network topology. Thus the group of clusters are enabled to communicate and cooperate with one another and solve problems in a collaborative manner.

TECHNOLOGIES USED

Programming languages
- Customer Side : Java
- Server Side : Core Java (Netty API)
FTP Server
- The File Transfer Protocol (FTP) is a standard network protocol used to transfer computer files
Netty API
- Netty is a Non‐Blocking Client‐Server Framework in Java .Its built over an API for Channel programming (An abstraction higher up the usual Socket programming) and thus imparts its not‐blocking nature to Netty).
- In addition to that the Concurrency model of cadre Coffee is coupled with Netty API for parallely listening on ports and separately serving the requests.
Google ProtoBuf
- Information technology is a multi‐platform data inter‐exchange format, compatible with Coffee, Python, C++. Data period over the network is beingness taken care of past the Protobuf with development being done over their respective interfaces causeless as beans.

FEATURES OF THE SOLUTION

Nosotros processed 10K files in 98 seconds with a leader node distributing jobs among iii follower nodes. The distributed file system implements Raft Algorithm. We had 2 clients, each connected to ane server and an FTP server. The client sends the file transfer request to a node in the cluster. The server so uploads the file to the FTP server and checks if it is the leader and if information technology is, it broadcasts the request and the logs to all the other nodes in the cluster. If it is non the leader, it frontwards the request to the leader and and then the leader broadcasts the request and logs. The node that is attached to the receiver customer then downloads the prototype file from the FTP server and sends it to the receiver customer. The leader ensures the log of the images is maintained on all the servers in a synchronized manner.

Raft Algorithm

A consensus algorithm is: a cluster of servers should record a series of records ("log entries") in response to requests from clients of the cluster. (It may too take activeness based on those entries.) It does so in a way that guarantees that the responses seen past clients of the cluster will exist consistent EVEN in the face of servers crashing in unpredictable ways (simply non losing data that was synced to disk), and networks introducing unpredictable delays or communication blockages.

Raft works every bit follows. First, information technology elects a leader, then the leader records the master version of the log, telling other cluster servers what's in that master tape and "committing" a log entry and so responding to the client of the cluster to admit that entry only when more than one-half the cluster has recorded a given entry. That works unless the leader crashes or loses communication with too many others; in such a case Raft elects a new leader. The ballot process is designed to guarantee that whatever newly elected leader will have (at to the lowest degree) all of the already-committed entries.

Raft nodes are always in one of three states: follower, candidate or leader. All nodes initially start out as a follower. In this state, nodes can accept log entries from a leader and cast votes. If no entries are received for some time, nodes self-promote to the candidate state. In the candidate country nodes request votes from their peers. If a candidate receives a quorum of votes, then it is promoted to a leader. The leader must accept new log entries and replicate to all the other followers. In improver, if dried reads are not adequate, all queries must also be performed on the leader.

Once a cluster has a leader, it is able to accept new log entries. A client tin can request that a leader suspend a new log entry, which is an opaque binary blob to Raft. The leader and so writes the entry to durable storage and attempts to replicate to a quorum of followers. In one case the log entry is considered committed, information technology can exist applied to a finite land motorcar. The finite country motorcar is awarding specific, and in Delegate's case, we use LMDB to maintain cluster land.

Optimisation in Favour of Speed, Fast Response to minimise fault tolerance

Off-white leader ballot: Leader is being elected efficiently and fairly. The candidate's term is matched with the follower's term every time a vote request is made. Likewise the candidate with the most updated logs is favoured.
Faster Recovery: If the network is currently without a leader, the election is started every bit soon as first HeartBeat (after the leader went down) is transmitted in the network, this allows an efficient network recovery.
App transmission is separate from Management transmission: Management ports are used merely for leader election and Append RPCs and app ports are used only for serving customer requests.
Minimal Network Clogging during App network advice: Although we are using a mesh topology followers respond directly to Leaders and non through any other path (not via whatever node) so in effect its point to point communication for serving client requests.
Faster fault recovery: If a node is downwardly immediately the node is removed from the management connections hashMap. Once the failed node joins back the network, its logs are updated with the current leader'south log.
Minimal Payload: The accepted message format in pipeline is of type "poke.comm.App.Request" so as to reduce payload weight over the network.

FTP STORAGE

The FTP server is used as a storage for storing all the image files that the clients ship to other clients. It acts as a cardinal repository for the nodes in the cluster to store files that need to be transferred to the client. FTP server also provides for a secure storage medium.

IMPLEMENTATION

Leader election using Raft Algorithm

Each node joins in a FOLLOWER state. A timer starts for each node. The nodes wait for the Suspend RPC'due south until timeout, if it receives a Append RPC's before timeout, then Leader exists, else it does not exist.
If Leader exists, the node functions as a follower.
If Leader does not exist, the node changes its state to CANDIDATE, increments the term and sends a RequestVote message.
Once the RequestVote message is received by the other nodes in the network, it checks if the candidate's log is as upward-to-date as its log and if the Candidate's term is greater than or equal to its term.
If both the weather condition are satisfied, then it casts its vote to the candidate.
If the Candidate receives majority, it declares itself as Leader, changes the land to LEADER and sends the message to all the other nodes about its Leadership.
If the Candidate does not get the majority, then no Leader is elected for this term, the term ends and new term starts.
For the next term, new Election process starts, where all the nodes wait until the timeout and the ballot procedure is carried nigh equally stated to a higher place.

Log Replication

The Leader creates the log file whenever it receives the request to transfer the images.
The Log created by the Leader contains the log index and the imageName.
Before appending, the Leader increments the index and updates the prevIndex to current index.
Once the log file is created, it is sent to the other nodes in the cluster past appending the log to the Append RPC message.
Keeping a Boolean variable appendLogs to check if the log needs to exist added to the Append RPC or not. It is gear up to true when a server receives a request to transfer an image.
If appendLogs = true, then append the log file to the heartbeat and and so it set to false.
If appendLogs = false, and so at that place is no new log to append.
When a node receives the logs, it checks the prevIndexes until the log entry matches with the entry in the Leader's log file. It and then updates its log file equally per the Leader log file.

High level menstruum of the solution

Two channels are established when a node joins the network, a direction channel and an App channel.
An App channel is also established when the client registers with the node.
Client and then sends the request to the server node with Routing_Id= JOBS
The node checks if it is the Leader. If the node is not the leader and then the node forrad the request via APP channel to the leader using Routing_Id = FORWARD (every node saves the current term's LeaderId). Then it creates the log entries, appends the updated log to the heartbeat and sends the message through the management channel and broadcast the request to the other nodes via app channel with Routing_Id = REPORTS.
If the node is a Leader, then equally explained above Leader does the log creation and broadcasts the asking.
All the nodes receive the request from the Leader and check if the request has the receiverclientID, if information technology contains the receiverClientId, and then the nodes bank check if the receiverClientId is connected to itself. If information technology is connected, and so information technology sends the request to that particular client.
If the request does not comprise the receiverClientId, then all the nodes broadcast the request to all the clients via APP channel.

Cluster Communication

A cluster is a connection of nodes in the network. The cluster components are continued to each other through LAN. Performance and availability is increased past the apply of cluster.

Intra-Cluster Communication: A server is selected among the nodes connected which works as a cluster leader. Information technology is responsible to listen to the requests from the network. Chore bidding is done further inside the cluster
Inter-Cluster Communication: The clusters connect with each other only later the Leader is elected in all the clusters. Then the Nodes in the cluster sends a JOIN Asking to nodes of the other cluster, then it establishes a connection. The aqueduct data and nodeId are stored in a hashMap. This information is used to communicate in future. The client Request contains the information whether the message is for intra-cluster or inter-cluster. If the message is for intra-cluster, then information technology flows through the cluster as explained above. If the message is for inter-cluster, then the asking is sent to all nodes in the other clusters. When asking is received by the node in our cluster, it first uploads the paradigm to the FTP server, then the flow continues every bit explained higher up (as explained in the high level period of solution).

Compages

alt tag