Galera Cluster for MySQL

The World’s most advanced open source database cluster

Galera Cluster for MySQL is an easy-to-use high-availability solution with high system up-time, no data loss, and scalability for future growth

http://galeracluster.com/

Big Data – What is it???

Ref:-http://gurkulindia.com/main/2013/06/bigdata/

Most of the technology geeks may have heard the recent buzz about Big Data; in recent times many of my colleagues and friends were asking several questions. So I thought I should write a blog post to better answer their questions.

What is Big Data?

Big Data is defined ‘n’ number of ways in the industry, so instead of trying to find the actual definition lets try to understand the concepts and idea behind it.

As the name says “big data” you may think it is all about size – but that is not just that. There is a lot more to deal with and enormous number of use cases in multiple domains. One of the ways to explain BigData is “V3”

V – Volume V – Variety V – Velocity

Picture Source: Wired.com

Another approach of this is “ABC”

A – Analytics B – Bandwidth C – Capacity

One can say “V3” or “ABC” both as the characteristics of big data rather than as a definition.

Let’s get into more details of the V’s

‘V’ – Volume of Data:

The sheer “volume” of data being stored is exploding. 90% of the world’s data is generated from last 4 years. We expect this number to reach 35 Zettabytes (ZB) by 2020. Companies like Facebook, Twitter, CERN generates Terabytes of data every single day. Interestingly 80% of the world’s data is unstructured, businesses today do not have enough resources or technology to store this and turn it as “useful” data or in other words its is very hard to get information out of the available data.

One of the well-observed phenomena is the data available to an organization is “raising” where as the percent of data an organization can process is declining, this is kind of depressing as a technology lover. But don’t feel bad, we have Hadoop to take fix this ☺

‘V’ – Variety of Data:

With the growing amounts of data we now have a new challenge to deal with: its variety. With growing variety of sources we have seen “variety of data” to deal with; sensors, social networking, feeds, smart devices, location info, and many more. This has left us in a complex situation, because it not only has traditional relational data (very less percent) but majority of it is raw, semi structured & unstructured data from web logs, click-stream data, search indexes, email, photo videos and soon.

For us to handle this kind of data on a traditional system is impossible “0”. We need a fundamental shift in analysis requirement from traditional structured data to include “variety” of data. But as traditional analytic platforms can’t handle variety due to the nature of its built for supporting relational kind that’s neatly formatted and fits nicely into the strict schemas.

As we know the “truth” about 80% data is left unprocessed we now have a need to build a system to efficiently store and process non-relational data and here by perform required analytics and generate report via the Business Intelligence (BI) tools and make real value to the business and to its growth.

‘V’ – Velocity of Data:

Just as the sheer volume and variety of data we collect and store has changed, so, too, has the “velocity” at which it is generated and needs to be handled. As we know the growth rates associated with data repositories is very high with growth in the number of source.
Rather than confining the idea of velocity the above mentioned we could intercept it as “data in motion”: ‘The speed at which data is flowing’:

Two most important challenges to deal with are:

1. Data in motion
2. Data in rest

Dealing effectively with Big Data requires us to perform analytics against the volume and variety of data while it is “still in motion”, not just after it is at “rest”.

Consider a fraud prevention at real time use case: Lets say a credit card is cloned and used at two different locations at the same time, with our existing ‘traditional’ systems we have lag involved to detect this. But imagine if we have “real time” data processing and analyzing technology to prevent this. Its just wonderful as it sounds.

Why Big Data?

• To analyze not only raw structured data, but semi structured, unstructured data from a variety of sources.
• To effectively process and analyze larger set of data instead of analyzing sample of the data.
• To solve information challenges that don’t natively fit within a traditional relational data base approach for handling the v3.
• To improve “intelligence in business” and to take quicker actions by developing “real” B.I tools and reaching the customer needs like never before.
• To develop business patterns and trends in “real time”
• To improve the quality of business in various sector like e-health, retail, IT , call centers, agriculture & so on.

“To handle, process the data and do magical things that were never imagined by anyone”

Working With Big data:

Google in its initial days was successfully able to download the Internet and index the available data when it was small. But when data started growing and new sources started increasing everyday things became complex to handle. So Google come with up solution internally to process this growing volume in a completely different way.

In that process they have started developing GFS – Google File System and also something called Map-Reduce (M to efficiently manage this growing data. But Google has kept this for their internal use and has not open sourced it. They have published a paper in 2004 called “Map-Reduce” to explain what and how this data is processed to make the internet searches possible.

Using that paper people in the industry started thinking in a different way. A guy named “Doug” has started developing a repository to handle the growing and unstructured data which is named as “Hadoop”, this is a open source project and is been actively developed and highly contributed by “Yahoo”.

Introduction to Hadoop

Ref:-  http://gurkulindia.com/main/2013/06/hadoop-intro/

Hadoop is a platform that is well suited to deal with semi-structured & unstructured data, as well as when a data discovery process is needed. That isn’t to say that Hadoop can’t be used for structured dara that is readily available in a raw format; because it can.

Traditionally, data goes through a lot of rigor to make it into the warehouse. This data is cleaned up via various cleansing, enrichment, modeling, master data management and other services before it is ready for analysis; which is expensive process. Because of that expense, its clear that data that lands in warehouse is not just high value, but has a broad purpose; it is used to generate reports & dash-board where the accuracy is the key.

In contrast, Big Data repositories very rarely undergo the full quality control versions of data injected into a warehouse, Hadoop is built for the purpose of handling larger volumes of data, so prepping data and processing it should be cost prohibitive.

I say Hadoop as a system designed for processing mind-boggling amounts of data

Two main components of Hadoop:

1. Map – Reduce = Computation
2. HDFS = Storage

Hadoop Distributed file system (HDFS):

Let’s discuss about the Hadoop cluster components before getting into details of HDFS.

A typical Hadoop environment consists of a master node, worker nodes with specialized software components.

Master node: There will be multiple master nodes to avoid single point of failure in any environment. The elements of master node are

1. Job Tracker
2. Task tracker
3. Name tracker

Job Tracker: Job tracker interacts with client applications. It is mainly responsible for distributing Map. Reduce tasks to particular nodes with in a cluster.

Task tracker: This process receives the tasks from a job tracker like Map, Reduce and shuffle.

Name node: All these processes are charged with storing a directory free of all files in the HDFS. They also keep track of where the file data is kept within the cluster. Client applications contact name nodes when they need to locate a file, or add, copy as delete a file.

Data Node: Data nodes stores data in the HDFS, it is responsible for replicating data across clusters. These interact with client apps and Name node supplied the data node’s address.

Worker Nodes: These are the commodity servers for processing the data that is coming through. Each worker node includes a data node and a task tracker

Scenario to better understand how “stuff” works:

1. Let’s say we have a 300mb file
2. By default we make it as 128mb blocks 
300mb= 128mb + 128mb + 44mb 
3. So HDFS splits 300mb into blocks as above
4. HDFS will keep 3 copies of each block
5. All these blocks are stored on data nodes

Bottom line is, Name node tracks blocks & data nodes and pays attention to all nodes in cluster. It do not save any data and no data goes through it.

• When a Data node (DN) fails it makes sure the copies are copied to another node and can handle upto 2 DN’s failure.
• Name node (NN) is a single point of failure.
• DN’s continuously runs check sums, if any block is corrupted then it will be process from other DN’s replicas.

There is lot more to discuss but let’s move on to M-R for now.

Map Reduce (M-R)

Google invented this. The main characteristics of M-R are:

1. Sort/merge is the primate
2. Batch oriented
3. Ad hoc queries (no schema)
4. Distribution handled by frame work

Let’s make it simple to understand, we get TB’s & PB’s of data to get processed & analyzed. So to handle it we use MR which basically has two major phases map & reduce.

Map: MR uses key/value pairs. Any data that comes in will be Splitted by HDFS into blocks and then we process it through M-R where we assign a value to every key.

Example: “Gurukulindia is the best site to learn big data”

Just to list network view & logical view and to make good view:

1. Input step: Load data into HDFS by splitting & load to DN’S. The blocks are replicated to overcome failures. The NN keeps track of blocks & DN’s.
2. Job step: Submit the MR Job & its details to the Job tracker.
3. Job init step: The Job tracker interacts with task tracker on each DN to schedule MR tasks.
4. Map step: Mapper process data blocks and generates a list of key value pairs
5. Sort step: Mapper sorts list of key value pair
6. Shuffle: Transfers mapped output to the reducers in sorted fashion.
7. Reduce: Reduces merge list of key value pairs to generate final result.

The results of Reduces are finally stored in HDFS replicated as per the configuration and then clients will be able to read from HDFS.

Latest Technologi​es To Learn

• Expertise in Big Data technologies like Apache Hadoop, HDFS, Map Reduce, Pig, Hive, SQOOP
• Installing and configuring the HADOOP in Pseudo Mode on Ubuntu
• Administration, installing, upgrading and managing distributions of Hadoop (CDH3, CDH4)
• knowledge in performance troubleshooting and tuning Hadoop clusters
• Knowledge in Map Reduce Hive, PIG, Sqoop.

OCFS2 Cluster File System Setup Guide in Linux

What is OCFS2?

• OCFS2 is a shared-disk cluster file system for Linux 
• Capable of providing both high performance and high availability.  
• Cluster-aware applications can make use of parallel I/O for higher performance with this FS 
• OCFS2 is mostly used to host Oracle Real application clusters (RAC) database on Linux clusters. 

The below are the high level steps for creating ocfs2 filesystem on top of a multipath'd SAN LUN:

1. Verify the nodes that will be part of your cluster.
2. Make sure your LUNs on the SAN end are accessible on all the nodes of the cluster.
3. If you need multipathing, configure multipath and the multipathing policy based on solution whatever you have.
4. The following example configuration ( /etc/ocfs2/cluster.conf ) shows you a sample configuration of a 2 node cluster pool. 
5. If you have heartbeat IP configured on these cluster nodes, use the heartbeat IP for ocfs2 cluster communication and specify the hostname without FQDN. 
6. Copy the same file to all the hosts in the cluster. 

Sample Config file:

[root@rac-cluster ~]# cat /etc/ocfs2/cluster.conf
node:
        ip_port = 7777
        ip_address = 192.168.1.2
        number = 0
        name = rac-cluster
        cluster = ocfs2
node:
        ip_port = 7777
        ip_address = 192.168.1.3
        number = 1
        name = rac-cluster
        cluster = ocfs2
 

On each node check the status of OCFS2 cluster service and stop "o2cb" if the service is already running. 

# service o2cb status # service o2cb stop

On each node, load the OCFS2 module. 

# service o2cb load

Make the OCFS2 service online on all the nodes. 

# service o2cb online 

Now your OCFS2 cluster is ready.

Format the SAN lun device from any one of the cluster node. 

# mkfs.ocfs2 -b 4k -C 32k -L oraclerac /dev/emcpowera

Where,

-b : Block size
-C : Cluster size
-L : Label

Update /etc/fstab on all the nodes in the cluster with the mount point. 

/dev/emcpowera /u01 ocfs2 _netdev 0 0 

Mount the /u01 

Enable ocfs and o2b service on boot

# chkconfig --level 345 o2cb on# chkconfig --level 345 ocfs2 on 

The /u01 repository setup on a SAN Lun is done.

You can now configure Oracle RAC on this file System.

Apache Hadoop Fundamentals – HDFS and MapReduce Explained with a Diagram

Hadoop is an open source software used for distributed computing that can be used to query a large set of data and get the results faster using reliable and scalable architecture.

This is the first article in our new ongoing Hadoop series.

In a traditional non distributed architecture, you’ll have data stored in one server and any client program will access this central data server to retrieve the data. The non distributed model has few fundamental issues. In this model, you’ll mostly scale vertically by adding more CPU, adding more storage, etc. This architecture is also not reliable, as if the main server fails, you have to go back to the backup to restore the data. From performance point of view, this architecture will not provide the results faster when you are running a query against a huge data set.

In a hadoop distributed architecture, both data and processing are distributed across multiple servers. The following are some of the key points to remember about the hadoop:

• Each and every server offers local computation and storage. i.e When you run a query against a large data set, every server in this distributed architecture will be executing the query on its local machine against the local data set. Finally, the resultset from all this local servers are consolidated.
• In simple terms, instead of running a query on a single server, the query is split across multiple servers, and the results are consolidated. This means that the results of a query on a larger dataset are returned faster.
• You don’t need a powerful server. Just use several less expensive commodity servers as hadoop individual nodes.
• High fault-tolerance. If any of the nodes fails in the hadoop environment, it will still return the dataset properly, as hadoop takes care of replicating and distributing the data efficiently across the multiple nodes.
• A simple hadoop implementation can use just two servers. But you can scale up to several thousands of servers without any additional effort.
• Hadoop is written in Java. So, it can run on any platform.
• Please keep in mind that hadoop is not a replacement for your RDBMS. You’ll typically use hadoop for unstructured data
• Originally Google started using the distributed computing model based on GFS (Google Filesystem) and MapReduce. Later Nutch (open source web search software) was rewritten using MapReduce. Hadoop was branced out of Nutch as a separate project. NowHadoop is a top-level Apache project that has gained tremendous momentum and popularity in recent years.

HDFS

HDFS stands for Hadoop Distributed File System, which is the storage system used by Hadoop. The following is a high-level architecture that explains how HDFS works.

The following are some of the key points to remember about the HDFS:

• In the above diagram, there is one NameNode, and multiple DataNodes (servers). b1, b2, indicates data blocks.
• When you dump a file (or data) into the HDFS, it stores them in blocks on the various nodes in the hadoop cluster. HDFS creates several replication of the data blocks and distributes them accordingly in the cluster in way that will be reliable and can be retrieved faster. A typical HDFS block size is 128MB. Each and every data block is replicated to multiple nodes across the cluster.
• Hadoop will internally make sure that any node failure will never results in a data loss.
• There will be one NameNode that manages the file system metadata
• There will be multiple DataNodes (These are the real cheap commodity servers) that will store the data blocks
• When you execute a query from a client, it will reach out to the NameNode to get the file metadata information, and then it will reach out to the DataNodes to get the real data blocks
• Hadoop provides a command line interface for administrators to work on HDFS
• The NameNode comes with an in-built web server from where you can browse the HDFS filesystem and view some basic cluster statistics

MapReduce

The following are some of the key points to remember about the HDFS:

• 
  MapReduce is a parallel programming model that is used to retrieve the data from the Hadoop cluster
• In this model, the library handles lot of messy details that programmers doesn’t need to worry about. For example, the library takes care of parallelization, fault tolerance, data distribution, load balancing, etc.
• This splits the tasks and executes on the various nodes parallely, thus speeding up the computation and retriving required data from a huge dataset in a fast manner.
• This provides a clear abstraction for programmers. They have to just implement (or use) two functions: map and reduce
• The data are fed into the map function as key value pairs to produce intermediate key/value pairs
• Once the mapping is done, all the intermediate results from various nodes are reduced to create the final output
• JobTracker keeps track of all the MapReduces jobs that are running on various nodes. This schedules the jobs, keeps track of all the map and reduce jobs running across the nodes. If any one of those jobs fails, it reallocates the job to another node, etc. In simple terms, JobTracker is responsible for making sure that the query on a huge dataset runs successfully and the data is returned to the client in a reliable manner.
• TaskTracker performs the map and reduce tasks that are assigned by the JobTracker. TaskTracker also constantly sends a hearbeat message to JobTracker, which helps JobTracker to decide whether to delegate a new task to this particular node or not.

We’ve only scratched the surface of the Hadoop. This is just the first article in our ongoing series on Hadoop. In the future articles of this series, we’ll explain how to install and configure Hadoop environment, and how to write MapReduce programs to retrieve the data from the cluster, and how to effectively maintain a Hadoop infrastructure.

Howto install GlusterFS on Centos/RHEL

1. Introduction
GlusterFS is a clustered file­system capable of scaling to several peta­bytes. It aggregates various storage
bricks over Infiniband RDMA or TCP/IP interconnect into one large parallel network file system. Storage
bricks can be made of any commodity hardware such as x86­64 server with SATA­II RAID and
Infiniband HBA.
2. Installation
First you need to install some software :
yum install make gcc gcc-c++
yum install flex bison byacc
We need to do a dirty trick to get the package fuse­sshfs :
vi /etc/yum.repos.d/CentOS-Base.repo
and add this at the end of the file :
[extras_fedora]
name=Fedora Core 6 Extras
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=extras-6&arch=$basea
rch
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-extras
gpgcheck=0
Next do :
yum install fuse-sshfs --enablerepo=extras_fedora
Now we will install GlusterFS :
Get the same exact version, otherwise there is good chances it wont work. I tried with 2.0.0rc1 and 1.3.12
and there was some issues.
cd /root/
wget http://ftp.gluster.com/pub/gluster/glusterfs/2.0/LATEST/glusterfs-
2.0.0rc2.tar.gz
tar -zxvf glusterfs-2.0.0rc2.tar.gz
cd /root/glusterfs-2.0.0rc2/
Take a minute break and compile :
./configure
make && make install
For some reasons, libraries are going in the wrong directory so we need to (if someone has a clean fix to
this please post it!) :
cp /usr/local/lib/* -R /usr/lib/
Now we create some folders that will be used later on :
mkdir    /mnt/glusterfs
mkdir    /data/
mkdir    /data/export
mkdir    /data/export-ns
mkdir    /etc/glusterfs/
The fun begins on the next page :)

3. Servers configuration
Before you go further, you need to know that GlusterFS works in a client/server way. What we will do is
to make our servers both client and server for GlusterFS.
Lets start with the server configuration file ON ALL SERVERS:
vi /etc/glusterfs/glusterfs-server.vol
and make it look like this :
# file: /etc/glusterfs/glusterfs-server.vol

volume posix
  type storage/posix
  option directory /data/export
end-volume

volume locks
  type features/locks
  subvolumes posix
end-volume

volume brick
  type performance/io-threads
  option thread-count 8
  subvolumes locks
end-volume

volume posix-ns
  type storage/posix
  option directory /data/export-ns
end-volume

volume locks-ns
  type features/locks
  subvolumes posix-ns
end-volume

volume brick-ns
  type performance/io-threads
  option thread-count 8
  subvolumes locks-ns
end-volume

volume server
  type protocol/server
  option transport-type tcp
  option auth.addr.brick.allow *
  option auth.addr.brick-ns.allow *
  subvolumes brick brick-ns
end-volume
Now do :
glusterfsd -f /etc/glusterfs/glusterfs-server.vol
to start the server daemon.

4. Clients configuration
In these example files, I will use the following hosts :
server1 : 192.168.0.1
server2 : 192.168.0.2
server3 : 192.168.0.3
server4 : 192.168.0.4
[...]
Now we edit the client configuration file ON ALL SERVERS (because servers are client as well in this
howto):
vi /etc/glusterfs/glusterfs-client.vol
2 servers configuration
### Add client feature and attach to remote subvolume of server1
volume brick1
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.0.1                      # IP address of the remote brick
 option remote-subvolume brick                           # name of the remote volume
end-volume

### Add client feature and attach to remote subvolume of server2
volume brick2
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.0.2                         # IP address of the remote brick
 option remote-subvolume brick                           # name of the remote volume
end-volume

### The file index on server1
volume brick1-ns
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.0.1    # IP address of the remote brick
 option remote-subvolume brick-ns        # name of the remote volume
end-volume

### The file index on server2
volume brick2-ns
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.0.2      # IP address of the remote brick
 option remote-subvolume brick-ns        # name of the remote volume
end-volume

#The replicated volume with data
volume afr1
 type cluster/afr
 subvolumes brick1 brick2
end-volume

#The replicated volume with indexes
volume afr-ns
 type cluster/afr
 subvolumes brick1-ns brick2-ns
end-volume

#The unification of all afr volumes (used for > 2 servers)
volume unify
  type cluster/unify
  option scheduler rr # round robin
  option namespace afr-ns
  subvolumes afr1
end-volume
4 servers configuration
### Add client feature and attach to remote subvolume of server1
volume brick1
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.0.1    # IP address of the remote brick
 option remote-subvolume brick        # name of the remote volume
end-volume

### Add client feature and attach to remote subvolume of server2
volume brick2
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.0.2      # IP address of the remote brick
 option remote-subvolume brick        # name of the remote volume
end-volume

### Add client feature and attach to remote subvolume of server3
volume brick3
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.0.3                 # IP address of the remote brick
 option remote-subvolume brick                   # name of the remote volume
end-volume

### Add client feature and attach to            remote subvolume of server4
volume brick4
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.0.4                 # IP address of the remote brick
 option remote-subvolume brick                   # name of the remote volume
end-volume

### Add client feature and attach to            remote subvolume of server1
volume brick1-ns
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.0.1              #  IP address of the remote brick
 option remote-subvolume brick-ns                   # name of the remote volume
end-volume

### Add client feature and attach to            remote subvolume of server2
volume brick2-ns
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.0.2                 # IP address of the remote brick
 option remote-subvolume brick-ns                   # name of the remote volume
end-volume

volume afr1
 type cluster/afr
 subvolumes brick1 brick4
end-volume

volume afr2
 type cluster/afr
 subvolumes brick2 brick3
end-volume

volume afr-ns
 type cluster/afr
 subvolumes brick1-ns brick2-ns
end-volume

volume unify
  type cluster/unify
  option scheduler rr # round robin
  option namespace afr-ns
  subvolumes afr1 afr2
end-volume
So on and so forth... For configuration over 4 servers, simply add brick volumes 2 by two, replicate them
and dont forget to put them in the "unify" volume.
Now mount the GlusterFS on all servers in the cluster :
glusterfs -f /etc/glusterfs/glusterfs-client.vol /mnt/glusterfs
------------------------------------------------------------------------------------
---
5. Testing
Once you mounted the GlusterFS to /mnt/glusterfs you can start copying files and see what is happening.
Below are my tests on 4 servers. Everything works as it should, files in /data/export only show in 2 out of
4 server and everything is there under /mnt/glusterfs and /data/export­ns :
server 1 (ls    -la /data/export)
-rwxrwxrwx 1    marc marc 215663 2007-09-14         14:14   6-instructions2.pdf
-rwxrwxrwx 1    marc marc       2256 2008-12-18     11:54   budget.ods
-rwxr--r-- 1    marc marc 21281 2009-02-18          16:45   cv_nouveau.docx
-rwxrwxrwx 1    marc marc 13308 2009-01-26          10:49   cv.pdf
-rwxrwxrwx 1    marc marc 196375 2008-04-02         18:48   odometre.pdf
-rwxrwxrwx 1    marc marc       5632 2008-05-23     19:42   Thumbs.db

server 4 (ls    -la /data/export)
-rwxrwxrwx 1    marc marc 215663 2007-09-14         14:14   6-instructions2.pdf
-rwxrwxrwx 1    marc marc       2256 2008-12-18     11:54   budget.ods
-rwxr--r-- 1    marc marc 21281 2009-02-18          16:45   cv_nouveau.docx
-rwxrwxrwx 1    marc marc 13308 2009-01-26          10:49   cv.pdf
-rwxrwxrwx 1    marc marc 196375 2008-04-02         18:48   odometre.pdf
-rwxrwxrwx 1    marc marc       5632 2008-05-23     19:42   Thumbs.db

server 2 (ls    -la /data/export)
-rwxr--r-- 1    marc marc 135793 2009-02-02         15:26   bookmarks.html
-rwxrwxrwx 1    marc marc 112640 2008-11-17         21:41   cv.doc
-rwxrwxrwx 1    marc marc 13546 2007-09-11          15:43   cv.odt
-rwxrwxrwx 1    marc marc 25088 2006-07-03          17:07   menulaurentien.doc
-rwxr--r-- 1    marc marc 33734 2009-02-06          12:58   opera6.htm

server 3 (ls    -la /data/export)
-rwxr--r-- 1    marc marc 135793 2009-02-02         15:26   bookmarks.html
-rwxrwxrwx 1    marc marc 112640 2008-11-17         21:41   cv.doc
-rwxrwxrwx 1    marc marc 13546 2007-09-11          15:43   cv.odt
-rwxrwxrwx 1    marc marc 25088 2006-07-03          17:07   menulaurentien.doc
-rwxr--r-- 1    marc marc 33734 2009-02-06          12:58   opera6.htm

server x (ls    -la /mnt/glusterfs)
-rwxrwxrwx 1    marc marc 215663 2007-09-14         14:14   6-instructions2.pdf
-rwxr--r-- 1    marc marc 135793 2009-02-02         15:26   bookmarks.html
-rwxrwxrwx 1    marc marc       2256 2008-12-18     11:54   budget.ods
-rwxrwxrwx 1    marc marc 112640 2008-11-17         21:41   cv.doc
-rwxr--r-- 1    marc marc 21281 2009-02-18          16:45   cv_nouveau.docx
-rwxrwxrwx 1    marc marc 13546 2007-09-11          15:43   cv.odt
-rwxrwxrwx 1    marc marc 13308 2009-01-26          10:49   cv.pdf
-rwxrwxrwx 1    marc marc 25088 2006-07-03          17:07   menulaurentien.doc
-rwxrwxrwx 1    marc marc 196375 2008-04-02         18:48   odometre.pdf
-rwxr--r-- 1    marc marc 33734 2009-02-06          12:58   opera6.htm
-rwxrwxrwx 1    marc marc       5632 2008-05-23     19:42   Thumbs.db

server 1 (ls    -la /data/export-ns)
-rwxrwxrwx 1    marc marc        0 2007-09-14 14:14 6-instructions2.pdf
-rwxr--r-- 1    marc   marc      0  2009-02-02    15:26  bookmarks.html
-rwxrwxrwx 1    marc   marc      0  2008-12-18    11:54  budget.ods
-rwxrwxrwx 1    marc   marc      0  2008-11-17    21:41  cv.doc
-rwxr--r-- 1    marc   marc      0  2009-02-18    16:45  cv_nouveau.docx
-rwxrwxrwx 1    marc   marc      0  2007-09-11    15:43  cv.odt
-rwxrwxrwx 1    marc   marc      0  2009-01-26    10:49  cv.pdf
-rwxrwxrwx 1    marc   marc      0  2006-07-03    17:07  menulaurentien.doc
-rwxrwxrwx 1    marc   marc      0  2008-04-02    18:48  odometre.pdf
-rwxr--r-- 1    marc   marc      0  2009-02-06    12:58  opera6.htm
-rwxrwxrwx 1    marc   marc      0  2008-05-23    19:42  Thumbs.db


server 2 (ls    -la /data/export-ns)
-rwxrwxrwx 1    marc marc        0 2007-09-14     14:14  6-instructions2.pdf
-rwxr--r-- 1    marc marc        0 2009-02-02     15:26  bookmarks.html
-rwxrwxrwx 1    marc marc        0 2008-12-18     11:54  budget.ods
-rwxrwxrwx 1    marc marc        0 2008-11-17     21:41  cv.doc
-rwxr--r-- 1    marc marc        0 2009-02-18     16:45  cv_nouveau.docx
-rwxrwxrwx 1    marc marc        0 2007-09-11     15:43  cv.odt
-rwxrwxrwx 1    marc marc        0 2009-01-26     10:49  cv.pdf
-rwxrwxrwx 1    marc marc        0 2006-07-03     17:07  menulaurentien.doc
-rwxrwxrwx 1    marc marc        0 2008-04-02     18:48  odometre.pdf
-rwxr--r-- 1    marc marc        0 2009-02-06     12:58  opera6.htm
-rwxrwxrwx 1    marc marc        0 2008-05-23     19:42  Thumbs.db
Now let say we want to test how redundant is the setup. Lets reboot server1 and create new files while its
down :
> /mnt/glusterfs/testfile
> /mnt/glusterfs/testfile2
> /mnt/glusterfs/testfile3
> /mnt/glusterfs/testfile4
Once server1 is back, lets check file consistency :
server 1 (ls    -la /data/export)
-rwxrwxrwx 1    marc marc 215663 2007-09-14          14:14 6-instructions2.pdf
-rwxrwxrwx 1    marc marc       2256 2008-12-18      11:54 b4udget.ods
-rwxr--r-- 1    marc marc 21281 2009-02-18           16:45 cv_nouveau.docx
-rwxrwxrwx 1    marc marc 13308 2009-01-26           10:49 cv.pdf
-rwxrwxrwx 1    marc marc 196375 2008-04-02          18:48 odometre.pdf
-rwxrwxrwx 1    marc marc       5632 2008-05-23      19:42 Thumbs.db

server 4 (ls    -la /data/export)
-rwxrwxrwx 1    marc marc 215663 2007-09-14          14:14 6-instructions2.pdf
-rwxrwxrwx 1    marc marc       2256 2008-12-18      11:54 budget.ods
-rwxr--r-- 1    marc marc 21281 2009-02-18           16:45 cv_nouveau.docx
-rwxrwxrwx 1    marc marc 13308 2009-01-26           10:49 cv.pdf
-rwxrwxrwx 1    marc marc 196375 2008-04-02          18:48 odometre.pdf
-rw-r--r-- 1    root root           0 2009-02-19     11:32 testfile
-rw-r--r-- 1    root root           0 2009-02-19     11:32 testfile3
-rwxrwxrwx 1    marc marc       5632 2008-05-23      19:42 Thumbs.db

server 1 (ls    -la /data/export-ns)
-rwxrwxrwx 1    marc marc        0 2007-09-14 14:14 6-instructions2.pdf
-rwxr--r-- 1    marc marc        0 2009-02-02 15:26 bookmarks.html
-rwxrwxrwx 1    marc marc        0 2008-12-18 11:54 budget.ods
-rwxrwxrwx     1  marc   marc      0  2008-11-17     21:41   cv.doc
-rwxr--r--     1  marc   marc      0  2009-02-18     16:45   cv_nouveau.docx
-rwxrwxrwx     1  marc   marc      0  2007-09-11     15:43   cv.odt
-rwxrwxrwx     1  marc   marc      0  2009-01-26     10:49   cv.pdf
-rwxrwxrwx     1  marc   marc      0  2006-07-03     17:07   menulaurentien.doc
-rwxrwxrwx     1  marc   marc      0  2008-04-02     18:48   odometre.pdf
-rwxr--r--     1  marc   marc      0  2009-02-06     12:58   opera6.htm
-rwxrwxrwx     1  marc   marc      0  2008-05-23     19:42   Thumbs.db
Oups, we have an inconstency here. To fix that, gluster documentation says missing files have to be read.
So lets do this simple command to read all files :
ls -lR /mnt/glusterfs/
Now, lets check what we have on server1 :
server1 (ls -la /data/export)
-rwxrwxrwx 1 marc marc 215663 2007-09-14 14:14 6-instructions2.pdf
-rwxrwxrwx 1 marc marc            2256 2008-12-18 11:54 budget.ods
-rwxr--r-- 1 marc marc 21281 2009-02-18 16:45 cv_nouveau.docx
-rwxrwxrwx 1 marc marc 13308 2009-01-26 10:49 cv.pdf
-rwxrwxrwx 1 marc marc 196375 2008-04-02 18:48 odometre.pdf
-rw-r--r-- 1 root root                0 2009-02-19 11:32 testfile
-rw-r--r-- 1 root root                0 2009-02-19 11:32 testfile3
-rwxrwxrwx 1 marc marc            5632 2008-05-23 19:42 Thumbs.db

server1 (ls -la /data/export-ns)
-rwxrwxrwx 1 marc marc             0 2007-09-14 14:14 6-instructions2.pdf
-rwxr--r-- 1 marc marc             0 2009-02-02 15:26 bookmarks.html
-rwxrwxrwx 1 marc marc             0 2008-12-18 11:54 budget.ods
-rwxrwxrwx 1 marc marc             0 2008-11-17 21:41 cv.doc
-rwxr--r-- 1 marc marc             0 2009-02-18 16:45 cv_nouveau.docx
-rwxrwxrwx 1 marc marc             0 2007-09-11 15:43 cv.odt
-rwxrwxrwx 1 marc marc             0 2009-01-26 10:49 cv.pdf
-rwxrwxrwx 1 marc marc             0 2006-07-03 17:07 menulaurentien.doc
-rwxrwxrwx 1 marc marc             0 2008-04-02 18:48 odometre.pdf
-rwxr--r-- 1 marc marc             0 2009-02-06 12:58 opera6.htm
-rw-r--r-- 1 root root             0 2009-02-19 11:29 testfile
-rw-r--r-- 1 root root             0 2009-02-19 11:29 testfile2
-rw-r--r-- 1 root root             0 2009-02-19 11:29 testfile3
-rw-r--r-- 1 root root             0 2009-02-19 11:29 testfile4
-rwxrwxrwx 1 marc marc             0 2008-05-23 19:42 Thumbs.db
Now everything is as it should be.
------------------------------------------------------------------------------------
6. Conclusion
GlusterFS has a lot of potential. What you saw here is a small portion of what GlusterFS can do. As I said
in the first page, this setup was not tested on a live webserver and very little testing was done. If you plan
to put this on a live server and test this setup in depth, please share your experience in the forums or
simply post a comment on this page. Also, it would be very interesting if someone can post benchmarks
to see how well it scale.
Further reading : http://www.gluster.org