66099bb0ee
There is problem with previous communication mechanism, and we got below
deadlock scenario with cluster which has 3 nodes.
Sender Receiver Receiver
token(EX)
message(EX)
writes message
downconverts message(CR)
requests ack(EX)
get message(CR) gets message(CR)
reads message reads message
requests EX on message requests EX on message
To fix this problem, we do the following changes:
1. the sender downconverts MESSAGE to CW rather than CR.
2. and the receiver request PR lock not EX lock on message.
And in case we failed to down-convert EX to CW on message, it is better to
unlock message otherthan still hold the lock.
Reviewed-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Lidong Zhong <ldzhong@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
Signed-off-by: NeilBrown <neilb@suse.com>
177 lines
7.0 KiB
Plaintext
177 lines
7.0 KiB
Plaintext
The cluster MD is a shared-device RAID for a cluster.
|
|
|
|
|
|
1. On-disk format
|
|
|
|
Separate write-intent-bitmap are used for each cluster node.
|
|
The bitmaps record all writes that may have been started on that node,
|
|
and may not yet have finished. The on-disk layout is:
|
|
|
|
0 4k 8k 12k
|
|
-------------------------------------------------------------------
|
|
| idle | md super | bm super [0] + bits |
|
|
| bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] |
|
|
| bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits |
|
|
| bm bits [3, contd] | | |
|
|
|
|
During "normal" functioning we assume the filesystem ensures that only one
|
|
node writes to any given block at a time, so a write
|
|
request will
|
|
- set the appropriate bit (if not already set)
|
|
- commit the write to all mirrors
|
|
- schedule the bit to be cleared after a timeout.
|
|
|
|
Reads are just handled normally. It is up to the filesystem to
|
|
ensure one node doesn't read from a location where another node (or the same
|
|
node) is writing.
|
|
|
|
|
|
2. DLM Locks for management
|
|
|
|
There are two locks for managing the device:
|
|
|
|
2.1 Bitmap lock resource (bm_lockres)
|
|
|
|
The bm_lockres protects individual node bitmaps. They are named in the
|
|
form bitmap001 for node 1, bitmap002 for node and so on. When a node
|
|
joins the cluster, it acquires the lock in PW mode and it stays so
|
|
during the lifetime the node is part of the cluster. The lock resource
|
|
number is based on the slot number returned by the DLM subsystem. Since
|
|
DLM starts node count from one and bitmap slots start from zero, one is
|
|
subtracted from the DLM slot number to arrive at the bitmap slot number.
|
|
|
|
3. Communication
|
|
|
|
Each node has to communicate with other nodes when starting or ending
|
|
resync, and metadata superblock updates.
|
|
|
|
3.1 Message Types
|
|
|
|
There are 3 types, of messages which are passed
|
|
|
|
3.1.1 METADATA_UPDATED: informs other nodes that the metadata has been
|
|
updated, and the node must re-read the md superblock. This is performed
|
|
synchronously.
|
|
|
|
3.1.2 RESYNC: informs other nodes that a resync is initiated or ended
|
|
so that each node may suspend or resume the region.
|
|
|
|
3.2 Communication mechanism
|
|
|
|
The DLM LVB is used to communicate within nodes of the cluster. There
|
|
are three resources used for the purpose:
|
|
|
|
3.2.1 Token: The resource which protects the entire communication
|
|
system. The node having the token resource is allowed to
|
|
communicate.
|
|
|
|
3.2.2 Message: The lock resource which carries the data to
|
|
communicate.
|
|
|
|
3.2.3 Ack: The resource, acquiring which means the message has been
|
|
acknowledged by all nodes in the cluster. The BAST of the resource
|
|
is used to inform the receive node that a node wants to communicate.
|
|
|
|
The algorithm is:
|
|
|
|
1. receive status
|
|
|
|
sender receiver receiver
|
|
ACK:CR ACK:CR ACK:CR
|
|
|
|
2. sender get EX of TOKEN
|
|
sender get EX of MESSAGE
|
|
sender receiver receiver
|
|
TOKEN:EX ACK:CR ACK:CR
|
|
MESSAGE:EX
|
|
ACK:CR
|
|
|
|
Sender checks that it still needs to send a message. Messages received
|
|
or other events that happened while waiting for the TOKEN may have made
|
|
this message inappropriate or redundant.
|
|
|
|
3. sender write LVB.
|
|
sender down-convert MESSAGE from EX to CW
|
|
sender try to get EX of ACK
|
|
[ wait until all receiver has *processed* the MESSAGE ]
|
|
|
|
[ triggered by bast of ACK ]
|
|
receiver get CR of MESSAGE
|
|
receiver read LVB
|
|
receiver processes the message
|
|
[ wait finish ]
|
|
receiver release ACK
|
|
|
|
sender receiver receiver
|
|
TOKEN:EX MESSAGE:CR MESSAGE:CR
|
|
MESSAGE:CR
|
|
ACK:EX
|
|
|
|
4. triggered by grant of EX on ACK (indicating all receivers have processed
|
|
message)
|
|
sender down-convert ACK from EX to CR
|
|
sender release MESSAGE
|
|
sender release TOKEN
|
|
receiver upconvert to PR of MESSAGE
|
|
receiver get CR of ACK
|
|
receiver release MESSAGE
|
|
|
|
sender receiver receiver
|
|
ACK:CR ACK:CR ACK:CR
|
|
|
|
|
|
4. Handling Failures
|
|
|
|
4.1 Node Failure
|
|
When a node fails, the DLM informs the cluster with the slot. The node
|
|
starts a cluster recovery thread. The cluster recovery thread:
|
|
- acquires the bitmap<number> lock of the failed node
|
|
- opens the bitmap
|
|
- reads the bitmap of the failed node
|
|
- copies the set bitmap to local node
|
|
- cleans the bitmap of the failed node
|
|
- releases bitmap<number> lock of the failed node
|
|
- initiates resync of the bitmap on the current node
|
|
|
|
The resync process, is the regular md resync. However, in a clustered
|
|
environment when a resync is performed, it needs to tell other nodes
|
|
of the areas which are suspended. Before a resync starts, the node
|
|
send out RESYNC_START with the (lo,hi) range of the area which needs
|
|
to be suspended. Each node maintains a suspend_list, which contains
|
|
the list of ranges which are currently suspended. On receiving
|
|
RESYNC_START, the node adds the range to the suspend_list. Similarly,
|
|
when the node performing resync finishes, it send RESYNC_FINISHED
|
|
to other nodes and other nodes remove the corresponding entry from
|
|
the suspend_list.
|
|
|
|
A helper function, should_suspend() can be used to check if a particular
|
|
I/O range should be suspended or not.
|
|
|
|
4.2 Device Failure
|
|
Device failures are handled and communicated with the metadata update
|
|
routine.
|
|
|
|
5. Adding a new Device
|
|
For adding a new device, it is necessary that all nodes "see" the new device
|
|
to be added. For this, the following algorithm is used:
|
|
|
|
1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues
|
|
ioctl(ADD_NEW_DISC with disc.state set to MD_DISK_CLUSTER_ADD)
|
|
2. Node 1 sends NEWDISK with uuid and slot number
|
|
3. Other nodes issue kobject_uevent_env with uuid and slot number
|
|
(Steps 4,5 could be a udev rule)
|
|
4. In userspace, the node searches for the disk, perhaps
|
|
using blkid -t SUB_UUID=""
|
|
5. Other nodes issue either of the following depending on whether the disk
|
|
was found:
|
|
ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CANDIDATE and
|
|
disc.number set to slot number)
|
|
ioctl(CLUSTERED_DISK_NACK)
|
|
6. Other nodes drop lock on no-new-devs (CR) if device is found
|
|
7. Node 1 attempts EX lock on no-new-devs
|
|
8. If node 1 gets the lock, it sends METADATA_UPDATED after unmarking the disk
|
|
as SpareLocal
|
|
9. If not (get no-new-dev lock), it fails the operation and sends METADATA_UPDATED
|
|
10. Other nodes get the information whether a disk is added or not
|
|
by the following METADATA_UPDATED.
|