 |
|
| iSCSI 기반 Device mapper Multipath 구축 실전 |
|
기가비트 스위치를 사이에 두고 2대의 노드(iSCSI initiator), 1대의 스토리지(iSCSI target) 준비
노드, 스토리지 각각 2개의 라인씩 뽑아내어 기가비트 스위치에 연결(기가비트 스위치 총 6개 포트 소모) 노드1 172.16.0.101 172.16.0.102 노드2 172.16.0.103 172.16.0.104 스토리지 172.16.0.105 172.16.0.106 iSCSI target 스토리지에 2개의 LUN 설정 [root@localhost ~]# /usr/sbin/tgtadm --lld iscsi --op show --mode target Target 1: iqn.storage:iscsi-storage System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk SCSI ID: IET 00010001 SCSI SN: beaf11 Size: 1045 MB, Block size: 512 Online: Yes Removable media: No Readonly: No Backing store type: rdwr Backing store path: /dev/vda1 Backing store flags: LUN: 2 Type: disk SCSI ID: IET 00010002 SCSI SN: beaf12 Size: 1045 MB, Block size: 512 Online: Yes Removable media: No Readonly: No Backing store type: rdwr Backing store path: /dev/vdb1 Backing store flags: Account information: ACL information: ALL [root@localhost ~]# 각 노드 device-mapper-multipath 패키지 설치 유무 확인 [root@node1 ~]# rpm -qa | grep device-mapper-multipath device-mapper-multipath-0.4.7-42.el5 [root@node1 ~]# 노드1, 노드2에서 아래 커맨드 실행(스토리지에 2개의 IP가 할당되어 있으므로) [root@node1 ~]# iscsiadm -m discovery -t st -p 172.16.0.105 -P 1 [root@node1 ~]# iscsiadm -m discovery -t st -p 172.16.0.105 -l [root@node1 ~]# iscsiadm -m discovery -t st -p 172.16.0.106 -P 1 [root@node1 ~]# iscsiadm -m discovery -t st -p 172.16.0.106 -l 스토리지에선 iSCSI target 정보가 아래와 같이 확인된다. [root@storage ~]# /usr/sbin/tgtadm --lld iscsi --op show --mode target Target 1: iqn.storage:iscsi-storage System information: Driver: iscsi State: ready I_T nexus information: I_T nexus: 5 Initiator: iqn.1994-05.com.redhat:3dd8dc7ed525 Connection: 0 IP Address: 172.16.0.103 I_T nexus: 6 Initiator: iqn.1994-05.com.redhat:3dd8dc7ed525 Connection: 0 IP Address: 172.16.0.101 I_T nexus: 7 Initiator: iqn.1994-05.com.redhat:3dd8dc7ed525 Connection: 0 IP Address: 172.16.0.103 I_T nexus: 8 Initiator: iqn.1994-05.com.redhat:3dd8dc7ed525 Connection: 0 IP Address: 172.16.0.101 LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk SCSI ID: IET 00010001 SCSI SN: beaf11 Size: 1045 MB, Block size: 512 Online: Yes Removable media: No Readonly: No Backing store type: rdwr Backing store path: /dev/vda1 Backing store flags: LUN: 2 Type: disk SCSI ID: IET 00010002 SCSI SN: beaf12 Size: 1045 MB, Block size: 512 Online: Yes Removable media: No Readonly: No Backing store type: rdwr Backing store path: /dev/vdb1 Backing store flags: Account information: ACL information: ALL [root@storage ~]# 이제 각 노드에서 iSCSI target의 공유 블록 디바이스를 확인하면 4개로 보인다(LUN 갯수 x 2). 아래 결과물을 보면 /dev/sda~/dev/sdd까지 확인되는 것이 스토리지의 공유 블록이다(1G 2ea). [root@node1 ~]# fdisk -l Disk /dev/vda: 21.4 GB, 21495808000 bytes 255 heads, 63 sectors/track, 2613 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/vda1 * 1 13 104391 83 Linux /dev/vda2 14 26 104422+ 82 Linux swap / Solaris /dev/vda3 27 2613 20780077+ 83 Linux Disk /dev/vdb: 5242 MB, 5242880000 bytes 16 heads, 63 sectors/track, 10158 cylinders Units = cylinders of 1008 * 512 = 516096 bytes Device Boot Start End Blocks Id System /dev/vdb1 1 195 98248+ 83 Linux /dev/vdb2 196 390 98280 83 Linux /dev/vdb3 391 779 196056 83 Linux /dev/vdb4 780 10158 4727016 5 Extended /dev/vdb5 780 1749 488848+ 83 Linux /dev/vdb6 1750 1944 98248+ 83 Linux /dev/vdb7 1945 3929 1000408+ 83 Linux /dev/vdb8 3930 4318 196024+ 83 Linux /dev/vdb9 5095 5676 293296+ 83 Linux /dev/vdb10 4319 5094 391072+ 83 Linux Partition table entries are not in disk order Disk /dev/sda: 1044 MB, 1044578304 bytes 33 heads, 61 sectors/track, 1013 cylinders Units = cylinders of 2013 * 512 = 1030656 bytes Disk /dev/sda doesn't contain a valid partition table Disk /dev/sdb: 1044 MB, 1044578304 bytes 33 heads, 61 sectors/track, 1013 cylinders Units = cylinders of 2013 * 512 = 1030656 bytes Disk /dev/sdb doesn't contain a valid partition table Disk /dev/sdc: 1044 MB, 1044578304 bytes 33 heads, 61 sectors/track, 1013 cylinders Units = cylinders of 2013 * 512 = 1030656 bytes Disk /dev/sdc doesn't contain a valid partition table Disk /dev/sdd: 1044 MB, 1044578304 bytes 33 heads, 61 sectors/track, 1013 cylinders Units = cylinders of 2013 * 512 = 1030656 bytes Disk /dev/sdd doesn't contain a valid partition table [root@node1 ~]# scsi_id 확인 [root@node1 ~]# scsi_id -gus /block/sda 1IET_00010001 [root@node1 ~]# scsi_id -gus /block/sdb 1IET_00010002 [root@node1 ~]# scsi_id -gus /block/sdc 1IET_00010001 [root@node1 ~]# scsi_id -gus /block/sdd 1IET_00010002 [root@node1 ~]# 위 출력값을 참고로 /etc/multipath.conf 작성 [root@node1 ~]# cat /etc/multipath.conf [root@node1 ~]# cat /etc/multipath.conf multipaths { multipath { wwid 1IET_00010001 alias red } multipath { wwid 1IET_00010002 alias blue } } [root@node1 ~]# multipathd 서비스 시작 [root@node1 ~]# service multipathd start fdisk로 다시 확인하면 /dev/dm-0, /dev/dm-1 장치가 확인됨 [root@node1 ~]# fdisk -l Disk /dev/vda: 21.4 GB, 21495808000 bytes 255 heads, 63 sectors/track, 2613 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/vda1 * 1 13 104391 83 Linux /dev/vda2 14 26 104422+ 82 Linux swap / Solaris /dev/vda3 27 2613 20780077+ 83 Linux Disk /dev/vdb: 5242 MB, 5242880000 bytes 16 heads, 63 sectors/track, 10158 cylinders Units = cylinders of 1008 * 512 = 516096 bytes Device Boot Start End Blocks Id System /dev/vdb1 1 195 98248+ 83 Linux /dev/vdb2 196 390 98280 83 Linux /dev/vdb3 391 779 196056 83 Linux /dev/vdb4 780 10158 4727016 5 Extended /dev/vdb5 780 1749 488848+ 83 Linux /dev/vdb6 1750 1944 98248+ 83 Linux /dev/vdb7 1945 3929 1000408+ 83 Linux /dev/vdb8 3930 4318 196024+ 83 Linux /dev/vdb9 5095 5676 293296+ 83 Linux /dev/vdb10 4319 5094 391072+ 83 Linux Partition table entries are not in disk order Disk /dev/sda: 1044 MB, 1044578304 bytes 33 heads, 61 sectors/track, 1013 cylinders Units = cylinders of 2013 * 512 = 1030656 bytes Disk /dev/sda doesn't contain a valid partition table Disk /dev/sdb: 1044 MB, 1044578304 bytes 33 heads, 61 sectors/track, 1013 cylinders Units = cylinders of 2013 * 512 = 1030656 bytes Disk /dev/sdb doesn't contain a valid partition table Disk /dev/sdc: 1044 MB, 1044578304 bytes 33 heads, 61 sectors/track, 1013 cylinders Units = cylinders of 2013 * 512 = 1030656 bytes Disk /dev/sdc doesn't contain a valid partition table Disk /dev/sdd: 1044 MB, 1044578304 bytes 33 heads, 61 sectors/track, 1013 cylinders Units = cylinders of 2013 * 512 = 1030656 bytes Disk /dev/sdd doesn't contain a valid partition table Disk /dev/dm-0: 1044 MB, 1044578304 bytes 255 heads, 63 sectors/track, 126 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/dm-0 doesn't contain a valid partition table Disk /dev/dm-1: 1044 MB, 1044578304 bytes 255 heads, 63 sectors/track, 126 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/dm-1 doesn't contain a valid partition table [root@node1 ~]# [root@node1 ~]# multipath -ll mulipath.conf line 1, invalid keyword: multipath mulipath.conf line 2, invalid keyword: wwid mulipath.conf line 3, invalid keyword: alias mulipath.conf line 4, invalid keyword: } mulipath.conf line 6, invalid keyword: multipath mulipath.conf line 7, invalid keyword: wwid mulipath.conf line 8, invalid keyword: alias mulipath.conf line 9, invalid keyword: } 1IET_00010001 dm-0 IET,VIRTUAL-DISK [size=996M][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 2:0:0:1 sde 8:64 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 3:0:0:1 sdf 8:80 [active][ready] 1IET_00010002 dm-1 IET,VIRTUAL-DISK [size=996M][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 2:0:0:2 sdg 8:96 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 3:0:0:2 sdh 8:112 [active][ready] [root@node1 ~]# mulipath.conf line 1, invalid keyword: multipath 위와 같은 에러가 나는 건 원본 multipath.conf 파일의 주석을 풀고 고쳐 쓰는 것으로 해결 봄 아마도 공백을 띄울 때 스페이스바로 띄우는 거랑 탭키로 띄우는 걸 가리는 모양.... [root@node1 ~]# cat /etc/multipath.conf multipaths { multipath { wwid 1IET_00010001 alias red } multipath { wwid 1IET_00010002 alias blue } } [root@node1 ~]# 이제 이상없이 잘 뜸 [root@node1 ~]# multipath -ll blue (1IET_00010002) dm-1 IET,VIRTUAL-DISK [size=996M][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 1:0:0:2 sdb 8:16 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:0:2 sdd 8:48 [active][ready] red (1IET_00010001) dm-0 IET,VIRTUAL-DISK [size=996M][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 1:0:0:1 sda 8:0 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:0:1 sdc 8:32 [active][ready] [root@node1 ~]# |
