Subject: EMC Cx300 multipath seems to work - msg#00018

List: linux.kernel.device-mapper.devel

Mail Archive Navigation:
by Date: Prev Next Date Index by Thread: Prev Next Thread Index

This is a multi-part message in MIME format. Hi,

multipath seems to work on gentoo, I have to load the modules in proper order, this is my configuration:

cat /etc/conf.d/local.start
# /etc/conf.d/local.start

# This is a good place to load any misc programs
# on startup ( use 1>&2 to hide output)
/sbin/modprobe dm-emc
/sbin/modprobe qla2300
/bin/sleep 5
/sbin/udevstart
/etc/init.d/multipathd start

is critical the command udevstart without only one path is seen,

I experienced a kernel oops with kernel-2.6.12 (both vanilla-sources and gentoo-sources) when I load qla2300 module, here are the logs:

Unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:
f8f262c8
*pde = 00000000
Oops: 0000 [#1]
SMP
Modules linked in: dm_round_robin qla2300 qla2xxx dm_emc dm_multipath dm_mod
CPU:    1
EIP:    0060:[<f8f262c8>]    Not tainted VLI
EFLAGS: 00010086   (2.6.12-gentoo-r10)
EIP is at rr_select_path+0x8/0x50 [dm_round_robin]
eax: f30311cc   ebx: 00000000   ecx: f7f4d08c   edx: f30311cc
esi: f46f1580   edi: 00000001   ebp: f46f1580   esp: f75f1ee0
ds: 007b   es: 007b   ss: 0068
Process kmpathd/1 (pid: 5836, threadinfo=f75f0000 task=c2171530)
Stack: f30311c0 f8f12bf6 f30311cc f46f15b8 f46f1580 f7f4d080 f8f12cd9 f46f1580
       f30311c0 f46f1580 f7f4d080 00000000 f46f158c f8f12fd9 f46f1580 00000001
       c2171530 00000246 00000000 f46f1590 f46f15c8 f7f4d080 f46f15cc f46f1580
Call Trace:
 [<f8f12bf6>] __choose_path_in_pg+0x26/0x60 [dm_multipath]
 [<f8f12cd9>] __choose_pgpath+0xa9/0xc0 [dm_multipath]
 [<f8f12fd9>] process_queued_ios+0xd9/0x100 [dm_multipath]
 [<c012b229>] worker_thread+0x1b9/0x260
 [<f8f12f00>] process_queued_ios+0x0/0x100 [dm_multipath]
 [<c01163c0>] default_wake_function+0x0/0x20
 [<c01163c0>] default_wake_function+0x0/0x20
 [<c012b070>] worker_thread+0x0/0x260
 [<c012f85a>] kthread+0xba/0xc0
 [<c012f7a0>] kthread+0x0/0xc0
 [<c0101015>] kernel_thread_helper+0x5/0x10
Code: 40 08 8b 10 8b 48 04 89 11 89 4a 04 8b 13 89 42 04 89 10 89 58 04 89 03 31 c0 5b c3 8d b4 26 00 00 00 00 53 8b 44 24 08 8b 58 04 <8b> 03 39 d8 74 32 89 c1 8b 50 04 8b 00 85 c9 89 50 04 89 02 8b
 <4>device-mapper: dm-emc: long trespass command will be send
device-mapper: dm-emc: honor reservation bit will not be set (default)
Unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:
f8f262c8
*pde = 00000000
Oops: 0000 [#2]
SMP
Modules linked in: dm_round_robin qla2300 qla2xxx dm_emc dm_multipath dm_mod
CPU:    0
EIP:    0060:[<f8f262c8>]    Not tainted VLI
EFLAGS: 00010086   (2.6.12-gentoo-r10)
EIP is at rr_select_path+0x8/0x50 [dm_round_robin]
eax: f748c8cc   ebx: 00000000   ecx: f7f4d00c   edx: f748c8cc
esi: f46f1680   edi: 00000001   ebp: f46f1680   esp: f7517ee0
ds: 007b   es: 007b   ss: 0068
Process kmpathd/0 (pid: 5835, threadinfo=f7516000 task=f7f0e530)
Stack: f748c8c0 f8f12bf6 f748c8cc f46f16b8 f46f1680 f7f4d000 f8f12cd9 f46f1680
       f748c8c0 f46f1680 f7f4d000 00000000 f46f168c f8f12fd9 f46f1680 c200c558
       f7f0e530 00000246 00000000 f46f1690 f46f16c8 f7f4d000 f46f16cc f46f1680
Call Trace:
 [<f8f12bf6>] __choose_path_in_pg+0x26/0x60 [dm_multipath]
 [<f8f12cd9>] __choose_pgpath+0xa9/0xc0 [dm_multipath]
 [<f8f12fd9>] process_queued_ios+0xd9/0x100 [dm_multipath]
 [<c012b229>] worker_thread+0x1b9/0x260
 [<f8f12f00>] process_queued_ios+0x0/0x100 [dm_multipath]
 [<c01163c0>] default_wake_function+0x0/0x20
 [<c01163c0>] default_wake_function+0x0/0x20
 [<c012b070>] worker_thread+0x0/0x260
 [<c012f85a>] kthread+0xba/0xc0
 [<c012f7a0>] kthread+0x0/0xc0
 [<c0101015>] kernel_thread_helper+0x5/0x10
Code: 40 08 8b 10 8b 48 04 89 11 89 4a 04 8b 13 89 42 04 89 10 89 58 04 89 03 31 c0 5b c3 8d b4 26 00 00 00 00 53 8b 44 24 08 8b 58 04 <8b> 03 39 d8 74 32 89 c1 8b 50 04 8b 00 85 c9 89 50 04 89 02 8b
 <4>device-mapper: dm-emc: long trespass command will be send
device-mapper: dm-emc: honor reservation bit will not be set (default)
Unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:
f8f262c8
*pde = 00000000
Oops: 0000 [#3]
SMP
Modules linked in: dm_round_robin qla2300 qla2xxx dm_emc dm_multipath dm_mod
CPU:    2
EIP:    0060:[<f8f262c8>]    Not tainted VLI
EFLAGS: 00010086   (2.6.12-gentoo-r10)
EIP is at rr_select_path+0x8/0x50 [dm_round_robin]
eax: f7e4850c   ebx: 00000000   ecx: f7f4d10c   edx: f7e4850c
esi: f46e4880   edi: 00000001   ebp: f46e4880   esp: f777dee0
ds: 007b   es: 007b   ss: 0068
Process kmpathd/2 (pid: 5837, threadinfo=f777c000 task=f7d11a40)
Stack: f7e48500 f8f12bf6 f7e4850c f46e48b8 f46e4880 f7f4d100 f8f12cd9 f46e4880
       f7e48500 f46e4880 f7f4d100 00000000 f46e488c f8f12fd9 f46e4880 c201c9d0
       f7d11a40 00000246 00000000 f46e4890 f46e48c8 f7f4d100 f46e48cc f46e4880
Call Trace:
 [<f8f12bf6>] __choose_path_in_pg+0x26/0x60 [dm_multipath]
 [<f8f12cd9>] __choose_pgpath+0xa9/0xc0 [dm_multipath]
 [<f8f12fd9>] process_queued_ios+0xd9/0x100 [dm_multipath]
 [<c012b229>] worker_thread+0x1b9/0x260
 [<f8f12f00>] process_queued_ios+0x0/0x100 [dm_multipath]
 [<c01163c0>] default_wake_function+0x0/0x20
 [<c01163c0>] default_wake_function+0x0/0x20
 [<c012b070>] worker_thread+0x0/0x260
 [<c012f85a>] kthread+0xba/0xc0
 [<c012f7a0>] kthread+0x0/0xc0
 [<c0101015>] kernel_thread_helper+0x5/0x10
Code: 40 08 8b 10 8b 48 04 89 11 89 4a 04 8b 13 89 42 04 89 10 89 58 04 89 03 31 c0 5b c3 8d b4 26 00 00 00 00 53 8b 44 24 08 8b 58 04 <8b> 03 39 d8 74 32 89 c1 8b 50 04 8b 00 85 c9 89 50 04 89 02 8b
 <4>device-mapper: dm-emc: long trespass command will be send

however kernel-2.6.13 works fine (both vanilla and gentoo-sources)

here are my multipath configuration

grep -v "#" /etc/multipath.conf
defaults {
        multipath_tool "/sbin/multipath -v0"
        udev_dir        /dev
        polling_interval 10
        default_selector        "round-robin 0"
        default_getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
        default_prio_callout    "/bin/true"
        failback        immediate
}
multipaths {

        multipath {
                wwid                    3600601608c901200ccfe543f4053d911
                alias                   200Gb
                path_grouping_policy    failover
                path_checker            readsector0
                path_selector           "round-robin 0"
                failback                immediate
        }
        multipath {
                wwid                    3600601608c9012006269b8f63b87d911
                alias                   5Gb
                path_grouping_policy    failover
                path_checker            readsector0
                path_selector           "round-robin 0"
                failback                immediate
        }
        multipath {
                wwid                    3600601608c9012008ae5de2cc985d911
                alias                   300Gb
                path_grouping_policy    failover
                path_checker            readsector0
                path_selector           "round-robin 0"
                failback                immediate
        }
}

The tests aren't very accurate because the machine is 100 Km away, tomorrow I'll do more accurate tests, I'll post results,

big thanks to Cristophe for multipath-tools,
big thanks to gistolero for his accurate testing on gentoo, your posts are very useful for me,

any suggestions to improve my setup are welcome,

I have to install naviagent? On redhat naviagent register powerpath name to cx 300, it works with multipath?

Nicola

P.S. for gistolero, I haven't your problem with multipathd (or so seems) however my hardware is different, I think also that the need to run udevstart is a gentoo bug, today gentoo has releases udev-0.68-r1 with start-stop script, if you read start script there is udevstart, however I haven't yet tested this script because they require unstable baselayout-1.12

Thread at a glance:

Previous Message by Date:

How about shortening SCSI timeout value dynamically?

This is a multi-part message in MIME format. Under the multipath environment, when the path is determined to fail, we have to wait 30 seconds because of the SCSI command default timeout(30 seconds). If the multiple paths are alive, I think it is more efficient to short the SCSI command timeout. Then we can shorten the delay of failover. Is there any critical reason to remain SCSI timeout as a 30 seconds? What do you think about my this?

Next Message by Date:

Re: Re: EMC Cx300 multipath seems to work

gistolero@xxxxxx ha scritto: > Hi Nicola, > > Nicola Murino wrote: > >> Hi, >> >> multipath seems to work on gentoo, I have to load the modules in proper >> order, this is my configuration: >> >> cat /etc/conf.d/local.start >> # /etc/conf.d/local.start >> >> # This is a good place to load any misc programs >> # on startup ( use 1>&2 to hide output) >> /sbin/modprobe dm-emc >> /sbin/modprobe qla2300 >> /bin/sleep 5 >> /sbin/udevstart >> /etc/init.d/multipathd start >> >> is critical the command udevstart without only one path is seen, > > > > After rebooting my system, I see only one path always: > > testhalde2 ~ # multipath -l > 150gb () > [size=150 GB][features="0"][hwhandler="0"] > \_ round-robin 0 [active] > \_ 0:0:0:1 sda 8:0 [active] > > > I have to run multipath again: > > testhalde2 ~ # multipath /dev/sda > reload: 150gb (3600508b40010079d0001900000460000) > [size=150 GB][features="0"][hwhandler="0"] > \_ round-robin 0 > \_ 0:0:0:1 sda 8:0 [active][ready] > \_ 1:0:0:1 sdb 8:16 [ready] > > Yes, after reboot I see only one path too, but when I run udevstart I see all paths, for this reason I put udevstart in /etc/conf.d/local.start >> I experienced a kernel oops with kernel-2.6.12 (both vanilla-sources and >> gentoo-sources) when I load qla2300 module, here are the logs: > > > I'm using a HP DL-380 G3 Proliant + Qlogic QLA 2340 (qla2300) with > 2.6.12.5 (vanilla). No kernel oops here. > I have qlogic 2322 and I must load dm-emc too, however kernel-2.6.13-gentoo-r1 works fine > >> P.S. for gistolero, I haven't your problem with multipathd (or so seems) >> however my hardware is different, I think also that the need to run >> udevstart is a gentoo bug, today gentoo has releases udev-0.68-r1 with >> start-stop script, if you read start script there is udevstart, however >> I haven't yet tested this script because they require unstable >> baselayout-1.12 > > > The same behavior with udev-068-r1. I don't think that this is a > gentoo bug, because distributions differ in udev _startup_ scripts > only. I know that I have to run udevstart after creating a new > udev.rules in /etc/udev/rules/. But after this, udev should parse > this rule after every change: > > udev 0.68-r1 needs unstable baselayout (1.12) to execute udev-start.sh, what baselayout are you using? > > After reboot... > > testhalde2 ~ # ls -lF /dev/mapper/ > brw------- 1 root root 254, 0 Sep 13 2005 150gb > brw------- 1 root root 254, 1 Sep 13 2005 150gb1 > brw------- 1 root root 254, 2 Sep 13 2005 150gb2 > crw-rw---- 1 root root 10, 63 Sep 13 2005 control > testhalde2 ~ # ls -lF /dev/1* > lrwxrwxrwx 1 root root 4 Sep 13 2005 /dev/150gb2 -> dm-2 > > testhalde2 ~ # udevstart > > testhalde2 ~ # ls -lF /dev/1* > lrwxrwxrwx 1 root root 4 Sep 13 13:00 /dev/150gb -> dm-0 > lrwxrwxrwx 1 root root 4 Sep 13 13:00 /dev/150gb1 -> dm-1 > lrwxrwxrwx 1 root root 4 Sep 13 13:00 /dev/150gb2 -> dm-2 > > Ok, udev knows the multipath udev-rule and creates this links. > > > > Deleting the multipath-table... > > testhalde2 ~ # dmsetup table > 150gb1: 0 64197 linear 254:0 63 > 150gb: 0 314572800 multipath 0 0 1 1 round-robin 0 2 1 8:0 1000 8:16 1000 > 150gb2: 0 314504505 linear 254:0 64260 > > testhalde2 ~ # dmsetup remove 150gb1 > testhalde2 ~ # dmsetup remove 150gb2 > testhalde2 ~ # dmsetup remove 150gb > testhalde2 ~ # dmsetup table > No devices found > > testhalde2 ~ # ls -lF /dev/mapper/ > total 0 > crw-rw---- 1 root root 10, 63 Sep 13 2005 control > > testhalde2 ~ # ls -lF /dev/1* > ls: /dev/1*: No such file or directory > > > > Creating the multipath-table... > > testhalde2 ~ # multipath /dev/sda > create: 150gb (3600508b40010079d0001900000460000) > [size=150 GB][features="0"][hwhandler="0"] > \_ round-robin 0 > \_ 0:0:0:1 sda 8:0 [ready] > \_ 1:0:0:1 sdb 8:16 [ready] > > testhalde2 ~ # ls -lF /dev/mapper/ > total 0 > brw------- 1 root root 254, 0 Sep 13 13:04 150gb > brw------- 1 root root 254, 1 Sep 13 13:04 150gb1 > brw------- 1 root root 254, 2 Sep 13 13:04 150gb2 > crw-rw---- 1 root root 10, 63 Sep 13 2005 control > > testhalde2 ~ # ls -lF /dev/1* > ls: /dev/1*: No such file or directory > > > No links in /dev/... I don't understand this, because > udev should read the multipath udev-rule again. > Perhaps I'm mistaken? > > Simon > > -- > dm-devel mailing list > dm-devel@xxxxxxxxxx > https://www.redhat.com/mailman/listinfo/dm-devel >

Previous Message by Thread:

Next Message by Thread:

Re: EMC Cx300 multipath seems to work

Hi Nicola, Nicola Murino wrote: Hi, multipath seems to work on gentoo, I have to load the modules in proper order, this is my configuration: cat /etc/conf.d/local.start # /etc/conf.d/local.start # This is a good place to load any misc programs # on startup ( use 1>&2 to hide output) /sbin/modprobe dm-emc /sbin/modprobe qla2300 /bin/sleep 5 /sbin/udevstart /etc/init.d/multipathd start is critical the command udevstart without only one path is seen, After rebooting my system, I see only one path always: testhalde2 ~ # multipath -l 150gb () [size=150 GB][features="0"][hwhandler="0"] \_ round-robin 0 [active] \_ 0:0:0:1 sda 8:0 [active] I have to run multipath again: testhalde2 ~ # multipath /dev/sda reload: 150gb (3600508b40010079d0001900000460000) [size=150 GB][features="0"][hwhandler="0"] \_ round-robin 0 \_ 0:0:0:1 sda 8:0 [active][ready] \_ 1:0:0:1 sdb 8:16 [ready] I experienced a kernel oops with kernel-2.6.12 (both vanilla-sources and gentoo-sources) when I load qla2300 module, here are the logs: I'm using a HP DL-380 G3 Proliant + Qlogic QLA 2340 (qla2300) with 2.6.12.5 (vanilla). No kernel oops here. P.S. for gistolero, I haven't your problem with multipathd (or so seems) however my hardware is different, I think also that the need to run udevstart is a gentoo bug, today gentoo has releases udev-0.68-r1 with start-stop script, if you read start script there is udevstart, however I haven't yet tested this script because they require unstable baselayout-1.12 The same behavior with udev-068-r1. I don't think that this is a gentoo bug, because distributions differ in udev _startup_ scripts only. I know that I have to run udevstart after creating a new udev.rules in /etc/udev/rules/. But after this, udev should parse this rule after every change: After reboot... testhalde2 ~ # ls -lF /dev/mapper/ brw------- 1 root root 254, 0 Sep 13 2005 150gb brw------- 1 root root 254, 1 Sep 13 2005 150gb1 brw------- 1 root root 254, 2 Sep 13 2005 150gb2 crw-rw---- 1 root root 10, 63 Sep 13 2005 control testhalde2 ~ # ls -lF /dev/1* lrwxrwxrwx 1 root root 4 Sep 13 2005 /dev/150gb2 -> dm-2 testhalde2 ~ # udevstart testhalde2 ~ # ls -lF /dev/1* lrwxrwxrwx 1 root root 4 Sep 13 13:00 /dev/150gb -> dm-0 lrwxrwxrwx 1 root root 4 Sep 13 13:00 /dev/150gb1 -> dm-1 lrwxrwxrwx 1 root root 4 Sep 13 13:00 /dev/150gb2 -> dm-2 Ok, udev knows the multipath udev-rule and creates this links. Deleting the multipath-table... testhalde2 ~ # dmsetup table 150gb1: 0 64197 linear 254:0 63 150gb: 0 314572800 multipath 0 0 1 1 round-robin 0 2 1 8:0 1000 8:16 1000 150gb2: 0 314504505 linear 254:0 64260 testhalde2 ~ # dmsetup remove 150gb1 testhalde2 ~ # dmsetup remove 150gb2 testhalde2 ~ # dmsetup remove 150gb testhalde2 ~ # dmsetup table No devices found testhalde2 ~ # ls -lF /dev/mapper/ total 0 crw-rw---- 1 root root 10, 63 Sep 13 2005 control testhalde2 ~ # ls -lF /dev/1* ls: /dev/1*: No such file or directory Creating the multipath-table... testhalde2 ~ # multipath /dev/sda create: 150gb (3600508b40010079d0001900000460000) [size=150 GB][features="0"][hwhandler="0"] \_ round-robin 0 \_ 0:0:0:1 sda 8:0 [ready] \_ 1:0:0:1 sdb 8:16 [ready] testhalde2 ~ # ls -lF /dev/mapper/ total 0 brw------- 1 root root 254, 0 Sep 13 13:04 150gb brw------- 1 root root 254, 1 Sep 13 13:04 150gb1 brw------- 1 root root 254, 2 Sep 13 13:04 150gb2 crw-rw---- 1 root root 10, 63 Sep 13 2005 control testhalde2 ~ # ls -lF /dev/1* ls: /dev/1*: No such file or directory No links in /dev/... I don't understand this, because udev should read the multipath udev-rule again. Perhaps I'm mistaken? Simon