Subject: Re: GRUB, zfs-root + Xen: Error 16: Inconsistent
filesystem structure - msg#00061

Previous Message by Date:

Re: Installing a Solaris domU under a Linux dom0

Well, with David's help (on IRC), I added to /etc/system: set xnf:xnf_cksum_offload = 0 then did bootadm update-archive, rebooted, and all seems to work. It turns out that the checksum offloading didn't work well on my setup, chance why... I suspect that the sis900 ethernet driver being used on the Linux dom0 side doesn't support checksum offloading all that well. The symptoms were: I could see traffic coming from the OpenSolaris domU into the bridge on the dom0 with no checksums, and then the traffic leaving the bridge on the dom0 with bad checksums. This, of course, should not happen. With the workaround above, the traffic from the domU has correct checksums in it (rather than expecting hardware elsewhere to insert it) and all is well... At the cost of CPU time, but as a workaround, that's acceptable to me. If I can be of any help to try to track down the actual problem, let me know. Gavin (Beirdo on IRC) On 7/23/07, David Edmondson <dme@xxxxxxx> wrote: > On Sun, 2007-07-22 at 15:39 -0400, Gavin Hurlbut wrote: > > vif = ['bridge=xenbr0'] > > ... > > Note: I'm not yet sure if my networking will work, or if I need to > > twiddle that. That vif setting works for all my Linux domUs so I used > > it as a starting place. I think it worked with the b44 drop too. > > It should work just fine. > > dme. > > >

Next Message by Date:

Re: GRUB, zfs-root + Xen: Error 16: Inconsistent filesystem structure

Chaning Xen's load address is something that we really don't have as much control over as one would want. We could customize the version of Xen that we ship with Solaris, but that makes our Dom0 incompatible with every other Xen being shipped in the open source world out there -- which would make Solaris look rather bad and rather defeats some of the purpose of open source. Even if the Xen people are willing to take back the change to the load address, it would thro a major monkey wrench into our current schedule. I'm wondering if it would be simpler to fix ZFS grub to not use hard coded physical addresses. That seems to be a rather poor design, especially since address space above 1Meg is what GRUB traditionally leaves for any OS it's booting. Did the ZFS module try to use the GRUB dynamic allocation mechanism and find problems or did you just not even try that direction? Joe Lin Ling wrote: > Hi Juergen, > > Thanks for the findings, see inline comments: > > Juergen Keil wrote: >> Hi Lin, >> >> In addition to bug 6541114... >> >> Bug ID 6541114 >> Synopsis GRUB/ZFS fails to load files from a default compressed (lzjb) root >> >> ... I found yet another way to get the "Error 16: Inconsistent filesystem >> structure" from GRUB. This time when trying to boot a Xen Dom0 from a >> zfs bootfs >> >> >> Synopsis: grub/zfs-root: cannot boot xen from a zfs root >> ======================================================================== >> >> I've tried to install snv66 + xen into an lzjb compressed zfs >> root filesystem. >> >> menu.lst entry for xen is: >> >> # ------------------------------------------------------------ >> title Solaris Nevada snv_66 X86 (xen dom0) >> root (,0,g) >> bootfs files/s11-root-xen >> kernel$ /boot/$ISADIR/xen.gz >> module$ /platform/i86xpv/kernel/$ISADIR/unix >> /platform/i86xpv/kernel/$ISADIR/uni >> x -B $ZFS-BOOTFS -vk >> module$ /platform/i86pc/$ISADIR/boot_archive >> # ------------------------------------------------------------ >> >> grub boot for xen crashes with the error message: >> >> Error 16: Inconsistent filesystem structure >> >> >> >> GRUB uses fixed memory locations for MOS, DNODE, ZFS_SCRATCH... >> >> MOS is at memory location 0x100000. >> DNODE is at memory location 0x140000. >> ZFS_SCRATCH is at memory location 0x180000. >> >> Standard Solaris kernel /platform/i86pc/kernel/amd64/unix loads at >> 0x400000, 0x800000 and 0xC00000, and /platform/i86pc/amd64/boot_archive >> is loaded at 0xd5d000 - all after grub's MOS / DNODE / ZFS_SCRATCH location. >> >> >> >> Xen hypervisor /boot/amd64/xen.gz is loaded at >> <0x100000:0x9c878:0x58788>. >> >> GRUB is able to read the first 128k of compressed data from the zfs >> root, decompresses the data to address 0x100000, and the attempt to >> read the next 128k block from xen.gz fails because the DNODE data is >> overwritten. Things start to fail when we find >> "DNODE->dn_datablkszsec == 35656" (should be 256) in zfs_read(), that is, >> a datablk size of ~18mbytes instead of the expected 128kbytes. >> >> >> Problem #1: >> =========== >> >> fsys_zfs.c is supposed to use the following memory map: >> >> * (memory addr) MOS DNODE ZFS_SCRATCH >> * | | | >> * +-------V---------V----------V---------------+ >> * memory | | dnode | dnode | scratch | >> * | | 512B | 512B | area | >> * +--------------------------------------------+ >> >> Using these defines... >> >> #define MOS ((dnode_phys_t *)(RAW_ADDR(0x100000))) >> #define DNODE ((dnode_phys_t *)(MOS + DNODE_SIZE)) >> #define ZFS_SCRATCH ((char *)(DNODE + DNODE_SIZE)) >> >> ... the DNODE area is located ``512*sizeof(dnode_phys_t)'' bytes after >> MOS, not 512 bytes! Instead of 512 bytes for MOS, fsys_zfs is using >> 256 kbytes. Same problem with the size for the DNODE area. >> >> Apparently we want: >> >> #define MOS ((dnode_phys_t *)(RAW_ADDR(0x100000))) >> #define DNODE ((dnode_phys_t *)((char*)MOS + DNODE_SIZE)) >> #define ZFS_SCRATCH ((char *)DNODE + DNODE_SIZE) >> >> >> > > I will putback this fix along with 6541114. > >> Problem #2: >> =========== >> >> We should find a better base address for MOS/DNODE/ZFS_SCRATCH >> >> This seems to be the memory in use by GRUB: >> >> 0x007be BOOT_PART_TABLE >> 0x01000-0x1fff STAGE1_STACK / real mode stage2 STACKOFF (< 0x2000) >> 0x02000 MB_CMDLINE_BUF >> 0x07C00 BOOTSEC_LOCATION / MBR >> 0x08000 stage1 / PBR (start.S) >> 0x08200 stage2 (asm.S) >> 0x10000 LINUX_ZIMAGE_ADDR >> 0x60000-0x67fff protected mode stack >> 0x68000 FSYS_BUF (filesystem (not raw device) buffer / 32k) >> 0x70000 BUFFERADDR (raw device buffer / 31.5K) >> 0x77e00 SCRATCHADDR (512-byte scratch area) >> 0x78000 PASSWORD_BUF ... MENU_BUF >> 0x80000 free? >> 0x90000 LINUX_OLD_REAL_MODE_ADDR >> 0xA0000 Video memory? >> 0xB0000 HERCULES_VIDEO_ADDR >> 0x100000 LINUX_BZIMAGE_ADDR / XEN >> >> Maybe reusing 0x90000 could work (because we don't want to boot old >> linux stuff)? >> >> Or the FSYS_BUF at 0x68000? Other fsys_xxx modules use the 32k at >> 0x68000 FSYS_BUF. >> >> >> Well, I experimented with these addresses, but the problem seems to be >> that ZFS_SCRATCH needs *lots* of free space. All the areas below 0x100000 >> appear to be too small for fsys_zfs.c >> >> >> I'm currently using 0x4000000 as MOS base address, as an ugly workaround, >> to boot both standard Solaris kernels and the xen hypervisor: >> >> >> #define MOS ((dnode_phys_t *)(RAW_ADDR(0x4000000))) >> #define DNODE ((dnode_phys_t *)((char *)MOS + DNODE_SIZE)) >> #define ZFS_SCRATCH ((char *)DNODE + DNODE_SIZE) >> >> >> >> I guess another option would be to change the load address in the >> xen hypervisor from 0x100000 to 0x400000 (just like >> /platform/i86pc/kernel/unix) ? That'll leave ~ 3MB of free space for >> ZFS_SCRATCH ... >> >> >> > > Changing the load address in the xen hypervisor from 0x100000 to 0x400000 > makes sense to me. > > Thanks, > Lin > _______________________________________________ > xen-discuss mailing list > xen-discuss@xxxxxxxxxxxxxxx

Previous Message by Thread:

GRUB, zfs-root + Xen: Error 16: Inconsistent filesystem structure

Hi Lin, In addition to bug 6541114... Bug ID 6541114 Synopsis GRUB/ZFS fails to load files from a default compressed (lzjb) root ... I found yet another way to get the "Error 16: Inconsistent filesystem structure" from GRUB. This time when trying to boot a Xen Dom0 from a zfs bootfs Synopsis: grub/zfs-root: cannot boot xen from a zfs root ======================================================================== I've tried to install snv66 + xen into an lzjb compressed zfs root filesystem. menu.lst entry for xen is: # ------------------------------------------------------------ title Solaris Nevada snv_66 X86 (xen dom0) root (,0,g) bootfs files/s11-root-xen kernel$ /boot/$ISADIR/xen.gz module$ /platform/i86xpv/kernel/$ISADIR/unix /platform/i86xpv/kernel/$ISADIR/uni x -B $ZFS-BOOTFS -vk module$ /platform/i86pc/$ISADIR/boot_archive # ------------------------------------------------------------ grub boot for xen crashes with the error message: Error 16: Inconsistent filesystem structure GRUB uses fixed memory locations for MOS, DNODE, ZFS_SCRATCH... MOS is at memory location 0x100000. DNODE is at memory location 0x140000. ZFS_SCRATCH is at memory location 0x180000. Standard Solaris kernel /platform/i86pc/kernel/amd64/unix loads at 0x400000, 0x800000 and 0xC00000, and /platform/i86pc/amd64/boot_archive is loaded at 0xd5d000 - all after grub's MOS / DNODE / ZFS_SCRATCH location. Xen hypervisor /boot/amd64/xen.gz is loaded at <0x100000:0x9c878:0x58788>. GRUB is able to read the first 128k of compressed data from the zfs root, decompresses the data to address 0x100000, and the attempt to read the next 128k block from xen.gz fails because the DNODE data is overwritten. Things start to fail when we find "DNODE->dn_datablkszsec == 35656" (should be 256) in zfs_read(), that is, a datablk size of ~18mbytes instead of the expected 128kbytes. Problem #1: =========== fsys_zfs.c is supposed to use the following memory map: * (memory addr) MOS DNODE ZFS_SCRATCH * | | | * +-------V---------V----------V---------------+ * memory | | dnode | dnode | scratch | * | | 512B | 512B | area | * +--------------------------------------------+ Using these defines... #define MOS ((dnode_phys_t *)(RAW_ADDR(0x100000))) #define DNODE ((dnode_phys_t *)(MOS + DNODE_SIZE)) #define ZFS_SCRATCH ((char *)(DNODE + DNODE_SIZE)) ... the DNODE area is located ``512*sizeof(dnode_phys_t)'' bytes after MOS, not 512 bytes! Instead of 512 bytes for MOS, fsys_zfs is using 256 kbytes. Same problem with the size for the DNODE area. Apparently we want: #define MOS ((dnode_phys_t *)(RAW_ADDR(0x100000))) #define DNODE ((dnode_phys_t *)((char*)MOS + DNODE_SIZE)) #define ZFS_SCRATCH ((char *)DNODE + DNODE_SIZE) Problem #2: =========== We should find a better base address for MOS/DNODE/ZFS_SCRATCH This seems to be the memory in use by GRUB: 0x007be BOOT_PART_TABLE 0x01000-0x1fff STAGE1_STACK / real mode stage2 STACKOFF (< 0x2000) 0x02000 MB_CMDLINE_BUF 0x07C00 BOOTSEC_LOCATION / MBR 0x08000 stage1 / PBR (start.S) 0x08200 stage2 (asm.S) 0x10000 LINUX_ZIMAGE_ADDR 0x60000-0x67fff protected mode stack 0x68000 FSYS_BUF (filesystem (not raw device) buffer / 32k) 0x70000 BUFFERADDR (raw device buffer / 31.5K) 0x77e00 SCRATCHADDR (512-byte scratch area) 0x78000 PASSWORD_BUF ... MENU_BUF 0x80000 free? 0x90000 LINUX_OLD_REAL_MODE_ADDR 0xA0000 Video memory? 0xB0000 HERCULES_VIDEO_ADDR 0x100000 LINUX_BZIMAGE_ADDR / XEN Maybe reusing 0x90000 could work (because we don't want to boot old linux stuff)? Or the FSYS_BUF at 0x68000? Other fsys_xxx modules use the 32k at 0x68000 FSYS_BUF. Well, I experimented with these addresses, but the problem seems to be that ZFS_SCRATCH needs *lots* of free space. All the areas below 0x100000 appear to be too small for fsys_zfs.c I'm currently using 0x4000000 as MOS base address, as an ugly workaround, to boot both standard Solaris kernels and the xen hypervisor: #define MOS ((dnode_phys_t *)(RAW_ADDR(0x4000000))) #define DNODE ((dnode_phys_t *)((char *)MOS + DNODE_SIZE)) #define ZFS_SCRATCH ((char *)DNODE + DNODE_SIZE) I guess another option would be to change the load address in the xen hypervisor from 0x100000 to 0x400000 (just like /platform/i86pc/kernel/unix) ? That'll leave ~ 3MB of free space for ZFS_SCRATCH ...

Next Message by Thread:

Re: GRUB, zfs-root + Xen: Error 16: Inconsistent filesystem structure

Chaning Xen's load address is something that we really don't have as much control over as one would want. We could customize the version of Xen that we ship with Solaris, but that makes our Dom0 incompatible with every other Xen being shipped in the open source world out there -- which would make Solaris look rather bad and rather defeats some of the purpose of open source. Even if the Xen people are willing to take back the change to the load address, it would thro a major monkey wrench into our current schedule. I'm wondering if it would be simpler to fix ZFS grub to not use hard coded physical addresses. That seems to be a rather poor design, especially since address space above 1Meg is what GRUB traditionally leaves for any OS it's booting. Did the ZFS module try to use the GRUB dynamic allocation mechanism and find problems or did you just not even try that direction? Joe Lin Ling wrote: > Hi Juergen, > > Thanks for the findings, see inline comments: > > Juergen Keil wrote: >> Hi Lin, >> >> In addition to bug 6541114... >> >> Bug ID 6541114 >> Synopsis GRUB/ZFS fails to load files from a default compressed (lzjb) root >> >> ... I found yet another way to get the "Error 16: Inconsistent filesystem >> structure" from GRUB. This time when trying to boot a Xen Dom0 from a >> zfs bootfs >> >> >> Synopsis: grub/zfs-root: cannot boot xen from a zfs root >> ======================================================================== >> >> I've tried to install snv66 + xen into an lzjb compressed zfs >> root filesystem. >> >> menu.lst entry for xen is: >> >> # ------------------------------------------------------------ >> title Solaris Nevada snv_66 X86 (xen dom0) >> root (,0,g) >> bootfs files/s11-root-xen >> kernel$ /boot/$ISADIR/xen.gz >> module$ /platform/i86xpv/kernel/$ISADIR/unix >> /platform/i86xpv/kernel/$ISADIR/uni >> x -B $ZFS-BOOTFS -vk >> module$ /platform/i86pc/$ISADIR/boot_archive >> # ------------------------------------------------------------ >> >> grub boot for xen crashes with the error message: >> >> Error 16: Inconsistent filesystem structure >> >> >> >> GRUB uses fixed memory locations for MOS, DNODE, ZFS_SCRATCH... >> >> MOS is at memory location 0x100000. >> DNODE is at memory location 0x140000. >> ZFS_SCRATCH is at memory location 0x180000. >> >> Standard Solaris kernel /platform/i86pc/kernel/amd64/unix loads at >> 0x400000, 0x800000 and 0xC00000, and /platform/i86pc/amd64/boot_archive >> is loaded at 0xd5d000 - all after grub's MOS / DNODE / ZFS_SCRATCH location. >> >> >> >> Xen hypervisor /boot/amd64/xen.gz is loaded at >> <0x100000:0x9c878:0x58788>. >> >> GRUB is able to read the first 128k of compressed data from the zfs >> root, decompresses the data to address 0x100000, and the attempt to >> read the next 128k block from xen.gz fails because the DNODE data is >> overwritten. Things start to fail when we find >> "DNODE->dn_datablkszsec == 35656" (should be 256) in zfs_read(), that is, >> a datablk size of ~18mbytes instead of the expected 128kbytes. >> >> >> Problem #1: >> =========== >> >> fsys_zfs.c is supposed to use the following memory map: >> >> * (memory addr) MOS DNODE ZFS_SCRATCH >> * | | | >> * +-------V---------V----------V---------------+ >> * memory | | dnode | dnode | scratch | >> * | | 512B | 512B | area | >> * +--------------------------------------------+ >> >> Using these defines... >> >> #define MOS ((dnode_phys_t *)(RAW_ADDR(0x100000))) >> #define DNODE ((dnode_phys_t *)(MOS + DNODE_SIZE)) >> #define ZFS_SCRATCH ((char *)(DNODE + DNODE_SIZE)) >> >> ... the DNODE area is located ``512*sizeof(dnode_phys_t)'' bytes after >> MOS, not 512 bytes! Instead of 512 bytes for MOS, fsys_zfs is using >> 256 kbytes. Same problem with the size for the DNODE area. >> >> Apparently we want: >> >> #define MOS ((dnode_phys_t *)(RAW_ADDR(0x100000))) >> #define DNODE ((dnode_phys_t *)((char*)MOS + DNODE_SIZE)) >> #define ZFS_SCRATCH ((char *)DNODE + DNODE_SIZE) >> >> >> > > I will putback this fix along with 6541114. > >> Problem #2: >> =========== >> >> We should find a better base address for MOS/DNODE/ZFS_SCRATCH >> >> This seems to be the memory in use by GRUB: >> >> 0x007be BOOT_PART_TABLE >> 0x01000-0x1fff STAGE1_STACK / real mode stage2 STACKOFF (< 0x2000) >> 0x02000 MB_CMDLINE_BUF >> 0x07C00 BOOTSEC_LOCATION / MBR >> 0x08000 stage1 / PBR (start.S) >> 0x08200 stage2 (asm.S) >> 0x10000 LINUX_ZIMAGE_ADDR >> 0x60000-0x67fff protected mode stack >> 0x68000 FSYS_BUF (filesystem (not raw device) buffer / 32k) >> 0x70000 BUFFERADDR (raw device buffer / 31.5K) >> 0x77e00 SCRATCHADDR (512-byte scratch area) >> 0x78000 PASSWORD_BUF ... MENU_BUF >> 0x80000 free? >> 0x90000 LINUX_OLD_REAL_MODE_ADDR >> 0xA0000 Video memory? >> 0xB0000 HERCULES_VIDEO_ADDR >> 0x100000 LINUX_BZIMAGE_ADDR / XEN >> >> Maybe reusing 0x90000 could work (because we don't want to boot old >> linux stuff)? >> >> Or the FSYS_BUF at 0x68000? Other fsys_xxx modules use the 32k at >> 0x68000 FSYS_BUF. >> >> >> Well, I experimented with these addresses, but the problem seems to be >> that ZFS_SCRATCH needs *lots* of free space. All the areas below 0x100000 >> appear to be too small for fsys_zfs.c >> >> >> I'm currently using 0x4000000 as MOS base address, as an ugly workaround, >> to boot both standard Solaris kernels and the xen hypervisor: >> >> >> #define MOS ((dnode_phys_t *)(RAW_ADDR(0x4000000))) >> #define DNODE ((dnode_phys_t *)((char *)MOS + DNODE_SIZE)) >> #define ZFS_SCRATCH ((char *)DNODE + DNODE_SIZE) >> >> >> >> I guess another option would be to change the load address in the >> xen hypervisor from 0x100000 to 0x400000 (just like >> /platform/i86pc/kernel/unix) ? That'll leave ~ 3MB of free space for >> ZFS_SCRATCH ... >> >> >> > > Changing the load address in the xen hypervisor from 0x100000 to 0x400000 > makes sense to me. > > Thanks, > Lin > _______________________________________________ > xen-discuss mailing list > xen-discuss@xxxxxxxxxxxxxxx