Re: [PATCH 0/8] KVM updates for 2.6.20-rc2

Ingo Molnar wrote:
> btw., we've got a problem with KVM in -rt:
>
>  BUG: scheduling while atomic: qemu/0x00000001/11445, CPU#0
>   [<c0104fce>] dump_trace+0x63/0x1e8
>   [<c010516c>] show_trace_log_lvl+0x19/0x2e
>   [<c0105541>] show_trace+0x12/0x14
>   [<c0105557>] dump_stack+0x14/0x16
>   [<c0322662>] __schedule+0xae/0xd9f
>   [<c0323597>] schedule+0xec/0x10f
>   [<c0324017>] rt_spin_lock_slowlock+0xd2/0x15b
>   [<c03244a6>] rt_spin_lock+0x1e/0x20
>   [<c0169d94>] kunmap_high+0x11/0x99
>   [<c0121139>] kunmap+0x40/0x42
>   [<c0121171>] kunmap_virt+0x36/0x38
>   [<f8f017b7>] emulator_read_std+0x92/0xaf [kvm]
>   [<f8f05292>] x86_emulate_memop+0xc6/0x295c [kvm]
>   [<f8f018c3>] emulate_instruction+0xef/0x202 [kvm]
>   [<f8e0f2a7>] handle_exception+0x107/0x1c8 [kvm_intel]
>   [<f8e0ef0c>] kvm_vmx_return+0x145/0x18d [kvm_intel]
>   [<f8f0253c>] kvm_dev_ioctl+0x253/0xd76 [kvm]
>   [<c01874f1>] do_ioctl+0x21/0x66
>   [<c018778b>] vfs_ioctl+0x255/0x268
>   [<c01877e6>] sys_ioctl+0x48/0x62
>   [<c0103f8d>] syscall_call+0x7/0xb
>   [<4a4538b2>] 0x4a4538b2
>
> NOTE: this is not a worry for upstream kernel, it is caused by 
> PREEMPT_RT scheduling in previously atomic APIs like kunmap(). But KVM 
> used to work pretty nicely in -rt and this problem got introduced fairly 
> recently, related to some big-page changes IIRC.
>   

This is not a recent change.  The x86 emulator is fetching an 
instruction or a memory operand.

> Any suggestions of how to best fix this in -rt? Basically, it would be 
> nice to decouple KVM from get_cpu/preempt_disable type of interfaces as 
> much as possible, to make it fully preemptible. 

kvm on intel uses hidden cpu registers, so it can't be migrated as a 
normal process.  We'd need a hook on context switch to check if we 
migrated to another cpu, issue an ipi to unload the registers on the 
previous cpu, and reload the registers on the current cpu.  The hook 
would need to be protected from context switches.  See 
vmx.c:vmx_vcpu_load() for the gory details.  The actual launch would 
require a small non-preemptible section as well, on both intel and AMD.

Talking about the scheduler (and to the scheduler's author :), it would 
be nice to hook the migration weight algorithm too.  kvm guests are 
considerably more expensive to migrate, at least on intel.


> Below are my current 
> fixups for KVM, but the problem above looks harder to fix.
>   

These make sense for mainline (especially the first hunk), so I'll apply 
them.

>       Ingo
>
> ---
>  drivers/kvm/vmx.c |    4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> Index: linux/drivers/kvm/vmx.c
> ===================================================================
> --- linux.orig/drivers/kvm/vmx.c
> +++ linux/drivers/kvm/vmx.c
> @@ -117,7 +117,7 @@ static void vmcs_clear(struct vmcs *vmcs
>  static void __vcpu_clear(void *arg)
>  {
>       struct kvm_vcpu *vcpu = arg;
> -     int cpu = smp_processor_id();
> +     int cpu = raw_smp_processor_id();
>  
>       if (vcpu->cpu == cpu)
>               vmcs_clear(vcpu->vmcs);
> @@ -576,7 +576,7 @@ static struct vmcs *alloc_vmcs_cpu(int c
>  
>  static struct vmcs *alloc_vmcs(void)
>  {
> -     return alloc_vmcs_cpu(smp_processor_id());
> +     return alloc_vmcs_cpu(raw_smp_processor_id());
>  }
>  
>  static void free_vmcs(struct vmcs *vmcs)
>   


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV