HPC Application Performance on ESX 4.1: Memory Virtualization

 Allgemein, Knowledge Base, Updates, VMware, VMware Partner, VMware Virtual Infrastructure, vSphere  Kommentare deaktiviert für HPC Application Performance on ESX 4.1: Memory Virtualization
Nov 302010
 
HPC Application Performance on ESX 4.1: Memory Virtualization

This is the third part in an ongoing series on exploring performance issues of virtualizing HPC applications. In the first part, we described the setup and considered pure memory bandwidth using Stream. The second part considered the effect of network latency in a scientific application (NAMD) that ran across several virtual machines.  Here we look at two of the tests in the HPC Challenge (HPCC) suite:  StarRandomAccess and HPL. While certainly not spanning all possible memory access patterns found in HPC apps, these two tests are very different from each other and should help to give bounds on virtualization overhead related to these patterns.

Virtualization adds indirection to memory page table mappings: in addition to the logical-to-physical mappings maintained by the OS (either native or in a VM), the hypervisor must maintain guest physical-to-machine mappings. A straightforward implementation of both mappings in software would result in enormous overhead. Prior to the introduction of hardware MMU features in Intel (EPT) and AMD (RVI) processors, the performance problem was solved through the use of “shadow” page tables. These collapsed the two mappings to one so that the processor TLB cache could be used efficiently; however, updating shadow page tables is expensive. With EPT and RVI, both mappings are cached in the TLB, eliminating the need for shadow page tables. The trade-off is that a TLB miss can be expensive: the cost is not just double the cost of a miss in a conventional TLB; it is the square of the number of steps in the TLB page walk. This cost can be reduced by using large memory pages (2MB in x86_64) which typically need four steps in the TLB, rather than small pages (4KB) which need five. This overview is highly simplified; see the performance RVI and EPT whitepapers for much more detail about MMU virtualization, as well as results from several benchmarks representing enterprise applications. Here we extend the EPT paper to HPC apps running on a current version of vSphere.

Although there are certainly exceptions, two memory characteristics are common to HPC applications: a general lack of page table manipulation, and heavy use of memory itself. Memory is allocated once (along with the associated page tables) and used for a long time. This use can either be dominated by sequential accesses (running through an array), or by random accesses. The latter will put more stress on the TLB. Common enterprise apps are often the opposite: much heavier page table activity but lighter memory usage. Thus HPC apps do not benefit much from the elimination of shadow page tables (this alone made many enterprise apps run close to native performance as shown in the above papers), but may be sensitive to the costs of TLB misses.

These points are illustrated by two tests from the HPCC suite. StarRandomAccess is a relatively simple microbenchmark that continuously accesses random memory addresses. HPL is a standard floating-point linear algebra benchmark that accesses memory more sequentially. For these tests, version 1.4.1 of HPCC was used on RHEL 5.5 x86_64. Hyper-threading was disabled in the BIOS and all work was limited to a single socket (automatically in the virtual cases and forced with numactl for native). In this way, the effects of differences between native and virtual in how HT and NUMA are treated were eliminated. For virtual, a 4-vCPU VM with 22GB was used on a lab version (build 294208) of ESX 4.1.  The relevant HPCC parameters are N=40000, NB=100, P=Q=2, and np=4. These values ensure all the CPU resources and nearly all the available memory of one socket was consumed, thereby minimizing memory cache effects. The hardware is the same as in the first part of this series. In particular, Xeon X5570 processors with EPT are used.

Throughput results for StarRandomAccess are shown in Table 1. The metric GUP/s is billions of updates per second, a measure of memory bandwidth. Small/large pages refers to memory allocation in the OS and application. For virtual, ESX always backs guest memory with large pages, if possible (as it is here). The default case (EPT enabled, small pages in the guest) achieves only about 85% of native throughput.  For an application with essentially no I/O or privileged instructions that require special handling by the hypervisor, this is surprisingly poor at first glance. However, this is a direct result of the hardware architecture needed to avoid shadow page tables. Disabling EPT results in near-native performance because, now, the TLB costs are essentially the same as for native and the software MMU costs are minimal. TLB costs are still substantial as seen by the effect of using large pages in native and the guest OS: more than doubling of the performance. The virtualization overhead is reduced to manageable levels, although there is still a 2% benefit from disabling EPT.

Table 1.  StarRandomAccess throughput, GUP/s (ratio to native)

  Native Virtual
EPT on EPT off
Small pages 0.01842 0.01561 (0.848) 0.01811 (0.983)
Large pages 0.03956 0.03805 (0.962) 0.03900 (0.986)

Table 2 shows throughput results for HPL. The metric Gflops/s is billions of floating point operations per second. Memory is largely accessed sequentially, greatly reducing the stress on the TLB and the effect of large pages. Large pages improve virtual performance by 4%, but improve native performance by less than 2%. Disabling EPT improves virtual performance by only 0.5%. It is not clear why virtual is slightly faster than native in the large pages case; this will be investigated further.

Table 2.  HPL throughput, Gflop/s (ratio to native)

  Native Virtual
EPT on EPT off
Small pages 37.04 36.04 (0.973) 36.22 (0.978)
Large pages 37.74 38.24 (1.013) 38.42 (1.018)

While hardware MMU virtualization with Intel EPT and AMD RVI has been a huge benefit for many applications, these test results support the expectation that the benefit for HPC apps is smaller, and can even increase overhead in some cases. However, the example shown here where the latter is true is a microbenchmark that focuses on the worst case for this technology. Most HPC apps will not have so many random memory accesses, so the effect of EPT is likely to be small.

 

       Download VMware Products  | Privacy  | Update Feed Preferences 
        Copyright © 2010 VMware, Inc. All rights reserved.

When importing a machine using the vCenter Converter 4.2 plug-in, the Sysprep files are not found (1029009)

 Allgemein, Knowledge Base, Updates, VMware, VMware Partner, VMware Virtual Infrastructure, vSphere  Kommentare deaktiviert für When importing a machine using the vCenter Converter 4.2 plug-in, the Sysprep files are not found (1029009)
Nov 302010
 
When importing a machine using the vCenter Converter 4.2 plug-in, the Sysprep files are not found (1029009)

· Importing a machine into vCenter Server 4.1 using the vCenter Converter 4.2 plug-in fails · The Guest Customization step of the import

       Download VMware Products  | Privacy  | Update Feed Preferences 
        Copyright © 2010 VMware, Inc. All rights reserved.

Windows 32bit virtual machines with the ESX 4.0 Update 1 VMware Tools XPDM SVGA II graphics driver installed fail (1020158)

 Allgemein, Knowledge Base, Updates, VMware, VMware Partner, VMware Virtual Infrastructure, vSphere  Kommentare deaktiviert für Windows 32bit virtual machines with the ESX 4.0 Update 1 VMware Tools XPDM SVGA II graphics driver installed fail (1020158)
Nov 302010
 
Windows 32bit virtual machines with the ESX 4.0 Update 1 VMware Tools XPDM SVGA II graphics driver installed fail (1020158)

If you have Windows 32bit virtual machines with more than 3 GB memory and the ESX 4.0 Update 1 version of VMware Tools with

       Download VMware Products  | Privacy  | Update Feed Preferences 
        Copyright © 2010 VMware, Inc. All rights reserved.

When the SRM dns_update.cmd command is run with the skipmode parameter, the DNS records are added or removed in both testFailover and Failover operations (1031095)

 Allgemein, Knowledge Base, Updates, VMware, VMware Partner, VMware Virtual Infrastructure, vSphere  Kommentare deaktiviert für When the SRM dns_update.cmd command is run with the skipmode parameter, the DNS records are added or removed in both testFailover and Failover operations (1031095)
Nov 302010
 
When the SRM dns_update.cmd command is run with the skipmode parameter, the DNS records are added or removed in both testFailover and Failover operations (1031095)

When running the VMware vCenter Server Site Recovery Manager dns_update.cmd command

       Download VMware Products  | Privacy  | Update Feed Preferences 
        Copyright © 2010 VMware, Inc. All rights reserved.

Qlogic HBA logs dropped frames (1031078)

 Allgemein, Knowledge Base, Updates, VMware, VMware Partner, VMware Virtual Infrastructure, vSphere  Kommentare deaktiviert für Qlogic HBA logs dropped frames (1031078)
Nov 302010
 
Qlogic HBA logs dropped frames (1031078)

· Unable to connect to the ESX host using VMware Infrastructure client · HBA scans take a long time · Qlogic HBAs show large number of dropped frames · The latency is as high as 25000 msecs · Verbose logging

       Download VMware Products  | Privacy  | Update Feed Preferences 
        Copyright © 2010 VMware, Inc. All rights reserved.

Microsoft Cluster Service (MSCS) support on ESX (1004617)

 Allgemein, Knowledge Base, Updates, VMware, VMware Partner, VMware Virtual Infrastructure, vSphere  Kommentare deaktiviert für Microsoft Cluster Service (MSCS) support on ESX (1004617)
Nov 302010
 
Microsoft Cluster Service (MSCS) support on ESX (1004617)

This article provides supportability and setup information for Microsoft Cluster Service (MSCS) on ESX. To determine which releases of ESX support MSCS, consult the table below. The

       Download VMware Products  | Privacy  | Update Feed Preferences 
        Copyright © 2010 VMware, Inc. All rights reserved.

Enabling verbose logging in VMware Update Manager (1031115)

 Allgemein, Knowledge Base, Updates, VMware, VMware Partner, VMware Virtual Infrastructure, vSphere  Kommentare deaktiviert für Enabling verbose logging in VMware Update Manager (1031115)
Nov 302010
 
Enabling verbose logging in VMware Update Manager (1031115)

This article provides information on how to enable verbose (debug) logging for various components in VMware Update Manager. You may want to enable verbose logging when you want diagnose

       Download VMware Products  | Privacy  | Update Feed Preferences 
        Copyright © 2010 VMware, Inc. All rights reserved.

vMotion fails between hosts using Intel Xeon 56xx or 36xx series processors to hosts using specific Intel processors (1021023)

 Allgemein, Knowledge Base, Updates, VMware, VMware Partner, VMware Virtual Infrastructure, vSphere  Kommentare deaktiviert für vMotion fails between hosts using Intel Xeon 56xx or 36xx series processors to hosts using specific Intel processors (1021023)
Nov 302010
 
vMotion fails between hosts using Intel Xeon 56xx or 36xx series processors to hosts using specific Intel processors (1021023)

You receive a vMotion migration error when migrating between a host using Intel Xeon Processor 56xx series or 36xx series

       Download VMware Products  | Privacy  | Update Feed Preferences 
        Copyright © 2010 VMware, Inc. All rights reserved.

Failover fails when the LUN is configured on a SUN 6180 storage array (1024004)

 Allgemein, Knowledge Base, Updates, VMware, VMware Partner, VMware Virtual Infrastructure, vSphere  Kommentare deaktiviert für Failover fails when the LUN is configured on a SUN 6180 storage array (1024004)
Nov 302010
 
Failover fails when the LUN is configured on a SUN 6180 storage array (1024004)

When using the SUN 6180 storage array, you experience these symptoms: · LUN Trespass completes, but the LUN is not accessible · Cannot access the LUN · VMkernel or

       Download VMware Products  | Privacy  | Update Feed Preferences 
        Copyright © 2010 VMware, Inc. All rights reserved.

How to release stuck modifier keys (1480)

 Allgemein, Knowledge Base, Updates, VMware, VMware Partner, VMware Virtual Infrastructure, vSphere  Kommentare deaktiviert für How to release stuck modifier keys (1480)
Nov 302010
 
How to release stuck modifier keys (1480)

This article provides information to release the modifier keys, such as Ctrl and Alt, when they get stuck in the virtual machine console. The modifier keys can occasionally become stuck when you quit the

       Download VMware Products  | Privacy  | Update Feed Preferences 
        Copyright © 2010 VMware, Inc. All rights reserved.

vCenter operation times out with the error: Operation failed since another task is in progress (1004790)

 Allgemein, Knowledge Base, Updates, VMware, VMware Partner, VMware Virtual Infrastructure, vSphere  Kommentare deaktiviert für vCenter operation times out with the error: Operation failed since another task is in progress (1004790)
Nov 302010
 
vCenter operation times out with the error: Operation failed since another task is in progress (1004790)

· vCenter task stops responding · Tasks time out · If you are using VMware Consolidated Backup (VCB), backup fails but snapshots are left behind

       Download VMware Products  | Privacy  | Update Feed Preferences 
        Copyright © 2010 VMware, Inc. All rights reserved.

Memory usage spikes in a 64bit Windows machine with vSphere SDK 4.0 .NET samples (1030786)

 Allgemein, Knowledge Base, Updates, VMware, VMware Partner, VMware Virtual Infrastructure, vSphere  Kommentare deaktiviert für Memory usage spikes in a 64bit Windows machine with vSphere SDK 4.0 .NET samples (1030786)
Nov 302010
 
Memory usage spikes in a 64bit Windows machine with vSphere SDK 4.0 .NET samples (1030786)

In a 64bit Windows machine with vSphere SDK 4.0 .NET samples, you experience these symptoms:  · The RetrieveServiceContent() and RetrieveProperties()

       Download VMware Products  | Privacy  | Update Feed Preferences 
        Copyright © 2010 VMware, Inc. All rights reserved.

New articles published for week ending 11/27/2010

 Allgemein, Knowledge Base, Updates, VMware, VMware Partner, VMware Virtual Infrastructure, vSphere  Kommentare deaktiviert für New articles published for week ending 11/27/2010
Nov 292010
 
New articles published for week ending 11/27/2010

VMware ESX
When installing Windows 2008 R2 in a virtual machine, Windows setup fails to detect the hard disks (1030750)
Date Published: 11/22/2010
Best practices for collecting events related to the High Availability Agent for vSphere 4.1 (1030992)
Date Published: 11/23/2010
How to disable the CIM agent on the ESX/ESXi host (1025757)
Date Published: 11/24/2010
Adding an RDM to the virtual machine in an ESX/ESXi 4.1 host fails with the error: vmdk is larger than the maximum size supported by datastore (1031041)
Date Published: 11/26/2010

VMware ESXi
The ESX host with Adaptec controller fails with a purple diagnostic screen when restoring from a tape (1029721)
Date Published: 11/23/2010
Hot-add fails with the error: File is larger than the maximum size supported by datastore (1031038)
Date Published: 11/25/2010

VMware Fusion
Unable to create and start a Boot Camp partition in VMware Fusion due to the error: resource busy (1030663)
Date Published: 11/22/2010

VMware Service Manager
VMware Service Manager error: Difference of two datetime columns caused overflow at runtime (1031017)
Date Published: 11/24/2010
The Add New option in the CMDB Item Search screen is disabled in VMware Service Manager 9.0 when CMDB Item Security by Type and CMDB Item New are enabled (1030985)
Date Published: 11/23/2010
Config Port import log errors: is read only and cannot be created (1030990)
Date Published: 11/23/2010
The Save, Delete, and Add Schedule buttons are missing on the Scheduled Report screen (1030991)
Date Published: 11/23/2010
Scrolling through the Quick Solutions in Admin causes the Description and Solution text to disappear or move to a different solution (1031019)
Date Published: 11/24/2010
Discrepancies in report results in VMware Service Manager 9.0 (1031050)
Date Published: 11/25/2010
Call details are not copied to the request when creating a request from a call (1031054)
Date Published: 11/25/2010
Knowledge Content Access Security in the Knowledge Management role is not working correctly in VMware Service Manager 9.0 (1031084)
Date Published: 11/26/2010

VMware ThinApp
PermittedComputers and PermittedComputersAccessDeniedMsg parameters in ThinApp Package.ini for Active Directory Groups of computers (1030989)
Date Published: 11/23/2010

VMware vCenter Chargeback
Connecting to Chargeback web interface fails with the error: webpage cannot be displayed (1030760)
Date Published: 11/22/2010 
Creating a report in VMware vCenter Chargeback fails with the error: Error in generating report with name (1030701)
Date Published: 11/22/2010

VMware vCenter Lab Manager
Deploying virtual machines in resource pool fails with multiple errors (1030966)
Date Published: 11/23/2010

VMware vCenter Server
The vpxd logs report the error: [VpxdDrmInterface] Vm id <-VM_ID> not found (1025877)
Date Published: 11/23/2010
vMotion fails after upgrading to vSphere 4.1 with a license error (1030410)
Date Published: 11/24/2010
VmConnected Event is not generated when a virtual machine is created against vSphere 4.1 (1030993)
Date Published: 11/24/2010
vCenter Server intermittently reports the error: HA agent on <FQDN> in cluster <Cluster_Name> in <DC_Name>has an error (1025879)
Date Published: 11/24/2010
The vpxd log reports the warning: Lock datastore-nnn mode EXCLUSIVE held for yyyy ms (1029836)
Date Published: 11/23/2010
Hardware Status tab error: No new host data available. Data will be updated in 5 minutes. (1030536)
Date Published: 11/24/2010

VMware vCenter Site Recovery Manager
Unable to create more than one protection group (1010817)
Date Published: 11/26/2010
Protecting a virtual machine within SRM causes the error: Host, resource pool and datastore are not compatible (1030975)
Date Published: 11/23/2010

VMware vCenter Update Manager
Cannot import the ESX 3.5 patch or offline bundle using VMware Update Manager 4.1 (1026383)
Date Published: 11/23/2010

VMware vCloud Director
Diagram - VMware vCloud Director Cell Architecture (1030954)
Date Published: 11/23/2010

VMware View Manager
Configuring VMware View Event database fails with the error: An error occurred while attempting to configure the database (1029537)
Date Published: 11/22/2010

VMware vShield Endpoint
Trying to vMotion or power on a virtual machine being protected by vShield Endpoint fails (1030463)
Date Published: 11/24/2010

VMware vSphere PowerCLI
Managing vSphere Storage using the Command Line Interface (CLI) (1029301)
Date Published: 11/24/2010

VMware vSphere Web Services
Making ReadNextTasks Query more reliable when using queuedTime of a task in the collection filter (1030909)
Date Published: 11/22/2010

       Download VMware Products  | Privacy  | Update Feed Preferences 
        Copyright © 2010 VMware, Inc. All rights reserved.

SOAP API Behavior in Lab Manager has Changed (1031149)

 Allgemein, Knowledge Base, Updates, VMware, VMware Partner, VMware Virtual Infrastructure, vSphere  Kommentare deaktiviert für SOAP API Behavior in Lab Manager has Changed (1031149)
Nov 292010
 
SOAP API Behavior in Lab Manager has Changed (1031149)

The ListConfigurations API with the configuration type parameter set to 1 (workspace) no longer uses the organization name listed in authentication header. If you have access to multiple

       Download VMware Products  | Privacy  | Update Feed Preferences 
        Copyright © 2010 VMware, Inc. All rights reserved.

Lab Manager Uses Username Attribute When Searching for LDAP Users (1031146)

 Allgemein, Knowledge Base, Updates, VMware, VMware Partner, VMware Virtual Infrastructure, vSphere  Kommentare deaktiviert für Lab Manager Uses Username Attribute When Searching for LDAP Users (1031146)
Nov 292010
 
Lab Manager Uses Username Attribute When Searching for LDAP Users (1031146)

When searching for an LDAP user, Lab Manager is unable to find this user. This is a correction to the Lab Manager 4.0 User's Guide. In Lab Manager, when searching for an LDAP

       Download VMware Products  | Privacy  | Update Feed Preferences 
        Copyright © 2010 VMware, Inc. All rights reserved.