Virtual Ascetic

everything about virtualization, storage, and technology

Your likes and shares help increase my enthusiasm to share knowledge. Please leave a comment if you like any post

Best practices for performance tuning of latency sensitive workloads in vsphere VMs

By

Best practices for performance tuning of latency sensitive workloads in vsphere VMs

Visit original link for full details

http://www.vmware.com/resources/techresources/10220

 

To download whitepaper – click here

Abstract from VMware site –

This white paper summarizes findings and recommends best practices to tune the different layers of an application’s environment for latency-sensitive workloads—those which customers are looking at optimizing for a few microseconds to a few tens of microseconds end-to-end latencies (not workloads in the hundreds of microseconds to tens of milliseconds end-to-end-latencies).

We (a cross-functional team at VMware) investigated the performance of an in-memory, distributed data management platform, measured in the number of puts/second of 1KB data objects from one VM on one ESXi host to another VM on a different ESXi host across physical Ethernet networks. The paper provides recommendations that helped us go from 4400 puts/sec when we started, to 7200 puts/sec for a specific single-threaded, single vCPU application performance benchmark, which is at 78% of the score of 9200 puts/sec on bare metal on the same hardware.

We also investigated the performance of two latency micro-benchmarks, one for Infiniband devices and another for networking devices, in VM DirectPath I/O (pass-through) mode. Applying the recommendations reduced latency and therefore increased the score of these latency micro-benchmarks in a virtualized environment, bringing it closer to bare metal performance.

– See more at: http://www.vmware.com/resources/techresources/10220#sthash.ZNYRBzHV.dpuf

vSphere 6 Challenge – Hands on Lab

By

VMware has some interesting labs this year and one of the top labs is the vSphere 6 Challenge Lab. This was obviously one of the labs to participate in and it had some good stuff to know about. The vSphere 6 Challenge asks you to put on the thinking cap and go through fictional scenarios in order to fix real world issues. The issues were related to Operational and Performance areas. It started off with some basic stuff like how to fix copy/paste issues or fix CPU performance problems. A benchmark tool is part of the lab so it is good to see how the changes you are doing are improving/degrading VM performance. Things like – have a diconnected host or VM which will not power on and a mysteriously poorly performing VM are covered here.

So I will recommend that you go to the lab area today and do take this one – for those of you who are not at VMworld, hopefully you will have the chance to take it soon remotely after VMworld is over – if it is released publicly prior to VMworld Barcelona.

VMWorld 2015 – the journey begins

By

With each new year there is further growth in the field of virtualization and new products/features are released. When it comes to showcasing the announcements, there is no better platform than VMWorld.

This year however, the focus is slightly different from my perspective – for e.g. There will be no major product release announcements and no big bang approaches – vSphere 6 was launched last year so it makes sense that Vmware might actually improve feature set and add new functionality. I am specifically seeing the adoption of Cloud channel into a more sustainable approach of hybrid infrastructure deployments. The likes of which include working with NSX, storage volumes (vVOLs), automation enhancements, and possibly replication enhancements. The focus as usual will also be in the field of EUC – with Horizon going maintstream.

I am personally looking forward to attending a few special sessions – one of which is and will always be about Performance. The Performance series sessions are always popular because technology keeps changing and the performance measurement tools and feature set also evolves. A critical piece of the Performance measurement is going to be around scripting – and PowerCLI has plenty of potential to do a lot for many organizations. It would be nice to see if some of the functionality that PowerCLI or PowerShell provides is embedded within the upcoming vSphere update or not. It has been a long lasting demand of VMKware customers to have some PowerCLI scripts as part of the graphic interface.

Now, HTML 5 has definitely helped ease the frustration with the web client but the client was always a winner. I am not sure what part of the client software was disliked by Vmware product engineers that they forced users to adopt web client. Maybe maintaining multiple client instances was a pain and I can fully understand if that was the case.

The conference is never complete without the ritual Hands on Lab (HOL) sessions. If you are a techie attending VMWorld but do not participate in the HOL sessions then there is something you lose out on – because later on, our work keeps us so busy that there is no time to do the labs. I have always made it a point to run through some important labs and ensure that I receive the practical understanding of some technologies or functionality. The  amount of effort that supports the labs is mind boggling and high end infrastructure really facilitates the labs. So for the lab setup effort – a big thank you to VMware engineers.

Of particular highlight is the ‘Meet the Experts’ sessions – these get full very quickly so booking the sessions early is important. As usual, the townhall sessions by the likes of Duncan Epping, Rawlinson Rivera, Chad Sakac, Chris Wahl are very interesting. We can actually ask questions directly to them and some very creative and deepful insights can be offered by the panel.

I am personally passionate about storage as much as I am about virtualization. To go to Vmworld and not talk about storage is a real loss personally speaking so I am going to learn about some new vendors out there and check out what is happening in the storage world. It’s not that we don’t know about it but everyone has new announcements at Vmworld so booth hopping is very helpful.

On Sunday, I will be starting with the labs, playing for the v0dgeball charity event, and then capping it off with a dinner followed by attending the VMUnderground event.  The rest of the week is crazy so I will follow-up with blog posts on sessions and other key topics of interest to me and as I acquire more knowledge on them.

vSphere Performance Concepts and Troubleshooting at MBVMUG

By

Do you know enough about vSphere Performance concepts and troubleshooting to handle performance problems. Are you able to identify specific metrics that should be reviewed when disk latency goes high.

Have you started testing vSphere Flash Read Cache or know how to troubleshoot issues related to it. Want to learn about vNUMA or learn how best to use esxtop.

Get these and a lot of other questions answered when you attend the upcoming Manitoba VMUG Meeting on October 9, 2014 at Delta Winnipeg.

Click here for details and to register – http://www.vmug.com/p/cm/ld/fid=7843

VMworld final day and recap

By

I am a little delayed posting this final recap but better late than never.  The only reason I am posting this information is to encourage those would like to go to VMworld next year and help out those who could not attend VMworld. As you can see with my previous posts there is a lot of value in attending and the volume of knowledge you gain is tremendous.

My final day at VMworld started off by attending Chris Wahal & Jason Nash’s session ‘vSphere Distributed switches – Deep dive’. I have put out a separate blog post on it but it was a good session to have. I was looking for more technical depth but it still was a great session. I headed over to VMUG Leader Lunch thereafter and the VMUG leaders from various geographies met together and were also joined by the top brass of VMware – Pat Gelsinger, Raghu Rajaram, and Ben Fathi.

They took questions during the lunch and Mr. Gelsinger gave us an insight of where VMware is going in terms of future innovations and how they would further participate in VMUG activities. A new thing that will probably get announced tomorrow is the launch of VMTN – yes you read it right. The scope and depth of the program is unknown to us at this time but we are told it will be comprehensive in nature. So I look forward to that announcement either tomorrow or in the upcoming days. This is not very confidential information by any means and VMware has been focussed on launching VMTN as soon as they could.

The VMware executive also spoke about the future of vCloud Director and vCloud Automation Center. They clarified that vCD is not going away – it will be available only to service providers. vCAC is for the enterprise environment. All features of vCD except for multi-tenancy are available in vSphere 6.0 which is in public beta. So if you are interested to try it out go for it.

A couple of VMUG leaders enquired about the vCloud Air (formerly vCHS) announcement and hopefully as the infrastructure scales up more people will be able to leverage that in different ways. Further conversation occurred around the VMworld announcement of EVO Rail and EVO Rack. As publicly known now there is some level of overlap with vendors that support VMware platforms but that is a common industry trend. If VMware does not push the innovation in that area, either the vendors will be slow to innovate or else competitors will eat into that market. So look forward to some new stuff happening in the EVO area.

We finally ended the VMUG Leader Lunch with awards and one of the VMUG Board of Directors and a person I know for a while – Ravi Venkatasubbaiah won the VMUG President’s Award for exemplary leadership. Congrats to Ravi for this achievement.

I then headed over STO 1153 – Storage Performance Best Practices for Tier1 Applications on Virtual SAN. However, as was common at VMworld this year (unfortunately) the room was switched again and was in a different building a couple of blocks away. Reaching there would have wasted a further 15 minutes so I sat for another session instead – EUC 2551 – Architecture for Next Gen desktops. They spoke about enhancements to the Horizon Suite, VMware’s acquisition of Cloud Volumes and their strategy of further simplifying desktop deployment. I was more interested in hearing to the presenter so I didn’t take any notes on this one.

The final session I attended was INF3037 – How to build and deploy a well run Hybrid Cloud. The presenters spoke about hybrid cloud strategies – enhancements to architectural products, automation tools, and deployment software. Look forward to the presentations being shared post VMworld for all attendees.

The day ended with attending the VMware Canada Customer Reception party that was just across Moscone West at Jillians. I also headed out later for a private dinner with one of our vendor SE’s who was also in town for VMworld.

With a great level of learning and a lot more insight into VMware technologies I am satisfied and pleased that the conference was a success and brought valuable content to its attendees. I also networked with a few great individuals and am returning more enlightened on VMware and vendor technologies.

If you didn’t attend VMworld but would like to view the content – you can sign up for a subscription (last year it was $600) to get access to all VMworld content (presentations, sessions, etc). Not sure about VMworld lab content but I believe that will be available as well. VMUG Advantage membership ($200) last year also provided free access to VMworld content. So check out what’s available and go for it.

To all my friends who met me at VMworld – a shout out to atleast a few of you – Mathew Brender, Sean Thulin, Mark Browne, Jonathan Frappier, Angelo Luciani, Ravi Venkatasubbaiah, Irfan Ahmad, Rob Kyle, Peter Chang, Dwayne Lessner, Chris Halverson, Avram Woroch, Manjeet Bavage, Brandi Collins, Dave Henry – hope to see you again next year.

 

 

STO2496 – vSphere Storage Best Practices: Next-Gen Storage Technologies

By

This was a panel like session that wasn’t vendor specific but broadly gave pointers on new type of arrays – like all vSAN,SDRS, VVOLs, flash arrays, datastore types, and jumbo frame usage etc. It truly lived up to its name – not just by content but also by its duration. The session ran over its scheduled duration of 1 hour and actually finished in 1.5 hours but no one was complaining since there was a lot of interesting stuff.

Presenters – Rawlinson Rivera (VMware), Chad Sakac (EMC),  Vaughn Stewart (Pure Storage)

The session kicked off by talking about enabling simplicity in the storage environment. Some key points discussed were –

1) Use large datastores

  • NFS16Tb and VMFS 64Tb
  • Backup and restore times and objectives should be considered

2) Limit use of RDMs to when required for application support

3) Use datastore clusters and SDRS

  • Match Service Levels on all datastores on each datastore cluster
  • Disable SDRS IO Metric on all flash arrays and arrays with storage tiering

4) Use automated storage array services

  • Auto tiering for performance
  • Auto grow/extend for datastores

5) Avoid Jumbo frames for iSCSI and NFS

  • Jumbo frames provide performance gains with increased complexity and the improvement in storage technology no longer requires Jumbo frames

They spoke about the forms of Hybrid Storage and categorized them based on their key functionality –

  • Hybrid arrays – Nimble, Tintri, All modern arrays
  • Host Caches – PernixData, vFRC, SanDisk
  • Converged Infrastructure – Nutanix, vSAN, Simplivity

Benchmark Principles

Good Benchmarking is NOT easyYou need to benchmark over time – most arrays have some degree of behaviour variability over time

  • You need to look at lots of hosts, VMs – not a ‘single guest’ or ‘single datastore’
  • You need to benchmark mixed loads – in practice, all forms of IO will be flinging at the persistence layer
  • If you use good tools like SLDB or IOmeter – recognize that they are still artificial workloads, and make sure to configure them to drive out a lot of different workloads
  • With modern systems (particularly AFA’s  or all flash hyper-converged), its really, REALLY hard to drive sufficient load to saturate the system. Have a lot of workload generators (generating more than 20K IOPS out of a host isn’t easy)
  • Absolute performance more often than not is not the only design consideration

virtual disk formart can be IO bottleneck

 

Storage Networking Guidance

VMFS and NFS provide similar performance

  • FC, FCoE and NFS tend to provide slightly better performance than iSCSI

Always separate guest VM traffic from storage and VMkernel network

  • Converged infrastructures require similar separation as data is written to 1+ remote nodes

Recommendation: avoid Jumbo frames as risk via human error outweighs any gain

  • Goal is to increase IO while reducing host CPU
  • Ethernet is 1500 MTU
  • Jumbo frames are often viewed as 9000 MTU (9216)
  • FCoE auto negotiates to ‘baby/ – jumbo frame of 2112 MTU (2158)
  • Jumbo frames provide modes benefits in mixed workload clouds
  • TOE adapters can produce issues uncommon in software stacks

jumbo frame performance example

 

Jumbo Frame summary – Is it worth it ?

Large environments may derive the most benefit from Jumbo frames but are also the most difficult to maintain compliance

– All the steps need to align – on every device

Mismatched settings can severely hinder performance

– A simple human error will result in significant storage issue for a large environment

Isolate jumbo frames iSCSI traffic (e.g. backup/replication) – apply CoS/QoS

Unless you have control over all host/network/storage settings, best practice is to use standard 1500 MTU

The future – Path Maximum Transmission Unit Discovery (PMTUD) – It is an IP packet (L3 routers) whereas Jumbo frames are L2 (switches)

It is part of ICMP protocol (same protocol that has Ping, Traceroute, etc) and is available on all modern Operating Systems.

The speakers then got into Data Reduction technologies – they are the new norm (specially de-duplication in arrays)

Deduplication is generally good at reducing VM Binaries (OS and application files). Deduplication block size variances can be impacted by GOS file system fragmentation

  • 512B – Pure Storage
  • 4KB – NetApp FAS
  • 4KB – XtremIO
  • 16KB – HP 3Par
There is a major operational difference between Inline (Pure Storage, XtremIO type) and post-process (NetApp FAS, EMC VNX)
– The advice they provided is that try it yourself or talk to another customer (use VMUGs) – don’t take vendor claims seriously.
Compression is generally good at reducing storage capacity of applications
– Inline compression tends to provide moderate savings (2:1 common) but there are CPU/latency tradeoffs
Post process compression tends to provide additional savings (3:1 common)
Data reduction in Virtual disks
Thin, thick, and EZ-Thick VMDKs reduce to the same size
– Differences exist between array vendors but not between various disk types
T10 UNMAP is still not here in vSphere 5.5 – in the way people ‘expect’ – UNMAP is a SCSI command that allows to reclaim space from blocks that have been deleted by virtual machine.
– It is one of the rare cases where Windows is still ahead – but only in Windows Server 2012 R2
– Manual ‘vmkfstools -k’ option for vSphere 5.1 is available. See Cormac Hogan’s blog post by clicking on this link
– Manual ‘esxcli storage vmfs UNMAP’ in vSphere 5.5 can do > 2Tb volumes (a diagram depicting UNMAP of 15TB over 2 hours was displayed)
– Not all GOS zero properly which means you may not reclaim space properly via UNMAP
An entire set of Horizon specific and Citrix specific Best Practices to follow (vSphere config and GOS config)
Rawlinson who had stepped away from the stage as Chad and Vaughn spoke about storage stuff earlier, then came on to talk about VMware vSAN Best Practices
Network Connectivity
– 10GbE Preferred Speed (previously 1Gb connectivity used to be good enough. But vSAN works best if 10GbE connectivity is available – specifically because of the volume of data that travels over the network)
– Leverage vSphere Distributed Switches (vDS) – NIOC is not commonly used in most organizations but acts like SIOC where it performs QoS for the network traffic and throttles traffic to offer the best performance. Specifically, vDS offer the best flexibility and control over network performance with the feature set that is required in Enterprise environments.
Storage Controller Queue Depth – The Queue Depth setting is something that should not be setup manually anymore unless you are observing performance issues. VMware has specifically reviewed it and officially set it up at 256. In some environments however you may have a requirement to change. Just don’t change it for the sake of setting something up manually without allowing uninterrupted operation and associated monitoring of default values.
– Queue depth support of 256 or higher
– Higher storage controller queue depth will increase
  • Performance
  • Resynchronization
  • Rebuilding Operations
  • Pass-through node preferred
Disks and Disk Groups
  • Don’t mix disk types in a cluster for predictable performance
  • More disk groups are better than one

The session finally concluded at 6:30pm and after a few hand shakes everyone was on their way. But it was completely worthwhile and goes on to show why attending VMworld offers great insights that you cannot learn in a 4 day course. The structure and content of these sessions is not limited by any way.

 

INF1864 – Software Defined Storage (What’s Next)

By

Chad Sakac is a very dynamic EMC executive with a reputation in the industry of being highly knowledgeable, technical, and innovative professional. He talked about the next steps in Software defined storage in the session INF1864 and I was right there to capture some key information to expand on my Storage Architecture knowledge.

He started off his talk by highlighting a few things and then spoke about the upcoming new technologies.

What is software defined

  • Decoupling and abstracting control and policy from physical stuff that does the work
  • Where the physical stuff that does work (data plane) can be software on commodity hardware – e.g. VSAN, VASA
  • Programmable infrastructure APIs: automate everything

 

Four ‘Data Plane’ architectures

  • Clustered Scale Up & Down e.g. Nimble, VNX, Nexenta
  • Tightly Coupled Scale out clusters – e.g. Hitachi, 3Par
  • Loosely Coupled Scale out -e.g. vSAN, ScaleIO, Nutanix, Simplivity
  • Distributed share nothing  e.g Swift, AWS S3

 

What to expect next

  • SDS control planes maturing
  • VVOL
  • ViPR 2.0
  • Cinder
  • SDS data services moving to real world use
  • Some Data Services become ‘features’
  • Acceleration of ‘Old App’ Hyper-Converged Stacks
  • Acceleration of ‘New App’ Hyper-Converged Stacks

VVOL EMC update

  • VNXe3200 will be first to to use VVOL
  • VMAX3 will be right behind VNXe
  • VPLEX will support VVOLs (for VERY async vMotion)
  • ViPR Controller will support VVOL control
  • XtremIO will support VVOL
  • ScaleIO will support VVOL
  • VNX will get VVOL through same path as VNXe
  • C4 – storage containerization that is adopted by VNX as a codestack. Used by entire VNX family

If you are testing the vSphere 6 beta release and trying out VVOLs then feel free to forward any feedback to veena.joshi@emc.com

Now EMC offers ScaleIO as a vCenter plugin to quickly deploy it. The plugin automatically goes through and deploys nodes and clients to access the underlying storage after some regular configuration stpes.

If you need to store stuff beyond VMDK’s then you can use ScaleIO vs vSAN (since a lot of people are thinking its the same as vSAN)

RecoverPoint for VMs 

download fully functional EMC recover point at no charge with no time limit – October 2014.

For image archive- An S3 compliant datastore is the best option to store data

‘Phoenix’ Server Hardware – designed for EVO Rail – It is the only server that EMC sells and since it is not in the server business there is usually not much discussion around it.

It is a 2U server with internal disk and you can google it for more details.

Big Trends

  • Increasing application diversity
  • World of how apps run is geting more diverse
  • SDS + commodity HW is awesome and belongs in many places but tightly coupled architectures will bias to appliance

The session ran slightly over time (Chad is so passionate while talking and takes questions so his sessions usually tend to go long) and we just glimpsed through the last slide.