Queue Depth

This was a panel like session that wasn’t vendor specific but broadly gave pointers on new type of arrays – like all vSAN,SDRS, VVOLs, flash arrays, datastore types, and jumbo frame usage etc. It truly lived up to its name – not just by content but also by its duration. The session ran over its scheduled duration of 1 hour and actually finished in 1.5 hours but no one was complaining since there was a lot of interesting stuff.

Presenters – Rawlinson Rivera (VMware), Chad Sakac (EMC), Vaughn Stewart (Pure Storage)

The session kicked off by talking about enabling simplicity in the storage environment. Some key points discussed were –

1) Use large datastores

NFS16Tb and VMFS 64Tb
Backup and restore times and objectives should be considered

2) Limit use of RDMs to when required for application support

3) Use datastore clusters and SDRS

Match Service Levels on all datastores on each datastore cluster
Disable SDRS IO Metric on all flash arrays and arrays with storage tiering

4) Use automated storage array services

Auto tiering for performance
Auto grow/extend for datastores

5) Avoid Jumbo frames for iSCSI and NFS

Jumbo frames provide performance gains with increased complexity and the improvement in storage technology no longer requires Jumbo frames

They spoke about the forms of Hybrid Storage and categorized them based on their key functionality –

Hybrid arrays – Nimble, Tintri, All modern arrays
Host Caches – PernixData, vFRC, SanDisk
Converged Infrastructure – Nutanix, vSAN, Simplivity

Benchmark Principles

Good Benchmarking is NOT easyYou need to benchmark over time – most arrays have some degree of behaviour variability over time

You need to look at lots of hosts, VMs – not a ‘single guest’ or ‘single datastore’
You need to benchmark mixed loads – in practice, all forms of IO will be flinging at the persistence layer
If you use good tools like SLDB or IOmeter – recognize that they are still artificial workloads, and make sure to configure them to drive out a lot of different workloads
With modern systems (particularly AFA’s or all flash hyper-converged), its really, REALLY hard to drive sufficient load to saturate the system. Have a lot of workload generators (generating more than 20K IOPS out of a host isn’t easy)
Absolute performance more often than not is not the only design consideration

Storage Networking Guidance

VMFS and NFS provide similar performance

FC, FCoE and NFS tend to provide slightly better performance than iSCSI

Always separate guest VM traffic from storage and VMkernel network

Converged infrastructures require similar separation as data is written to 1+ remote nodes

Recommendation: avoid Jumbo frames as risk via human error outweighs any gain

Goal is to increase IO while reducing host CPU
Ethernet is 1500 MTU
Jumbo frames are often viewed as 9000 MTU (9216)
FCoE auto negotiates to ‘baby/ – jumbo frame of 2112 MTU (2158)
Jumbo frames provide modes benefits in mixed workload clouds
TOE adapters can produce issues uncommon in software stacks

Jumbo Frame summary – Is it worth it ?

Large environments may derive the most benefit from Jumbo frames but are also the most difficult to maintain compliance

– All the steps need to align – on every device

Mismatched settings can severely hinder performance

– A simple human error will result in significant storage issue for a large environment

Isolate jumbo frames iSCSI traffic (e.g. backup/replication) – apply CoS/QoS

Unless you have control over all host/network/storage settings, best practice is to use standard 1500 MTU

The future – Path Maximum Transmission Unit Discovery (PMTUD) – It is an IP packet (L3 routers) whereas Jumbo frames are L2 (switches)

It is part of ICMP protocol (same protocol that has Ping, Traceroute, etc) and is available on all modern Operating Systems.

The speakers then got into Data Reduction technologies – they are the new norm (specially de-duplication in arrays)

Deduplication is generally good at reducing VM Binaries (OS and application files). Deduplication block size variances can be impacted by GOS file system fragmentation

512B – Pure Storage
4KB – NetApp FAS
4KB – XtremIO
16KB – HP 3Par

There is a major operational difference between Inline (Pure Storage, XtremIO type) and post-process (NetApp FAS, EMC VNX)

– The advice they provided is that try it yourself or talk to another customer (use VMUGs) – don’t take vendor claims seriously.

Compression is generally good at reducing storage capacity of applications

– Inline compression tends to provide moderate savings (2:1 common) but there are CPU/latency tradeoffs

Post process compression tends to provide additional savings (3:1 common)

Data reduction in Virtual disks

Thin, thick, and EZ-Thick VMDKs reduce to the same size

– Differences exist between array vendors but not between various disk types

T10 UNMAP is still not here in vSphere 5.5 – in the way people ‘expect’ – UNMAP is a SCSI command that allows to reclaim space from blocks that have been deleted by virtual machine.

– It is one of the rare cases where Windows is still ahead – but only in Windows Server 2012 R2

– Manual ‘vmkfstools -k’ option for vSphere 5.1 is available. See Cormac Hogan’s blog post by clicking on this link

– Manual ‘esxcli storage vmfs UNMAP’ in vSphere 5.5 can do > 2Tb volumes (a diagram depicting UNMAP of 15TB over 2 hours was displayed)

– Not all GOS zero properly which means you may not reclaim space properly via UNMAP

An entire set of Horizon specific and Citrix specific Best Practices to follow (vSphere config and GOS config)

Rawlinson who had stepped away from the stage as Chad and Vaughn spoke about storage stuff earlier, then came on to talk about VMware vSAN Best Practices

Network Connectivity

– 10GbE Preferred Speed (previously 1Gb connectivity used to be good enough. But vSAN works best if 10GbE connectivity is available – specifically because of the volume of data that travels over the network)

– Leverage vSphere Distributed Switches (vDS) – NIOC is not commonly used in most organizations but acts like SIOC where it performs QoS for the network traffic and throttles traffic to offer the best performance. Specifically, vDS offer the best flexibility and control over network performance with the feature set that is required in Enterprise environments.

Storage Controller Queue Depth – The Queue Depth setting is something that should not be setup manually anymore unless you are observing performance issues. VMware has specifically reviewed it and officially set it up at 256. In some environments however you may have a requirement to change. Just don’t change it for the sake of setting something up manually without allowing uninterrupted operation and associated monitoring of default values.

– Queue depth support of 256 or higher

– Higher storage controller queue depth will increase

Performance
Resynchronization
Rebuilding Operations
Pass-through node preferred

Disks and Disk Groups

Don’t mix disk types in a cluster for predictable performance
More disk groups are better than one

The session finally concluded at 6:30pm and after a few hand shakes everyone was on their way. But it was completely worthwhile and goes on to show why attending VMworld offers great insights that you cannot learn in a 4 day course. The structure and content of these sessions is not limited by any way.