vmware

EMC Elect 2016 – What it means for me in 2016

February 1, 2016 By asceticadmin

I got an exciting piece of news when I woke up this morning and as usual looked at my twitter feed. The EMC Elect list of 2016 was announced and I was honoured again with this title this year.

Since 2013 when this program was first announced, I have had the fortune of being part of the EMCElect group which is purely honorary in nature and like all vendor programs is about evangelizing and talking about technology – but obviously reflects and promotes EMC products.

The significance of EMC Elect title may mean different things for different people. Being an EMC customer I definitely look at it from the perspective that it allows me more insight into EMC products that I use and also get to share my knowledge with others through various social platforms. Overall, I have always loved sharing my knowledge and thoughts on various technologies so this title is a wonderful reward in return and I have a similar title – VMware vExpert as well. I do utilize my self learning to understand and get knowledge on other technologies as well but that’s a different track.

All social community programs have employees, evangelists and customers and their goals may differ. From my standpoint, at no point of time am I asked to specifically talk about EMC products or promote them – as I am a EMC customer I don’t want to do EMC or VMware’s marketing for them. But when you love a technology and use it or get familiar with it – anyone interested and liking the technology will share that story. That is exactly what I do and the program appreciates a customer talking about the good things they do – I share the things that need improving in private .

Since the program started in 2013, it has evolved significantly. It all started with the fabulous efforts of my dear friend – Matt Brender (@mjbrender), followed by by Sean Thulin (@sthulin) and now for last couple of years successfully managed by Mark Browne (@dathbrun) and most recently Mary Kilgallen (@marykilgallen) who has joined forces with Mark in 2015.

From online knowledge sharing expert sessions to VMworld, EMCWorld, and other worldwide conversations – it has been present in all places.

Personally, I want to see a couple of benefits – Eval/lab license to try out new EMC products, trial or special licenses for EMC cloud being on top of my list.

Attending EMCWorld and VMworld as an EMCElect is special because you get exclusive access to people you want to interact with (engineering and product teams), are invited to special events – some executive and some technical, and finally at the end of the day social events to hang out and network with other professionals.

So there’s a lot of benefit in being an EMCElect and I sincerely appreciate being part of this fabulous group.

Monitoring and Troubleshooting Virtual SAN, Current and Future

September 3, 2015 By asceticadmin

Health Check Plugin in vSphere 6

Used for Virtual SAN in vSphere 6 – improved

Does 30 health checks – explains nature of failure and so on

vRealize Operations Management Pack for Storage Devices (MPSD) latest update released last week

vROps MPSD – Custom Dashboards

Build bespoke VSAN cluster info

Disk Group throughput

Log Insight – super tool for looking at logs generated by ESXi host and vSAN. Allows us to do analytics on logs – patterns, behaviours, intermittent issues

VSAN extension pack available with Log Insight

Real Life troubleshooting scenarios

We have realized there are gaps in Management story around VSAN – VMware has been working to provide right tools at the right places.

Low level – esxcli, rvc, observer

Need vSphere web client and vRealize as a standard way of consuming things. PowerCLI is also a tool in use

Things you probably want to check

components must be on HCL
Confirm network is good – e.g. Multicast
Make sure VMs can be deployed successfully
Test underlying storage components with a “stress test”
Inject failures, ensuring that VMs remain available
Test performance of Virtual SAN (VSAN)

Scenario#1

Assume HCL check fails.

Download the latest HCL file – maybe the device/driver/firmware support status has changed

Scenario#2

Run a storage performance test. This doubles as a stress test – VMware has released a tool called HCIbench – test performance of infrastructure

To test a bad drive don’t just remove it – run the special error injector with health check – simulate drive going bad – see POC guide

Scenario#3

Injecting failures in Virtual SAN can be done quite simply
- Hosts (reboot or power off)
- Network (disconnect uplinks or disable VSAN traffic service)
- Disks (special error injector wit health check – simulate drive going bad – see POC guide)
Use the health check to understand failures

When you pull out a drive – VSAN knows it – VSAN will wait 60 minutes before remediation

When a drive fails – VSAN knows that and puts it in Degraded mode. VSAN will perform immediate remediation

VSAN requirements are three nodes – it can tolerate failure of one node. But if we do four nodes – we can have one node failure. But it protects us against one more node failure

So it is preferred that you start with 4 nodes for VSAN.

Multicast – Multicast configuration is the most common issue. When alarm in Proactive Tests shows status as failed then there is an issue with multicast. This proactive test will verify if multicast performance is acceptable for VSAN cluster

Sneak peak of ‘Performance Service’ – new tool. To be released hopefully in near futue

Stores history VSAN performance statistics

Stored on VSAN itself

Always-on

Fully Integrated – no need to install anything

Exposed via vSphere Web Client (and API)

Distributed architecture, built directly into ESXi

No network traffic going outside the cluster
No CPU/Memory usage in VC
Tiny impact on ESXi hosts

Benchmarked 50K IOPS (see image) using Performance Service

VSAN – When performing benchmark tests note that read cache in VSAN needs to warm up

Outstanding Ios (OIO) – not enough outstanding I/O for VM to push performance to its limits

You need to have enough VMs to perform a true performance test. We don’t run one giant VM but we run multiple VMs to ensure enough parallel I/O – that is what VSAN is built for.

Two new metrics in performance graphs – Delayed I/O percentage and Delayed I/O average latency

How many IOs were delayed
How many IOs did not make it to the pipeline

vMotion enhancements in vSphere 6

September 2, 2015 By asceticadmin

Long distance vMotion – now supported over network latency upto 150 ms – support for Geo distances

In vSphere 5.5 it was 10 ms latency

Standard vMotion guarantee – 1 sec execution switchover time

Batched RPCs, tcp congestion window handoff

vMotion – had to find a way to bypass TCP delays

Used control channel to send all content, data channel was used to send VM data

Very large bandwidth delay product – tcp performs poorly

Revised tcp algorithm – condition control algorithm not suited for large bandwidth delay projec
Changed the delay product to HSTCP

Packet loss was another concern –

Avoided packet drop within ESX host by adding mechanisms like flow control
Helped target packet loss rate reduce to 0.01%

What if the ESX hosts are in different subnets and you want to do long distance vMotion

vMotion across L3 network

Use ESX tcp/ip network stack virtualization (an ESX host can have multiple default gateways) – default gateways of the management network.
vMotion gets its own network stack instance (separate from management network)
Use default gateway for vMotion to go across L3 (and go across subnets)

VM network still requires L2 network stretching

DRS cluster requirements

All ESX hosts on L2 or all on L3 vMotion network
Mixed mode of L2 and L3 not supported

L2 adjacency limitations

Fault tolerance not supported
DPM using Wake on LAN requires subnet directed broadcast (if you are not different subnets then routers filter out broadcast traffic).

What if ESX hosts have different virtual switches

vSphere standard switch (VSS)
vSphere Distributed switch (VDS)

Ability to go from one vds to another vds – transfers all properties (e.g. Network io control, bandwidth restrictions etc)

Don’t support downgrade from vds to standard switch – since standard switch does not have vds type capabilities

What if ESX hosts are managed by different vCenters

In Vsphere 6 there is ability to vMotion across vCenters

Simultaneously change compute, storage, networks, and vCenter servers
Leverage vMotion without shared storage

Works with local, metro, and long distance vMotion

Preserve instance UUID and bios UUID (VM UUID)

Preserve VM historical data

Events, alarms, task history (pulled from original vcenter – not moved to new vCenter)
Preserve all HA and DRS properties, affinity/anti-affinity rules

SSO domain support

Vsphere web client requires same SSO domain
API support across SSO domains

What if you can take advantage of replication to the other site

Replication assisted vMotion

vSphere 6 supports Active/Active Async storage replication

Disk copy takes majority of migration time
Use replication to avoid disk copy
Leverage virtual volume (VVOL) technology

VVOL are primary unit of data management going forward – storage array knows vvol mapping to vm

Secondary site storage array promotes LUN containing replicated data

Active – active async replciation

Switch replication mode to sync
Migration start
- Prepare destination ESX host for VVOL binding
- Switch from async to sync replication
Migration end
- Complete vvol binding on esx hosts
- Switch back to async replication

vSphere 6.0 vMotion features interop

What if you have 40GBE NICs for vMotion network (vMotion performance and scalability improvements)

vMotion scalability

-Rearchitect vMotion to saturate 40GbE NIC

Zero copy transmission
New threading model for better CPU utilization on the receive side
Reduce locking

For e.g in maintenance mode – we may have to move 400gb memory from one host to another host. The entire maintenance mode will take only 5 mins on 40GbE env.

Reduce vMotion execution switchover time (improved)

Constant time VM memory metadata updates
Not a function of VM memory size

Reduce stack overhead

Improve VM power on time (all power on optimizations have been optimized)

Performance and Debugging

How to Gauge vMotion performance

Migration time (memory, disk, total)
Switchover time
Impact on guest applications
- Application latency and throughput during vMotion
- Time to resume to normal level of performance after migration

Monster VM vMotion performance

In vsphere 5.5 each vMotion was by default assigned 2 helper threads – running on 2 cores .. Could only go 20GB/sec
By increasing number of helper threads (tuned ESX) – throughput increased slightly
In 60Gb scenario the increase of helper threads does not help futher
In vsphere 6.0 locking has been removed and thus dynamically creates appropriate number of tcp channels and helper thread is automatically created. Thus performance improves significantly without any tuning

Debugging tips

Each vMotion has a unique id associated with it called Migration ID

Grep that migration id since it is unique across both source and destination hosts

From web client – select VM – and go to tasks

See the high level details in the task info – Vmware is adding migration id there

VPXD – find operation id of vMotion (in VPXD logs)

What’s next for vMotion

Cross cloud vMotion

vMotion between vCLoud air (vCA) and on-premise datacenter
No vendor lock-in; vMotion to vCA and from vCA to on-premise
Support for vSphere 5.5 for on-premise version (will be backwards compatible)

Non-volatile Memory (NVM) – disks, ssd’s

NVM resides in a Dual Inline Memory Module (DIMM)
Exposed as Memory and Virtual disk to VMs (persistent memory and disks through SCSI card)
Enable vMotion for VMs and NVM
Explore NVM to improve vMotion performance and scalability

Active/Passive Storage Replciation

Leverage broad partner ecosystem to optimize disk copy
VVOL required to reverse replication direction after vMotion
vMotion support for RDMA (don’t have to use CPU and can use RDMA for performance improvement instead)

Conclusions

Vsphere 6 vMotion is a big step towards vMotion anywhere

Cross geo boundaries
Cross management boundaries
Cross cloud vMotion

Best practices for performance tuning of latency sensitive workloads in vsphere VMs

September 1, 2015 By asceticadmin

Best practices for performance tuning of latency sensitive workloads in vsphere VMs

Visit original link for full details

http://www.vmware.com/resources/techresources/10220

To download whitepaper – click here

Abstract from VMware site –

This white paper summarizes findings and recommends best practices to tune the different layers of an application’s environment for latency-sensitive workloads—those which customers are looking at optimizing for a few microseconds to a few tens of microseconds end-to-end latencies (not workloads in the hundreds of microseconds to tens of milliseconds end-to-end-latencies).

We (a cross-functional team at VMware) investigated the performance of an in-memory, distributed data management platform, measured in the number of puts/second of 1KB data objects from one VM on one ESXi host to another VM on a different ESXi host across physical Ethernet networks. The paper provides recommendations that helped us go from 4400 puts/sec when we started, to 7200 puts/sec for a specific single-threaded, single vCPU application performance benchmark, which is at 78% of the score of 9200 puts/sec on bare metal on the same hardware.

We also investigated the performance of two latency micro-benchmarks, one for Infiniband devices and another for networking devices, in VM DirectPath I/O (pass-through) mode. Applying the recommendations reduced latency and therefore increased the score of these latency micro-benchmarks in a virtualized environment, bringing it closer to bare metal performance.

– See more at: http://www.vmware.com/resources/techresources/10220#sthash.ZNYRBzHV.dpuf

Infrastructure as a Service – VMworld Hands on Lab

September 1, 2015 By asceticadmin

VMware has a lab around the use of deploying VM’s using a workflow system in an automated fashion – using vRealize Automation (formerly vCAC).

The lab is very straightforward and at a regular technical level to make things easier. Blueprints have been created in advance. But you get to understand how to initiate a VM creation request and then switch to approving the VM, modifying the resources of a VM, changing users/roles for the approval mechanisms, and finally deploying VMs.

The workflow mechanism is what most organizations use for automated VM provisioning and the lab walks you through the process.

All senior IT professionals must look into the adoption of vCAC to automate your own IT environment as much as possible and this is a step in the right way.

vSphere 6 Challenge – Hands on Lab

September 1, 2015 By asceticadmin

VMware has some interesting labs this year and one of the top labs is the vSphere 6 Challenge Lab. This was obviously one of the labs to participate in and it had some good stuff to know about. The vSphere 6 Challenge asks you to put on the thinking cap and go through fictional scenarios in order to fix real world issues. The issues were related to Operational and Performance areas. It started off with some basic stuff like how to fix copy/paste issues or fix CPU performance problems. A benchmark tool is part of the lab so it is good to see how the changes you are doing are improving/degrading VM performance. Things like – have a diconnected host or VM which will not power on and a mysteriously poorly performing VM are covered here.

So I will recommend that you go to the lab area today and do take this one – for those of you who are not at VMworld, hopefully you will have the chance to take it soon remotely after VMworld is over – if it is released publicly prior to VMworld Barcelona.

VMWorld 2015 – the journey begins

August 31, 2015 By asceticadmin

With each new year there is further growth in the field of virtualization and new products/features are released. When it comes to showcasing the announcements, there is no better platform than VMWorld.

This year however, the focus is slightly different from my perspective – for e.g. There will be no major product release announcements and no big bang approaches – vSphere 6 was launched last year so it makes sense that Vmware might actually improve feature set and add new functionality. I am specifically seeing the adoption of Cloud channel into a more sustainable approach of hybrid infrastructure deployments. The likes of which include working with NSX, storage volumes (vVOLs), automation enhancements, and possibly replication enhancements. The focus as usual will also be in the field of EUC – with Horizon going maintstream.

I am personally looking forward to attending a few special sessions – one of which is and will always be about Performance. The Performance series sessions are always popular because technology keeps changing and the performance measurement tools and feature set also evolves. A critical piece of the Performance measurement is going to be around scripting – and PowerCLI has plenty of potential to do a lot for many organizations. It would be nice to see if some of the functionality that PowerCLI or PowerShell provides is embedded within the upcoming vSphere update or not. It has been a long lasting demand of VMKware customers to have some PowerCLI scripts as part of the graphic interface.

Now, HTML 5 has definitely helped ease the frustration with the web client but the client was always a winner. I am not sure what part of the client software was disliked by Vmware product engineers that they forced users to adopt web client. Maybe maintaining multiple client instances was a pain and I can fully understand if that was the case.

The conference is never complete without the ritual Hands on Lab (HOL) sessions. If you are a techie attending VMWorld but do not participate in the HOL sessions then there is something you lose out on – because later on, our work keeps us so busy that there is no time to do the labs. I have always made it a point to run through some important labs and ensure that I receive the practical understanding of some technologies or functionality. The amount of effort that supports the labs is mind boggling and high end infrastructure really facilitates the labs. So for the lab setup effort – a big thank you to VMware engineers.

Of particular highlight is the ‘Meet the Experts’ sessions – these get full very quickly so booking the sessions early is important. As usual, the townhall sessions by the likes of Duncan Epping, Rawlinson Rivera, Chad Sakac, Chris Wahl are very interesting. We can actually ask questions directly to them and some very creative and deepful insights can be offered by the panel.

I am personally passionate about storage as much as I am about virtualization. To go to Vmworld and not talk about storage is a real loss personally speaking so I am going to learn about some new vendors out there and check out what is happening in the storage world. It’s not that we don’t know about it but everyone has new announcements at Vmworld so booth hopping is very helpful.

On Sunday, I will be starting with the labs, playing for the v0dgeball charity event, and then capping it off with a dinner followed by attending the VMUnderground event. The rest of the week is crazy so I will follow-up with blog posts on sessions and other key topics of interest to me and as I acquire more knowledge on them.