EMC World Round Up

By all accounts, EMC World was a tremendous success for vLab!  We saw great buzz around the show and on social media.  Attendees were excited to gain access to EMC’s new products and visited vLab as a result.  vLab executed a total of 3847 labs which was up 17% YoY.  The system itself had 100% uptime, with only a few small hiccups along the way.  The automation of the system was going full bore for most of the week, provisioning 24,902 VMs over the course of the show.  That’s 10.6 VMs provisioned per minute!  If you include the tears downs, that’s nearly 50,000 provisioning / deprovisioning actions in the four days of the show.  Simply tremendous!

As the show wound down, the team had a feeling of relief.  It is a very stressful experience, even when it went as well as it did.  Thursday was busier then normal, with a few stragglers hanging on even as we tore down the room.  We had to do setup in reverse – tearing down all the thin clients and monitors, boxing them all up, squaring with the equipment rental company.  Even when we’re done delivering labs, there’s more work to do!  Once we were done with that, the show setup crew comes in and begins to tear down the walls.  Within the space of a few hours, its as if we never existed.  Amazing the transformation these convention spaces go through!

Once the show is completed, we have some time to relax before we all travel back to our homes for a well earned weekend.  We all went out to dinner together and shared a few more hours together.  This team is really amazing.  Every person is dedicated to the success of the show – and it shows.  We come from separate organizations but those lines disappears for this effort.  It is and amazing experience and is what I believe teams should aspire to be.  We’re all in this together.




Thanks for reading!  If you found this post but haven’t read the earlier ones, links are below.

Day 1 – Wheels down in Vegas
Day 2 – Gotta have snacks!
Day 3 – As ready as we’ll ever be…
Day 4 – Release the hounds!
Day 5 – Keep It Between the Lines!
Day 6 – Technical Overview of vLab

vLab Technical Overview – vLab at EMC World – Day 6

We usually aren’t that busy in the latter half of the show but today has surprised us.  We were totally full on the main floor and had to open all three Guided vLab rooms to students.  At peak, we were serving up right around 200 labs.  This is certainly the busiest we’ve been this year!  I’m going to use this post to go through a little of what makes vLab tick under the covers.  How is it that we support such a large event?

vLab relies on the capabilities of XtremeIO, Cisco UCS, VMware vSphere, vCloud Director, and custom software.  I will walk through the stack from the bottom up…

The underlying hardware starts with a 6x40TB XtremeIO cluster.  That is hosted in EMC’s facility in Durham, NC.  The array has amazing capabilities and truthfully, it is not overly taxed during the show.  We also run a fair amount of additional compute capacity off the same array during the other 51 weeks of the year.  It’s actually busiest when we run our unmaps, and we’ve only had to do that once (the night before the show started).

On top of the XtremeIO, we run Cisco B240 servers, most with 512GB of RAM each.  There are a few at 256GB RAM and we use those for content that is CPU, rather than RAM, intensive.  We are leveraging 192 such hosts to support the show.  They live in Durham as well and provide the raw horsepower necessary to run the show.  88TB of RAM seems like a lot but with some labs checking in at over 200GB RAM each, it gets burned through fairly quickly.

We run a mix of vSphere 5.1 and 5.5 for our hypervisors.  The requirements for this layer are driven by the content.  Many of the Virtual Storage Appliances we run have specifics needs that lock us into one version or another of ESX.  Other portions of our infrastructure run 6.0 but not for EMC World this year.  On top of ESX, we are running vCD – it handles our orchestration.

We break our capacity into widgets – groups of 32 hosts.  Each widget gets divided into two oVDCs / two vCenters and gets its own vCD instance.  This may seem like overkill but in addition to the physical resources required to run the labs, we must also provide the automation bandwidth to support provisioning and tearing down potentially hundreds of vApps per hour.  In fact, we peaked at around 700 vApp provisioning / deprovisioning actions / hour.  Multiply that by and average of 6 VMs / vApp and we’re performing 4200 VM provisions / deprovisions per hour at peak.  More than 1 VM / second for you math geeks!

Here’s a basic picture:

Couple notes on this picture… it refers to CEC-D which stands for Common Engineering Cloud – Dynamic.  This is an internal service – self service VM provisioning for EMC engineers.  Like vLab, it is engineered / supported by my team and runs on the same stack as vLab.  Also, we are leveraging NSX within CEC but are not yet using it within vLab.

Over the top of all this hardware and virtualization stands our vLab application.  Our application provides the user experience of vLab, lab scheduling / assignment, lab queue management, and interfaces with the APIs of both vCD and vC to control all the orchestration we need.

Starting from the layer closest to the virtualization… the vLab application divides our total population of widgets into cloud segments.  These segments provide a logical grouping of our capacity and provide buckets that we can fail between should something happen.  This is our “Plan B” for the show should something go wrong in the cloud segment we are using.  We have many of the labs pre-staged in another cloud segment and can fail over to that segment should something go wrong in our primary capacity.

For each cloud segment we have a “backend” application that ties directly to the vC and vCD APIs.  It is also this layer that controls the queueing we use to provide the pre-population of labs within vLab.  This allows us to have a vApp ready for a student when s/he sits down in vLab rather than having to wait for provisioning.  Once a student chooses a lab, it is assigned to her and the backend spins up another vApp to refill the queue.  This isolates the time / processing it takes to bring the lab into being.  Above the backend stand the adapter and consumer layers.  These components bring together multiple cloud segments and expose them to the upper UX layer.

The UX of vLab consists of a few major components.  HOLA (Hands on Labs Application) is the main interface that is in play for EMC World.  We also have a standard portal that’s available for EMC and Partner SEs to provide sales demos to customers.  There are additional admin interfaces for managing events and user sessions.  HOLA is fully configurable to include the specific labs intended for an event.  The user is presented with a series of labs and is able to choose one to run through.  It is a fairly clean interface and allows the user to review vLab’s whole catalog of offerings.

Categories of labs are down the left side and are essentially filters on the main catalog.  It allows a student to zero in on a topic / lab of interest.  We also show a bit of social media and other items of interest.  The general purpose is to get the student to select a lab and get into it as quickly as possible.  Once the user selects a lab, the consumer / adapter layer makes a request to the backend for a session (which is in reality a vApp designation and duration of the request, for example, a Unity lab for 2 hours).  The backend receives the request and assigns a running deployment of that lab to the session request.  This is then passed back up to the UX and the user is connected via RDP into their session.  If no deployment of the lab is available (pre-pops are exhausted), the backend spins up a new vApp and assigns it to the session request.  Our goal is to avoid that completely for EMC World and so far we have been 100% successful.  We don’t want students to wait!

We watch the entire stack with the Eye of Sauron, Zabbix, and the proprietary tools that accompany each component.  We monitor physical resources across all of our capacity as well as the resources consumed by every vC, vCD, and our application.  These resources are normally leading indicators of issues that will reveal themselves as performance problems on the show floor.  We also watch various other application components – internal queues, web service connections, RDP connections.  It all looks like this:

Thats a quick overview of how vLab does what it does – thanks for reading!


Chris and Joe Tucci

Keep It Between the Lines! – vLab at EMC World, Day 5

We finished up yesterday with over 1000 labs taken within vLab – 1075 to be exact!  It was a great day by all accounts.  We had a few hiccups here and there but no sustained issues.  The team recovered very well and on the whole, everything ran smoothly.  We expect another busy day today with keynotes and product announcements happening this morning.  We usually get a rush with attendees looking to get a glimpse of new and improved offerings.  We usually have about 60% of the seats full but after keynotes we can get filled to capacity.  When that happens, we have the option to open our Guided vLab classrooms for general use.  If we’re full and there isn’t a class going on, we’ll leverage that option.

Also I met this guy in the Expo:

Chris and Joe Tucci

It’s mid-morning and vLab is completely full!  We have even expanded the main floor into one of the Guided vLab rooms so we have a total of 150+ attendees taking labs.  Amazing stuff!  The backend is humming along just fine.

We are watching the big board in the NOC carefully to catch any issues before they become impactful to the show floor.  There are a ton of components to monitor and many multiples of each… all together they make up vLab so all must be watched.

Release The Hounds! – vLab at EMC World, Day 4

Day 4 begins rather early… we begin at 6am.  We are doing our final show checks – making sure all the labs have spun up properly and the platform itself is looking healthy.  The registration desk is open at 7am – attendees can come and sign up for Guided vLabs.  There’s a nervous energy in the NOC… anticipation of opening, hopeful that all will go well.  Calm before the storm so to speak.

With customers showing up in a few short hours, all eyes are on the vLab system.  We have our monitoring in place and visible in the NOC.  @eoinok77 did a great job configuring our eye of Sauron.  Second year in a row it has been watching over us… will become a bit of a tradition.

@timmywu and @eoinok77 are paying attention

It’s early and some of us didn’t stop for breakfast on the way.  Dmitri grabbed a breakfast of champions from our snap table.  A Starbucks drink and Oreos will get your day going!

As customers begin to finally file in, activity in the back increases.  Slightly before open, we ran into an issue with PSOD on a couple hosts.  The labs running on those hosts died but we were able to clean up the backend with minimal impact to the customers on the floor.  Issues like this will crop up from time to time (chaos monkey happens).  Our goal is to minimize the impact when they do.  Constant communication is critical so the NOC and the front of house can stay in sync.

Its about an hour after open and we’re already over half full.

You can see in the front of the lab, the big screen is displaying the number of labs that have been taken, number of VMs created and destroyed, lab popularity, and other stats.  It will climb during the week… hopefully setting some records along the way.

Update 1 … around 4pm LV time

The day has been running fairly smoothly and the lab has been pretty full since we’ve been open.  The main room of vLab has 130 stations and is open for any lab.  We have also run a handful of Guided vLabs – each with about 20 students, all taking the same lab.  vLab got a shout out in the Service360 (press release) keynote when the speaker mentioned that customers should head to vLab and take a session on MyService360.  Great publicity for us!  A lot of the content in the vLab Experience is aligned with product launches.  In addition to Service360, for example, we also have a couple labs on Unity (press release), the new offering from EMC’s midrange.  As of now, a Unity lab is the most popular in vLab – almost 100 sessions of “Unity – Simplified Storage Management and Administration” have been taken by customers.  On the whole, the day has been very good so far – we have delivered close to 900 labs consisting of nearly 5000 VMs.  And we have 3+ hours left!

We also got some good publicity on @emctv as they came and interviewed @vlabkreis.  She talked all about the vLab Experience and the opportunity it provides for the attendees of EMC World.  Education on the entire catalog of EMC’s products is readily available!

Another member of our team, @vfspb was watching live on the broadcast stream!

Ready As We’ll Ever Be! – vLab at EMC World, Day 3

Day 3 is our last day of prep before customers arrive.  We have to button up the vLab area on the show floor and make sure that all the content and the platform is good to go.  The cables we needed have arrived so getting the thin clients functional is the focus.  Overnight, the platform team drained all vApps out of the system and have pre-popped all labs to pre-set amounts.  We have more pre-pops for labs we expect to be popular and less for those we don’t.  The whole point is to prevent students from having to wait for a lab to provision without consuming an inordinate amount of capacity.  We’ve also pre-popped the Guided vLab sessions.  These are classroom style labs, with all students taking the same lab and guided by an instructor.  They differ from the “Choose Your Own” style of the main room.  The system is essentially at its peak minus the active student sessions.  Those will come tomorrow!

The vLab Experience is starting to look more like its final state.  Signage is going up all over the place.  Monitors are flickering to life and the EMC World screen saver is gliding all over.  The theme this year is Modernize and there is a ton of marketing with contrast between old and new.  I quite like the shot they have of the Louvre with the iconic (and more modern) pyramid against the backdrop of the older portion of the museum.

Photo by @AnnetteOHerlihy

The registration desk is also coming together.  This is the first stop of a student wanting to attend vLab.  Students can register for a Guided vLab or receive a token to take a lab in the general space.  This token is single use and allows the student to choose a lab.  It prevents a student from starting a bunch of different labs all at once and also allows us to tie together the student to the lab / product they’re interested in.  That data is critical to judge which labs are most popular and for potential followup post show.

Photo by @AnnetteOHerlihy – Hey there @TriciaKreis!

The vertical screens in the shot above are for digital signage.  These will display our lab listings for the Hands on Labs as well as the schedule for Guided vLabs.  Far more cool than standard paper signs!  As you can see, the registration desk now has a fancy skin and is far better looking than it was yesterday.  The plexiglass walls are all covered in a white plastic film.  That will be peeled off later today – the lab space feels much more open once that happens.

Update 1 – 6pm LV time

The lab has come together nicely.  All of the student stations are up and running.  The big screen where we display statistics about labs taken, VMs, popularity etc. is up and running.  Most all of the signage has been completed as well.  The team has moved from room 407 into the NOC that is positioned behind the lab itself.  We’ll live in this NOC for most of the next four days.  Communication between “front of house” and “back of house” will be key because we need to respond quickly to any attendee issues that may pop up.  Walkies will be distributed and we’ll have loads of fun with those!


Original drawing of the space – should look vaguely like this by the end!

Looks awesome – eerily calm this close to the start of the show…

@eoinok77 is making sure the LAN is square
He thinks he did a good job…

The NOC is up and running – where we all assemble while vLab is open

The proctors have arrived and are in the midst of proctor training.  The proctors are the folks who know the lab content inside and out.  They help to staff vLab and are available to answer any questions that the attendees may have.  There is a button in the vLab interface to raise your virtual hand.  When clicked, a lab number and a seat number goes up in the corner of the big screen at the front of the room.  That way the proctors know who has a question and generally what the topic is so that it can be quickly addressed.

@alanmcgra (the guy in charge) is speaking to the lab proctors


Once training completes, the proctors will help us conduct a scale test.  They’ll all start taking labs and thus put load on the system.  Well be back in the NOC watching the various components of the system: vLab application, virtualization layer, compute, and storage.  We’ll be looking for any signs of distress.  If found, we’ll need to get those resolve before tomorrow!

Gotta Have Snacks! – vLab at EMC World, Day 2

Day 2 begins with a run to the local Walmart for provisions.  An army runs on its stomach and so does the vLab team.  In this case the fuel is lots of soda, water, and chips of all kinds.  We all piled into room 407 at the Sands Expo to begin our final prep.  Conversations around the content and guides will run most of the day.  Meanwhile the platform is being prepped for the massive VM churn that the show brings.  Sessions will be drained and refilled to test the ability of vLab’s orchestration to deal with students taking labs throughout the day.  The labs within vLab are pre-popped so that students don’t have to wait more than a few seconds to get a lab.  We avoid waiting for provisioning by having the labs ready well before the customer steps foot into vLab.

The weather has been strange since we got here.  It has rained more in Las Vegas in the last two days than I’ve seen in all my prior trips combined.  They don’t do a great job of street drainage which isn’t surprising in the middle of the desert…


Luckily we survived and once we returned, we dragged the snack and a whole bunch of network switches, cables, and power strips to room 407.  As food is a priority, the table of snacks is set up first!

Fuel!  Introduced the Irish to spicy pork rinds… they were surprisingly open to the idea!

The show floor is starting to come together… first photo was Friday night (courtesy of @quinnwill) and the second was Saturday morning.  Power and network have been run and the drapes are starting to come down.  Carpet will come in along with the plexiglass dividers that form the walls of the vLab space.  It’s unreal the change in this space over time and the logistics companies do this almost every weekend as one trade show flips to another.  Amazing stuff.

A bare canvas

Each student station (200 total) gets a LAN drop and power… everything is run before the carpet goes down

Unfortunately there is a grand total of one power outlet and network port in the entirety of room 407.  Tables are moved and clustered around them.  Power strips are chained together and spidered throughout.  Wired and wireless (affectionate SSID: GoatRodeo) networks appear and we’re good to go.  This will be our home for the next couple days until we move onto the show floor.  I like what we’ve done with the place!

Update 1 – 3pm ish LV time

The lab is starting to come together.  The carpet is completely down and the walls are beginning to be assembled.  Tables have arrive and are being set up in the main room.  Monitors have arrived and so has vLab’s reception desk.  Long way to go but good progress has been made!

Walls being assembled on the ground
Walls standing up with the open doors being used to support them

Massive amounts of materials show up on pallets and in crates.  There are about 20 people milling about vLab’s space building various portions.  The crates on the right contain all the monitors for the lab – ~200 plus spares.  We got some shiny new ones this year… you can probably guess why!

Update 2 – writing Sunday morning – too tired to post last night!


A ton of activity took place in the vLab space on the show floor.  All user stations were deployed – monitors, keyboards, thin clients and everything else.  Deploying 200 such stations isn’t nearly as fun as you think it would be.  Our shiny new monitors and thin client came with about 5 cable combinations from the rental company.  We should have had multiple options to actually get them to work but unfortunately, none of them did.  After many phone conversations, we got line of sight on a solution.  Something as simple as a cable with mini displayport on one end and displayport on the other caused a lot of stress!  Thankfully they would arrive in the morning.

Waiting for stations.  And a Coke.

Registration desk for vLab

Each thin client had a numbered sticker on it… with a corresponding numbered box.  We’ve been asked to get them all back in the correctly numbered box… should be a blast!

It was a long day but we are ahead of where we normally are.  FitBit let me know that it was quite and active day.  Tiring but good thing for sure!

Wheels Down in Vegas! – vLab at EMC World, Day 1

Executing vLab at EMC World has become somewhat of a spring ritual for those of us involved in it every year.  Months of prep culminate in a pilgrimage to Las Vegas.  The journey starts with three days of intense testing, setup, testing, and more testing.  Then the doors open and hundreds of customers flow through over the 4 days of the show.  Each customer selects a lab demonstrating an EMC product – this lab is in reality a live virtual environment designed specifically for EMC World.  They take the lab and when complete (hopefully) walk out a little more informed than when they walked in.  If all goes well, the customers won’t even know most of us are there.  We live in the back of the room behind blocked out plexiglass – a hive of activity designed to keep everything running smoothly out front.  Helping to run vLab at EMCWorld is one of the most stressful weeks of the year.  It’s also one of the best demonstrations of teamwork I’ve been part of.  A small team comes together and works toward success.  No bullshit.  Just get the job done.  And we do.

This beginning of this year’s show will be remembered for terrible travel experiences.  Missed connections, plane re-routes, and hours sitting on the tarmac marred most of our trips to Las Vegas.  I was one of the lucky ones, arriving on time without any issues.  Some landed at different airports to refuel, others were forced to LA and took a bus to Las Vegas.  Connections were missed, sleep was lost.  Most of us were at the Palazzo / Venetian by late Thursday or early Friday and were able to meet for breakfast before getting down to work.  It is an awesome thing to see friends and colleagues that you work with year round, always from a distance.  EMC World brings us together – literally and figuratively – to face the challenge of delivering vLab to EMC’s customers.

After breakfast we headed off to a conference room at an area data center to meet and begin testing.  We go over various operational plans, test plans, communication, failover / failback contingencies.  We QA the lab content, making sure the students will have a clean experience.  We test, test, and test again.

Our fearless leader makes speeches – note the vLab SuperMan T-Shirt!

Tomorrow we will be in a conference room at the Sands Expo.  Building the show floor will commence overnight – luckily others take care of that but we still make sure walls, tables, network, and power are put in the right places for the lab layout.  vLab at EMC World is starting to come together!