By all accounts, EMC World was a tremendous success for vLab!  We saw great buzz around the show and on social media.  Attendees were excited to gain access to EMC’s new products and visited vLab as a result.  vLab executed a total of 3847 labs which was up 17% YoY.  The system itself had 100% uptime, with only a few small hiccups along the way.  The automation of the system was going full bore for most of the week, provisioning 24,902 VMs over the course of the show.  That’s 10.6 VMs provisioned per minute!  If you include the tears downs, that’s nearly 50,000 provisioning / deprovisioning actions in the four days of the show.  Simply tremendous!

As the show wound down, the team had a feeling of relief.  It is a very stressful experience, even when it went as well as it did.  Thursday was busier then normal, with a few stragglers hanging on even as we tore down the room.  We had to do setup in reverse – tearing down all the thin clients and monitors, boxing them all up, squaring with the equipment rental company.  Even when we’re done delivering labs, there’s more work to do!  Once we were done with that, the show setup crew comes in and begins to tear down the walls.  Within the space of a few hours, its as if we never existed.  Amazing the transformation these convention spaces go through!

Once the show is completed, we have some time to relax before we all travel back to our homes for a well earned weekend.  We all went out to dinner together and shared a few more hours together.  This team is really amazing.  Every person is dedicated to the success of the show – and it shows.  We come from separate organizations but those lines disappears for this effort.  It is and amazing experience and is what I believe teams should aspire to be.  We’re all in this together.

Thanks for reading!  If you found this post but haven’t read the earlier ones, links are below.

Day 1 – Wheels down in Vegas
Day 2 – Gotta have snacks!
Day 3 – As ready as we’ll ever be…
Day 4 – Release the hounds!
Day 5 – Keep It Between the Lines!
Day 6 – Technical Overview of vLab

We finished up yesterday with over 1000 labs taken within vLab – 1075 to be exact!  It was a great day by all accounts.  We had a few hiccups here and there but no sustained issues.  The team recovered very well and on the whole, everything ran smoothly.  We expect another busy day today with keynotes and product announcements happening this morning.  We usually get a rush with attendees looking to get a glimpse of new and improved offerings.  We usually have about 60% of the seats full but after keynotes we can get filled to capacity.  When that happens, we have the option to open our Guided vLab classrooms for general use.  If we’re full and there isn’t a class going on, we’ll leverage that option.

Also I met this guy in the Expo:

It’s mid-morning and vLab is completely full!  We have even expanded the main floor into one of the Guided vLab rooms so we have a total of 150+ attendees taking labs.  Amazing stuff!  The backend is humming along just fine.

We are watching the big board in the NOC carefully to catch any issues before they become impactful to the show floor.  There are a ton of components to monitor and many multiples of each… all together they make up vLab so all must be watched.

Day 4 begins rather early… we begin at 6am.  We are doing our final show checks – making sure all the labs have spun up properly and the platform itself is looking healthy.  The registration desk is open at 7am – attendees can come and sign up for Guided vLabs.  There’s a nervous energy in the NOC… anticipation of opening, hopeful that all will go well.  Calm before the storm so to speak.

With customers showing up in a few short hours, all eyes are on the vLab system.  We have our monitoring in place and visible in the NOC.  @eoinok77 did a great job configuring our eye of Sauron.  Second year in a row it has been watching over us… will become a bit of a tradition.

@timmywu and @eoinok77 are paying attention

It’s early and some of us didn’t stop for breakfast on the way.  Dmitri grabbed a breakfast of champions from our snap table.  A Starbucks drink and Oreos will get your day going!

As customers begin to finally file in, activity in the back increases.  Slightly before open, we ran into an issue with PSOD on a couple hosts.  The labs running on those hosts died but we were able to clean up the backend with minimal impact to the customers on the floor.  Issues like this will crop up from time to time (chaos monkey happens).  Our goal is to minimize the impact when they do.  Constant communication is critical so the NOC and the front of house can stay in sync.

Its about an hour after open and we’re already over half full.

You can see in the front of the lab, the big screen is displaying the number of labs that have been taken, number of VMs created and destroyed, lab popularity, and other stats.  It will climb during the week… hopefully setting some records along the way.

Update 1 … around 4pm LV time

The day has been running fairly smoothly and the lab has been pretty full since we’ve been open.  The main room of vLab has 130 stations and is open for any lab.  We have also run a handful of Guided vLabs – each with about 20 students, all taking the same lab.  vLab got a shout out in the Service360 (press release) keynote when the speaker mentioned that customers should head to vLab and take a session on MyService360.  Great publicity for us!  A lot of the content in the vLab Experience is aligned with product launches.  In addition to Service360, for example, we also have a couple labs on Unity (press release), the new offering from EMC’s midrange.  As of now, a Unity lab is the most popular in vLab – almost 100 sessions of “Unity – Simplified Storage Management and Administration” have been taken by customers.  On the whole, the day has been very good so far – we have delivered close to 900 labs consisting of nearly 5000 VMs.  And we have 3+ hours left!

We also got some good publicity on @emctv as they came and interviewed @vlabkreis.  She talked all about the vLab Experience and the opportunity it provides for the attendees of EMC World.  Education on the entire catalog of EMC’s products is readily available!

Another member of our team, @vfspb was watching live on the broadcast stream!

Day 3 is our last day of prep before customers arrive.  We have to button up the vLab area on the show floor and make sure that all the content and the platform is good to go.  The cables we needed have arrived so getting the thin clients functional is the focus.  Overnight, the platform team drained all vApps out of the system and have pre-popped all labs to pre-set amounts.  We have more pre-pops for labs we expect to be popular and less for those we don’t.  The whole point is to prevent students from having to wait for a lab to provision without consuming an inordinate amount of capacity.  We’ve also pre-popped the Guided vLab sessions.  These are classroom style labs, with all students taking the same lab and guided by an instructor.  They differ from the “Choose Your Own” style of the main room.  The system is essentially at its peak minus the active student sessions.  Those will come tomorrow!

The vLab Experience is starting to look more like its final state.  Signage is going up all over the place.  Monitors are flickering to life and the EMC World screen saver is gliding all over.  The theme this year is Modernize and there is a ton of marketing with contrast between old and new.  I quite like the shot they have of the Louvre with the iconic (and more modern) pyramid against the backdrop of the older portion of the museum.

Photo by @AnnetteOHerlihy

The registration desk is also coming together.  This is the first stop of a student wanting to attend vLab.  Students can register for a Guided vLab or receive a token to take a lab in the general space.  This token is single use and allows the student to choose a lab.  It prevents a student from starting a bunch of different labs all at once and also allows us to tie together the student to the lab / product they’re interested in.  That data is critical to judge which labs are most popular and for potential followup post show.

Photo by @AnnetteOHerlihy – Hey there @TriciaKreis!

The vertical screens in the shot above are for digital signage.  These will display our lab listings for the Hands on Labs as well as the schedule for Guided vLabs.  Far more cool than standard paper signs!  As you can see, the registration desk now has a fancy skin and is far better looking than it was yesterday.  The plexiglass walls are all covered in a white plastic film.  That will be peeled off later today – the lab space feels much more open once that happens.

Update 1 – 6pm LV time

The lab has come together nicely.  All of the student stations are up and running.  The big screen where we display statistics about labs taken, VMs, popularity etc. is up and running.  Most all of the signage has been completed as well.  The team has moved from room 407 into the NOC that is positioned behind the lab itself.  We’ll live in this NOC for most of the next four days.  Communication between “front of house” and “back of house” will be key because we need to respond quickly to any attendee issues that may pop up.  Walkies will be distributed and we’ll have loads of fun with those!

Original drawing of the space – should look vaguely like this by the end!

@eoinok77 is making sure the LAN is square

The proctors have arrived and are in the midst of proctor training.  The proctors are the folks who know the lab content inside and out.  They help to staff vLab and are available to answer any questions that the attendees may have.  There is a button in the vLab interface to raise your virtual hand.  When clicked, a lab number and a seat number goes up in the corner of the big screen at the front of the room.  That way the proctors know who has a question and generally what the topic is so that it can be quickly addressed.

Once training completes, the proctors will help us conduct a scale test.  They’ll all start taking labs and thus put load on the system.  Well be back in the NOC watching the various components of the system: vLab application, virtualization layer, compute, and storage.  We’ll be looking for any signs of distress.  If found, we’ll need to get those resolve before tomorrow!

Day 2 begins with a run to the local Walmart for provisions.  An army runs on its stomach and so does the vLab team.  In this case the fuel is lots of soda, water, and chips of all kinds.  We all piled into room 407 at the Sands Expo to begin our final prep.  Conversations around the content and guides will run most of the day.  Meanwhile the platform is being prepped for the massive VM churn that the show brings.  Sessions will be drained and refilled to test the ability of vLab’s orchestration to deal with students taking labs throughout the day.  The labs within vLab are pre-popped so that students don’t have to wait more than a few seconds to get a lab.  We avoid waiting for provisioning by having the labs ready well before the customer steps foot into vLab.

The weather has been strange since we got here.  It has rained more in Las Vegas in the last two days than I’ve seen in all my prior trips combined.  They don’t do a great job of street drainage which isn’t surprising in the middle of the desert…

Fuel!  Introduced the Irish to spicy pork rinds… they were surprisingly open to the idea!

The show floor is starting to come together… first photo was Friday night (courtesy of @quinnwill) and the second was Saturday morning.  Power and network have been run and the drapes are starting to come down.  Carpet will come in along with the plexiglass dividers that form the walls of the vLab space.  It’s unreal the change in this space over time and the logistics companies do this almost every weekend as one trade show flips to another.  Amazing stuff.

A bare canvas

Each student station (200 total) gets a LAN drop and power… everything is run before the carpet goes down

Unfortunately there is a grand total of one power outlet and network port in the entirety of room 407.  Tables are moved and clustered around them.  Power strips are chained together and spidered throughout.  Wired and wireless (affectionate SSID: GoatRodeo) networks appear and we’re good to go.  This will be our home for the next couple days until we move onto the show floor.  I like what we’ve done with the place!

Update 1 – 3pm ish LV time

The lab is starting to come together.  The carpet is completely down and the walls are beginning to be assembled.  Tables have arrive and are being set up in the main room.  Monitors have arrived and so has vLab’s reception desk.  Long way to go but good progress has been made!

Massive amounts of materials show up on pallets and in crates.  There are about 20 people milling about vLab’s space building various portions.  The crates on the right contain all the monitors for the lab – ~200 plus spares.  We got some shiny new ones this year… you can probably guess why!

Update 2 – writing Sunday morning – too tired to post last night!

A ton of activity took place in the vLab space on the show floor.  All user stations were deployed – monitors, keyboards, thin clients and everything else.  Deploying 200 such stations isn’t nearly as fun as you think it would be.  Our shiny new monitors and thin client came with about 5 cable combinations from the rental company.  We should have had multiple options to actually get them to work but unfortunately, none of them did.  After many phone conversations, we got line of sight on a solution.  Something as simple as a cable with mini displayport on one end and displayport on the other caused a lot of stress!  Thankfully they would arrive in the morning.

Waiting for stations.  And a Coke.

Registration desk for vLab

Each thin client had a numbered sticker on it… with a corresponding numbered box.  We’ve been asked to get them all back in the correctly numbered box… should be a blast!

It was a long day but we are ahead of where we normally are.  FitBit let me know that it was quite and active day.  Tiring but good thing for sure!

Executing vLab at EMC World has become somewhat of a spring ritual for those of us involved in it every year.  Months of prep culminate in a pilgrimage to Las Vegas.  The journey starts with three days of intense testing, setup, testing, and more testing.  Then the doors open and hundreds of customers flow through over the 4 days of the show.  Each customer selects a lab demonstrating an EMC product – this lab is in reality a live virtual environment designed specifically for EMC World.  They take the lab and when complete (hopefully) walk out a little more informed than when they walked in.  If all goes well, the customers won’t even know most of us are there.  We live in the back of the room behind blocked out plexiglass – a hive of activity designed to keep everything running smoothly out front.  Helping to run vLab at EMCWorld is one of the most stressful weeks of the year.  It’s also one of the best demonstrations of teamwork I’ve been part of.  A small team comes together and works toward success.  No bullshit.  Just get the job done.  And we do.

This beginning of this year’s show will be remembered for terrible travel experiences.  Missed connections, plane re-routes, and hours sitting on the tarmac marred most of our trips to Las Vegas.  I was one of the lucky ones, arriving on time without any issues.  Some landed at different airports to refuel, others were forced to LA and took a bus to Las Vegas.  Connections were missed, sleep was lost.  Most of us were at the Palazzo / Venetian by late Thursday or early Friday and were able to meet for breakfast before getting down to work.  It is an awesome thing to see friends and colleagues that you work with year round, always from a distance.  EMC World brings us together – literally and figuratively – to face the challenge of delivering vLab to EMC’s customers.

After breakfast we headed off to a conference room at an area data center to meet and begin testing.  We go over various operational plans, test plans, communication, failover / failback contingencies.  We QA the lab content, making sure the students will have a clean experience.  We test, test, and test again.

Our fearless leader makes speeches – note the vLab SuperMan T-Shirt!

Tomorrow we will be in a conference room at the Sands Expo.  Building the show floor will commence overnight – luckily others take care of that but we still make sure walls, tables, network, and power are put in the right places for the lab layout.  vLab at EMC World is starting to come together!