Chris and Joe Tucci

Keep It Between the Lines! – vLab at EMC World, Day 5

We finished up yesterday with over 1000 labs taken within vLab – 1075 to be exact!  It was a great day by all accounts.  We had a few hiccups here and there but no sustained issues.  The team recovered very well and on the whole, everything ran smoothly.  We expect another busy day today with keynotes and product announcements happening this morning.  We usually get a rush with attendees looking to get a glimpse of new and improved offerings.  We usually have about 60% of the seats full but after keynotes we can get filled to capacity.  When that happens, we have the option to open our Guided vLab classrooms for general use.  If we’re full and there isn’t a class going on, we’ll leverage that option.

Also I met this guy in the Expo:

Chris and Joe Tucci

It’s mid-morning and vLab is completely full!  We have even expanded the main floor into one of the Guided vLab rooms so we have a total of 150+ attendees taking labs.  Amazing stuff!  The backend is humming along just fine.

We are watching the big board in the NOC carefully to catch any issues before they become impactful to the show floor.  There are a ton of components to monitor and many multiples of each… all together they make up vLab so all must be watched.

Release The Hounds! – vLab at EMC World, Day 4

Day 4 begins rather early… we begin at 6am.  We are doing our final show checks – making sure all the labs have spun up properly and the platform itself is looking healthy.  The registration desk is open at 7am – attendees can come and sign up for Guided vLabs.  There’s a nervous energy in the NOC… anticipation of opening, hopeful that all will go well.  Calm before the storm so to speak.

With customers showing up in a few short hours, all eyes are on the vLab system.  We have our monitoring in place and visible in the NOC.  @eoinok77 did a great job configuring our eye of Sauron.  Second year in a row it has been watching over us… will become a bit of a tradition.

@timmywu and @eoinok77 are paying attention

It’s early and some of us didn’t stop for breakfast on the way.  Dmitri grabbed a breakfast of champions from our snap table.  A Starbucks drink and Oreos will get your day going!

As customers begin to finally file in, activity in the back increases.  Slightly before open, we ran into an issue with PSOD on a couple hosts.  The labs running on those hosts died but we were able to clean up the backend with minimal impact to the customers on the floor.  Issues like this will crop up from time to time (chaos monkey happens).  Our goal is to minimize the impact when they do.  Constant communication is critical so the NOC and the front of house can stay in sync.

Its about an hour after open and we’re already over half full.

You can see in the front of the lab, the big screen is displaying the number of labs that have been taken, number of VMs created and destroyed, lab popularity, and other stats.  It will climb during the week… hopefully setting some records along the way.

Update 1 … around 4pm LV time

The day has been running fairly smoothly and the lab has been pretty full since we’ve been open.  The main room of vLab has 130 stations and is open for any lab.  We have also run a handful of Guided vLabs – each with about 20 students, all taking the same lab.  vLab got a shout out in the Service360 (press release) keynote when the speaker mentioned that customers should head to vLab and take a session on MyService360.  Great publicity for us!  A lot of the content in the vLab Experience is aligned with product launches.  In addition to Service360, for example, we also have a couple labs on Unity (press release), the new offering from EMC’s midrange.  As of now, a Unity lab is the most popular in vLab – almost 100 sessions of “Unity – Simplified Storage Management and Administration” have been taken by customers.  On the whole, the day has been very good so far – we have delivered close to 900 labs consisting of nearly 5000 VMs.  And we have 3+ hours left!

We also got some good publicity on @emctv as they came and interviewed @vlabkreis.  She talked all about the vLab Experience and the opportunity it provides for the attendees of EMC World.  Education on the entire catalog of EMC’s products is readily available!

Another member of our team, @vfspb was watching live on the broadcast stream!

Ready As We’ll Ever Be! – vLab at EMC World, Day 3

Day 3 is our last day of prep before customers arrive.  We have to button up the vLab area on the show floor and make sure that all the content and the platform is good to go.  The cables we needed have arrived so getting the thin clients functional is the focus.  Overnight, the platform team drained all vApps out of the system and have pre-popped all labs to pre-set amounts.  We have more pre-pops for labs we expect to be popular and less for those we don’t.  The whole point is to prevent students from having to wait for a lab to provision without consuming an inordinate amount of capacity.  We’ve also pre-popped the Guided vLab sessions.  These are classroom style labs, with all students taking the same lab and guided by an instructor.  They differ from the “Choose Your Own” style of the main room.  The system is essentially at its peak minus the active student sessions.  Those will come tomorrow!

The vLab Experience is starting to look more like its final state.  Signage is going up all over the place.  Monitors are flickering to life and the EMC World screen saver is gliding all over.  The theme this year is Modernize and there is a ton of marketing with contrast between old and new.  I quite like the shot they have of the Louvre with the iconic (and more modern) pyramid against the backdrop of the older portion of the museum.

Photo by @AnnetteOHerlihy

The registration desk is also coming together.  This is the first stop of a student wanting to attend vLab.  Students can register for a Guided vLab or receive a token to take a lab in the general space.  This token is single use and allows the student to choose a lab.  It prevents a student from starting a bunch of different labs all at once and also allows us to tie together the student to the lab / product they’re interested in.  That data is critical to judge which labs are most popular and for potential followup post show.

Photo by @AnnetteOHerlihy – Hey there @TriciaKreis!

The vertical screens in the shot above are for digital signage.  These will display our lab listings for the Hands on Labs as well as the schedule for Guided vLabs.  Far more cool than standard paper signs!  As you can see, the registration desk now has a fancy skin and is far better looking than it was yesterday.  The plexiglass walls are all covered in a white plastic film.  That will be peeled off later today – the lab space feels much more open once that happens.

Update 1 – 6pm LV time

The lab has come together nicely.  All of the student stations are up and running.  The big screen where we display statistics about labs taken, VMs, popularity etc. is up and running.  Most all of the signage has been completed as well.  The team has moved from room 407 into the NOC that is positioned behind the lab itself.  We’ll live in this NOC for most of the next four days.  Communication between “front of house” and “back of house” will be key because we need to respond quickly to any attendee issues that may pop up.  Walkies will be distributed and we’ll have loads of fun with those!


Original drawing of the space – should look vaguely like this by the end!

Looks awesome – eerily calm this close to the start of the show…

@eoinok77 is making sure the LAN is square
He thinks he did a good job…

The NOC is up and running – where we all assemble while vLab is open

The proctors have arrived and are in the midst of proctor training.  The proctors are the folks who know the lab content inside and out.  They help to staff vLab and are available to answer any questions that the attendees may have.  There is a button in the vLab interface to raise your virtual hand.  When clicked, a lab number and a seat number goes up in the corner of the big screen at the front of the room.  That way the proctors know who has a question and generally what the topic is so that it can be quickly addressed.

@alanmcgra (the guy in charge) is speaking to the lab proctors


Once training completes, the proctors will help us conduct a scale test.  They’ll all start taking labs and thus put load on the system.  Well be back in the NOC watching the various components of the system: vLab application, virtualization layer, compute, and storage.  We’ll be looking for any signs of distress.  If found, we’ll need to get those resolve before tomorrow!

Gotta Have Snacks! – vLab at EMC World, Day 2

Day 2 begins with a run to the local Walmart for provisions.  An army runs on its stomach and so does the vLab team.  In this case the fuel is lots of soda, water, and chips of all kinds.  We all piled into room 407 at the Sands Expo to begin our final prep.  Conversations around the content and guides will run most of the day.  Meanwhile the platform is being prepped for the massive VM churn that the show brings.  Sessions will be drained and refilled to test the ability of vLab’s orchestration to deal with students taking labs throughout the day.  The labs within vLab are pre-popped so that students don’t have to wait more than a few seconds to get a lab.  We avoid waiting for provisioning by having the labs ready well before the customer steps foot into vLab.

The weather has been strange since we got here.  It has rained more in Las Vegas in the last two days than I’ve seen in all my prior trips combined.  They don’t do a great job of street drainage which isn’t surprising in the middle of the desert…


Luckily we survived and once we returned, we dragged the snack and a whole bunch of network switches, cables, and power strips to room 407.  As food is a priority, the table of snacks is set up first!

Fuel!  Introduced the Irish to spicy pork rinds… they were surprisingly open to the idea!

The show floor is starting to come together… first photo was Friday night (courtesy of @quinnwill) and the second was Saturday morning.  Power and network have been run and the drapes are starting to come down.  Carpet will come in along with the plexiglass dividers that form the walls of the vLab space.  It’s unreal the change in this space over time and the logistics companies do this almost every weekend as one trade show flips to another.  Amazing stuff.

A bare canvas

Each student station (200 total) gets a LAN drop and power… everything is run before the carpet goes down

Unfortunately there is a grand total of one power outlet and network port in the entirety of room 407.  Tables are moved and clustered around them.  Power strips are chained together and spidered throughout.  Wired and wireless (affectionate SSID: GoatRodeo) networks appear and we’re good to go.  This will be our home for the next couple days until we move onto the show floor.  I like what we’ve done with the place!

Update 1 – 3pm ish LV time

The lab is starting to come together.  The carpet is completely down and the walls are beginning to be assembled.  Tables have arrive and are being set up in the main room.  Monitors have arrived and so has vLab’s reception desk.  Long way to go but good progress has been made!

Walls being assembled on the ground
Walls standing up with the open doors being used to support them

Massive amounts of materials show up on pallets and in crates.  There are about 20 people milling about vLab’s space building various portions.  The crates on the right contain all the monitors for the lab – ~200 plus spares.  We got some shiny new ones this year… you can probably guess why!

Update 2 – writing Sunday morning – too tired to post last night!


A ton of activity took place in the vLab space on the show floor.  All user stations were deployed – monitors, keyboards, thin clients and everything else.  Deploying 200 such stations isn’t nearly as fun as you think it would be.  Our shiny new monitors and thin client came with about 5 cable combinations from the rental company.  We should have had multiple options to actually get them to work but unfortunately, none of them did.  After many phone conversations, we got line of sight on a solution.  Something as simple as a cable with mini displayport on one end and displayport on the other caused a lot of stress!  Thankfully they would arrive in the morning.

Waiting for stations.  And a Coke.

Registration desk for vLab

Each thin client had a numbered sticker on it… with a corresponding numbered box.  We’ve been asked to get them all back in the correctly numbered box… should be a blast!

It was a long day but we are ahead of where we normally are.  FitBit let me know that it was quite and active day.  Tiring but good thing for sure!

Wheels Down in Vegas! – vLab at EMC World, Day 1

Executing vLab at EMC World has become somewhat of a spring ritual for those of us involved in it every year.  Months of prep culminate in a pilgrimage to Las Vegas.  The journey starts with three days of intense testing, setup, testing, and more testing.  Then the doors open and hundreds of customers flow through over the 4 days of the show.  Each customer selects a lab demonstrating an EMC product – this lab is in reality a live virtual environment designed specifically for EMC World.  They take the lab and when complete (hopefully) walk out a little more informed than when they walked in.  If all goes well, the customers won’t even know most of us are there.  We live in the back of the room behind blocked out plexiglass – a hive of activity designed to keep everything running smoothly out front.  Helping to run vLab at EMCWorld is one of the most stressful weeks of the year.  It’s also one of the best demonstrations of teamwork I’ve been part of.  A small team comes together and works toward success.  No bullshit.  Just get the job done.  And we do.

This beginning of this year’s show will be remembered for terrible travel experiences.  Missed connections, plane re-routes, and hours sitting on the tarmac marred most of our trips to Las Vegas.  I was one of the lucky ones, arriving on time without any issues.  Some landed at different airports to refuel, others were forced to LA and took a bus to Las Vegas.  Connections were missed, sleep was lost.  Most of us were at the Palazzo / Venetian by late Thursday or early Friday and were able to meet for breakfast before getting down to work.  It is an awesome thing to see friends and colleagues that you work with year round, always from a distance.  EMC World brings us together – literally and figuratively – to face the challenge of delivering vLab to EMC’s customers.

After breakfast we headed off to a conference room at an area data center to meet and begin testing.  We go over various operational plans, test plans, communication, failover / failback contingencies.  We QA the lab content, making sure the students will have a clean experience.  We test, test, and test again.

Our fearless leader makes speeches – note the vLab SuperMan T-Shirt!

Tomorrow we will be in a conference room at the Sands Expo.  Building the show floor will commence overnight – luckily others take care of that but we still make sure walls, tables, network, and power are put in the right places for the lab layout.  vLab at EMC World is starting to come together!

I’m Hiring!

I’m hiring a Manager to lead a team of very talented cloud / virtualization engineers!  The team provides cloud-based solutions internally to EMC Engineering, Pre-Sales, and other constituents. Please take a look and send me a note if you’d like to discuss the position in more detail.   
Job Title: Manager, Cloud Engineering – Platform Engineering
Job Location: US – Massachusetts – Hopkinton

The Manager, Cloud Engineering – Platform Engineering position is accountable for managing a distributed team that provides virtualization and orchestration engineering for Common Engineering Cloud, our internal cloud service for Engineering, Sales, and other functions. This includes working with leading-edge technologies to proactively maintain a world-class cloud environment that meets the needs of the EMC. This position will participate in strategy formation but also in the tactical activities of the team as well, so strong technical qualifications are necessary. 

This position reports to the Senior Manager of the Cloud Engineering Team. 
Job Responsibilities:
• Supervise a team of eight engineers who are distributed globally
• Align group with overall strategy created in partnership with the rest of the Cloud Engineering team – in particular, Software Engineering
• Maintain relationship with the Cloud Operations organization, ensuring successful transition of support for new services / technologies
• Measurement and reporting of service demand / consumption – leading to proper capacity planning
• Maintain and continually improve Cloud Engineering’s services
• Provide cloud expertise, best practices, and assistance to content developers and Cloud Operations
• Fluent English language skills (written and verbal) 
• University Degree with a major in Computer Science or equivalent
• 2+ years in a management position or equivalent experience
• 6+ years, detailed experience in VMware virtualization deployments
• Familiarity with DevOps and experience in a DevOps culture a big plus 
• VCP certification required
• In depth knowledge of vSphere, vRA, vCD, and NSX required
• In depth knowledge of storage technologies (EMC a plus)
• In depth knowledge of networking (Cisco primarily)
• Working knowledge of ITIL and its implementation in the production environment (certification a plus)
• Working knowledge of SQL Server / database technologies
• Working knowledge Windows and Linux
• Working knowledge of Docker a plus but not required
• Working knowledge of 3rd party clouds – AWS, Azure, etc.
• Ability to work in a global team environment with participatory decision-making
Job ID: 163233BR

Click to Apply

Presenting at UConn – Solving Business Problems with the Cloud

Last night, I had the wonderful opportunity to present to a group of engineering students at the University of Connecticut.  It was great to be back on a college campus after quite a number of years away.  The energy and curiosity in the room was wonderful to experience.  I was excited and more than a little bit nervous about speaking.  No matter how many times I present to groups, I have pre-game jitters.  Regardless, it was a topic I knew well and once we got rolling, the discussion was engaging and free flowing.  The topic for discussion was “Solving Business Problems with the Cloud”.  I framed the discussion in terms of the needs of the business rather than the technical intricacies of the cloud itself (I’m hardly qualified to do that anyway).  I did so by walking through use cases from my personal experience.  My hope was to provide my audience with some real-world insight into the technical concepts they were learning in class.

The first use case I went through was vLab, cloud success story from early in my career at EMC.  I worked within the team that created and evangelized a cloud platform for virtual sales demos.  We took the corporation from sales engineers developing demos on laptops to a centralized platform that provides fully functioning, virtualized content to the tune of almost 100,000 demos per year.  The content is co-developed with the product business units and aligns with sales campaigns.  vLab is a perfect example of the value of an internal, private cloud.

The second use case I described was the application of cloud based services to enable the engineering teams that deliver EMC’s products.  I introduced the concept of a Product Delivery Platform (PDP) which is essentially a combination of tools and resources designed to be a workspace for engineers across EMC.  It is certainly a challenge to standardize and extend services such as source code control, build, and test / QA to engineers across diverse teams.  Having acquired over 75 companies in the last decade, you can imagine the diverse processes and tools that exist within EMC.  We’re well on our way but there is a lot of work left to do.

If you’re interested, here’s the deck I used:

Solving Business Problems using the Cloud