Sam Guckenheimer - DevOps Enterprise Summit 2018

Bob chats with Microsoft Azure DevOps Product Owner and author of
Agile Software Engineering with Visual Studio, Sam
Guckenheimer, at the DevOps Enterprise Summit 2018.


Connect with Sam and Bob on Twitter.


 


Transcript


Sam Guckenheimer ‑ DevOps Enterprise Summit 2018


Bob Payne:  "The Agile Toolkit."


[music]


Bob:  Hi, I'm your host Bob Payne. I'm here
at the DevOps Enterprise Summit 2018 with Sam Guckenheimer.
Welcome, Sam.


Sam Guckenheimer:  Thank you, Bob. It's
great to be here.


Bob:  It's the first time we're really
chatting. We chatted a tiny bit last night. My colleague Sanjiv
Augustine said you were instrumental in hosting The Agile
leadership network when it formed and came up with the
declaration of Interdependence. How did that end up coming about?


Sam:  Well, that was no what 14 years ago
or something like that.


[laughter]


Sam:  What we saw was at that time that
this was of course way pre‑DevOps. The Agile community had
fractured into many groups saying "More agile than thou." That
seemed stupid.


Bob:  That fracturing has continued and
remains as stupid today or... [laughs]


Sam:  Yes. Unfortunately, the fracturing
has continued and it hasn't gotten less stupid. That was the
reason for trying to get the interdependence declaration together
to get these leading lights from what was then the Agile
community working together.


In the meantime, the pure Agile has largely been eclipsed by
DevOps. As you see something like this DevOps Enterprise Summit
going on its fifth year roughly doubling every year in scale. I'm
here now. Still there.


[laughter]


Bob:  There are a number of things that I
found at this conference that I haven't been able to make a ton
of sessions because we have a booth. I've found that I haven't
really learned, maybe this is my own fault, anything at the Agile
conferences for probably about 10 years. It wasn't any
substantially interesting information.


Sam:  That's correct. I last keynoted at
the Agile conference in 2014. That's probably the last time I've
been there. It got kind of stale. The energy in innovation, in
practice I think has really shifted to DevOps. That's come about,
because the DevOps' definition of Dunn is not potentially
shippable and promotes...


[crosstalk]


Bob:  It's captured a value, enlargement
value.


Sam:  It's live in production with
Telemetry that is demonstrating the value delivered.


Going from a world where you were effectively stopping at an
intermediate activity that didn't reach the customer or end‑user
to go to one where you have to reach the end‑customer and you
have to measure the value delivered, is much, much more powerful
for all the stakeholders, for the business, for the people
involved.


It's much more satisfying. You disintermediate the development to
customer relationship. You think of things as one engineering
discipline, not as silos post the Scrums, so to speak.


Bob:  Certainly there were a number of
great Agile teams and organizations that fully believed that Dunn
meant in the hands of customers and delivering whatever goal,
that...


[crosstalk]


Sam:  I do not mean to bash anyone. I
certainly think there great Agile teams. A lot of what we do
today has its roots in extreme programming, but things like XP at
the time, had this notion of, for example, pair programming.


We have largely, as a community, moved to the notion of a pull
request as a virtual pair programming. We have moved from the
idea of onsite customer to measuring customer impact, which isn't
to say onsite customer is a bad idea, it's a great idea, it's a
rarely achievable one.


All of these seeds that were planted back then in the late '80s
by the early Agilists were important seeds. The garden where I
think they're really bearing fruit now is in this DevOps
community.


Bob:  The other thing that I think is
probably the next wave that we will see in organizations that are
not already there, certainly, many organizations have already
integrated business into this flow. Without that DevOps is
necessary but not sufficient to actually change the outcomes that
businesses are seeing.


That's the next frontier for those companies that we're not sort
of born in the world of IT as the fundamental driver of business
outcomes.


Sam:  That's correct. DevOps is the flip
side of the coin from digital transformation. Digital
transformation is the business term for taking your business
model and turning it into one that can improve continuously in an
Internet‑powered age. DevOps is the shorthand for the technical
practices that enable that.


Bob:  I see way too many organizations
mistaking a DevOps transformation for digital transformation.
They're fundamentally doing the DevOps practices, but they're not
backing up into the initial value proposition to begin with. That
will sort itself out.


Sam:  This is a common thing of confusing
means and ends. The ends are things around growing the business
customer, acquisition customer, engagement customer, employee
delight, all of these measures of happiness and success.


The practices are ways of getting there. The goal is to focus
however on those end results. The clear sign of dysfunction is
when you see people measuring the inputs, not the outputs.


Bob:  If Deming or [inaudible 7:33] came
back and saw that Toyota was doing the same practices it was
doing 75 years ago they would drop dead after having just come
back to life. [laughs] In real systems the practices and the
processes are never the ends. They are all in service of
maximizing flow...


[crosstalk]


Sam:  Exactly. If you think about the
evolution, the practices today are different because the
constraints are different. One of the overriding constraints was
for example infrastructure availability. You get all of the stuff
around how to manage and schedule the infra.


Today with the public cloud that constraint is gone. It's a
classic example of, in Eliyahu Goldratt terms, elevating the
constraint or removing the bottleneck. Then you see the
constraint shifting. As you're adopting these practices what
happens is you have a continual shift of the constraint, and you
have the next one to attack the next bowling pin to knock down.


[crosstalk]


Sam:  Right. What DevOps says has basically
taught us as well. You can remove infrastructure a constraint by
using the cloud. You can focus on the value delivered to the
customer and measure it so you can have both qualitative and
quantitative view of that.


You can take the quality game and shift it left and right so that
quality does not become this big testing bottleneck in the
middle. It can become part of the pull request flow. It can
happen before code merges.


Then you can in production gradually expose value to more and
more of users so that the blast radius is something that's
flexible, so you don't have the constraint of saying, "I need to
master my MTBF in order to release."


You can say, "I need to maximize my ability to recover and may
have the shortest time to recover, so that by controlling the
blast radius and being able to recover quickly I can
experimentally by increasing the rate of experimentation I can
deliver and measure value delivered on a cycle that was never
possible in the old days."


It wasn't possible before we had the Internet, it wasn't possible
before it hit the public cloud, it wasn't possible before we had
these practices of high‑quality, highly‑rugged automation that we
do today.


Bob:  Yeah it has been a sea change since I
did Fortran on punch cards [laughs] .


Sam:  There have been many sea changes yes.
Mike Pearson gave a great talk yesterday, borrowing from Carlota
Perez on the structure of industrial revolutions, and postulates
that we're at the point of disruption from the period of adoption
to the period of dispersion.


That would account for a lot of the changes that we're seeing,
and it would account for a lot of the anxiety that you see among
people who are saying, "How do I learn fast enough? How do I
catch up fast enough? How do I get ahead?"


At the same time, what you see very clearly reflected in company
success, company's market gap, and company's ability to innovate
and pivot, is that the ones who have mastered the
go‑fast‑without‑breaking‑things‑and‑adjust‑course‑as‑you‑go, are
the ones that are winning in pretty much every sector.


Bob:  I love Mark Schwartz's analogy of the
battle of the Russians with Napoleon, and the speed of
decision‑making being fundamentally out of sync with the reality
of the battle.


Sam:  Exactly, that was also true on Omaha
beach in Normandy, that was true in Vietnam, that's been true in
pretty much every military conflict, that the degree of autonomy
and speed of innovation has determined the outcome in the end,
and people who are great at enabling the next war instead of
fighting the last ones, are the victors.


The latest example decide or...I don't know if that's politically
correct to go there, but you see it now in...


Bob:  [laughs] That have been substantially
politically correct on this podcast [laughs] .


Sam:  You see this in cyber. The Russian
budget for cyber is less than the cost of an F‑35.


Bob:  No one could argue that the F‑35 is
more costly than it needs to be but it's... [laughs] .


[crosstalk]


Sam:  Who cares? The point is, they're not
trying to win the manned aerial dogfight. They are extending the
notion of total war to a new battlefield and they've been very
successful, but finding the place where there are no defenses and
where it's possible to innovate quickly and it's proven to work.


You could also argue that as David Sanger does in "The Perfect
Weapon", that the US started this cyber‑war arms race. In any
event, we've not follow through on the consequences of what we
started.


The military analogies, they turn some people off, but they have
their value. We are, and the rest of society also, in a place
where we need to be winning the future, not the past.


Bob:  It's actually one of the analogies I
quite often use when I'm talking to people that are OK with the
military analogies.


The OODA loop, the Boyd loop of observe‑orient‑decide‑act. The
team that can turn that loop the fastest, whether it's Amazon,
Netflix, or a manned‑aerial dogfight, or a cyber‑attack, is going
to win.


Sam:  Exactly. In our world, the OODA loop
results in some kind of software or service delivered. One of the
things we know from measuring it is that about a third of the
time, we get the results we'd want, a third of the time, we get
opposite result from what we hypothesized, and about a third of
the time, it makes no difference.


The implication of that is that you want to be able as quickly as
possible, to double down on the successful third and fail fast or
pivot away from the other two thirds, which means that you need
to make the OODA loop as short as possible, which is what Boyd
talked about in his idea of aircraft design and aerial battle.


That's exactly true in how we develop and that means not just
using small batches which Agile taught us. That means not just
breaking down the silos, but it means really focusing on time to
remediate and focusing on quality to the left so that you have
clean delivery and you have the mechanism in production to
control exposure and to go faster and wider as you need to.


Bob:  You mentioned the one‑third,
one‑third, one‑third, I know that was a study that came out in
Microsoft. Actually...


[crosstalk]


Sam:  Ronick O' Harvey was behind that.
Ronny is now a technical fellow, he wasn't back then. He
basically took a very large sample of "improvements" that were
delivered.


Let's measure, are these really improving, what we wanted? The
result was a third of the time, in other words, I've confirmed
the others' change is bad, unless is great. That was quantitative
demonstration of that. I don't know if he published that before
he did a stand for PhD or after, but it was a famous study and it
holds up.


Bob:  I also very much like this idea of
very small batches, because without the small batches, it's hard
to get attribution of what improved the customer experience and
what was neutral or negative, because you're conflating way too
many changes if the batch is large.


Sam:  That's why the pull request flows
becomes successful, because you can make the pull request a batch
that is a few lines of change, it's possible to have a human‑code
review on it, and it's possible to have extensive automation on
it.


Again, an example of a practice that wouldn't have been possible
pre‑cloud is when we do pull requests, we run the build‑in
automation on them with typically 80‑some thousand tests before
asking for the human‑code review. Human eyes are only focused on
those things that automation has said looked good already.


As opposed to the way things were done, pre‑cloud in the XP pair
programming model, where human eyes were first defense. That was
very appropriate given the constraints at the time. The
constraints of today are different.


Bob:  That was certainly one aspect of
pairing. The other is just as the design discipline getting the
collaborative design quite often yields better results, but...


[crosstalk]


Sam:  I totally think that people should
collaborate on design. I'm totally for that. I'm not trying to..


Bob:  I totally get the point about the
quality. Is automation...we want lazy engineers [laughs] . We
want engineers focusing on creative thought, rather than
repetitive action.


Sam:  Exactly. Another example of that
that's possible these days, is you want a very high reuse, an
open source. If you can solve a problem with 30 lines of code and
reuse thousands, that's much better than creating 3000 lines of
codes that need to be maintained.


In effect, we want to reward people for writing less code, which
again turns on it's head, one of those classic input matrix and
myths of, "Well, how much code did you write? How busy were you?
How many hours did you put in?" As opposed to, "What result did
you achieve?"


Bob:  What are some DevOps practices that
have really changed Microsoft fundamentally? I know you've got a
couple of talks related to that here at the conference.


Sam:  I bucket our lessons learned, usually
in five groups. One is how we focus on value delivered to the
customer, both quantitatively and qualitatively, and let that
drive the way we think about what we're delivering and how we
measure that.


Two is how we apply production‑first mindset. Our CEO tends to
call this a live‑site culture, in other words, you build it, you
test it, you run it, you secure it, you troubleshoot it, you
improve it with responsibility residing in you, the creating
team, not getting fragmented across these Silos.


Three is the idea of team autonomy and enterprise alignment that
you want to let teams at the level of the feature crew Scrum,
cream squad, whatever your favorite term is, you want to let
these small feature crews work autonomously on their stuff, and
control what they are taking into the next sprint or what items
they're doing next.


You want them to support their stuff in production and you want
the mechanisms to align their work up to the common business
results, so that they know which needles they need to move by the
work that they do and they know how to view those gauges.


The fourth is shifting quality left and right so that you can get
a signal in the pipeline of green meaning green and red meaning
red, and in production, you can see continually what is happening
with every changes, you expand its exposure.


Fifth finally is using cloud to make infrastructure‑flexible
resource.


That's how I bucket it. I did one talk yesterday with my friend
Ellen Smith about how we moved our DevOps' ass. It's really a
story about eight years of taking what's started as a non‑premise
product and turning it into a cloud service and on‑premise
product. That was an attempt to myth‑bust the idea that if you're
going to the cloud, you need to start in the cloud and throw
everything away.


It was an attempt to say, "Here's a proof instance where we had a
business, pre‑cloud, with on‑prem product. We preserve that and
made it better, and use the same codebase to go the cloud where
the cloud is making the on‑prem better." Of course the cloud's
the majority of usage and the fastest growing now, but it wasn't
a throwaway, which of that story.


The other one which is similar, which I'm doing tomorrow, is a
talk about Windows' journey to DevOps. Windows division is the
extreme case of scale and legacy, and they have successfully
moved to DevOps. There were a bunch of bumps along the way. For
example, to get Windows to be able to use Git at their scale, we
needed to fix the Git, and that took three attempts.


Bob:  Really?


Sam:  Yes.


When we started doing something like a Git clone of the main
Windows repo, took 12 hours. That was if the network didn't burp,
or your laptop didn't go to sleep, or nothing else wrong
happened. If any of those things did happen, then the whole
operation needed to start over.


Bob:  Need to start again, yeah.


Sam:  That now takes a couple minutes. We
did a series of 300x or better improvements in Git performance
with what is now open‑sourced as the virtual file system for Git.
Windows motivated all of that to be able to support their scale
of codebase, which was hundreds of times larger than anything
else anyone was using.


Bob:  That's interesting. I did not know
that you guys were major contributors to the Git.


Sam:  We're one of the top two. We're the
largest open‑source contributor of any company, have been for
about two years now. Git is a project where we have been very
heavily in, and virtual file system is one of the latest aspects
of that.


Come to the talk tomorrow.


Bob:  OK, I may. What time is it?


Sam:  11:25, I think? 11 something. The
times here are weird. All these weird five‑minute increments.


Bob:  [laughs] . It is five‑minute
increments and three hours off, because I'm an East Coast person.
Are you out of...


[crosstalk]


Sam:  Along Seattle.


Bob:  Thank you so much, Sam. This has been
great. Is there any one thing you'd like to close off with that
you're interested in?


Sam:  Yeah. There's something that I'd like
to make our listener aware of, and that is I curate a website.
The short link to it is aka.ms/DevOps. It's, DevOps and
Microsoft.


What I try to do is to put up our experience reports there, not
the high‑level marketing level stuff, but like, "How did you
actually do the change in testing? How did you go to no downtime
deployments? How did you start using service reliability
engineering? Etc."


There're about 50 articles up there, but half of them with good
video. They're just stories about how we work. I love people to
use that as a...


Bob:  As a resource?


Sam:  ...open resource.


Bob:  Thank you very much. It was very nice
meeting you and chatting.


Sam:  Thanks a lot, Bob.


Bob:  Thanks.


The Agile Toolkit Podcast is brought to you by LightSpeed.


Thanks for tuning in. I hope you enjoyed today's show. If you'd
like to give feedback or be on the show, you can ping me on
Twitter. I am @AgileToolkit. You can also reach me at
Bob.Payne@lithespeed.com.


For more free resources, transcripts to the show, and information
about our services, head over to LightSpeed.com. Thanks for
listening.


[music]