Carlos Rojas - Lean+Agile DC 2018

Carlos Rojas, Director of Technology and Operations
at Fannie Mae, sat down with Bob Payne at Lean+Agile DC 2018
to discuss Business-Driven Agile Engineering.  Carlos shares
thoughts on Fannie Mae’s trailblazing Agile transformation – from
prioritizing agility-supporting corporate shared services
(recruiting, contracts & procurement, and facilities), to
machine learning and a mantra of “Automate Everything.” 


 


Transcript


Bob Payne: [00:00:01] Start there.


Carlos Rojas: [00:00:02] OK.


Bob Payne: [00:00:03] Hi I'm your host Bob
Payne. I'm here with Carlos Rojas. And Carlos you're talking
about transformation. We were chatting you mentioned how to how
to use shared services. So tell me a little bit about your your
your talk here today at Lean Agile DC.


Carlos Rojas: [00:00:24] Absolutely. So couple
of things to highlight right. One is you go and change a culture,
Most people start with the processes. Most people start with
their methodology and then they say we're Agile. And I think that
part of making it a transformational for an enterprise you know
when you're talking about 10000 people when you're talking about
300 product teams that are developing software you know you have
to take into account what we call the corporate shared services
as well. You know think about the recruiting recruiting efforts.
Think about the procurement. How do you write your contracts.
Right. Think about your facilities you know. Do you have an open
space that will be a reflection of what your mentality looks
like. So you have to account for those things. Otherwise you know
you end up with a great process but the culture is not impacted
by what you're doing. So that's one of those things that I
encourage companies to always think about.


Bob Payne: [00:01:16] Yeah. And we just I just
recently talked with Jose from from from Fannie. And I know you
guys are doing a very large transformation over there lot of
DevOps. We do some some training with you guys. 


Bob Payne: [00:01:34] And I'm am extraordinarily
pleased to see how how big the transformation has been over it
over Fannie because I worked there on a couple of early Agile
projects in the early 2000s and it was it was bureaucratically
painful to do it.


Carlos Rojas: [00:01:56] I know exactly wh at
you're saying. It was six checkpoints hundred twenty five
deliverable is about nine months for a single release just to put
a line of code in production. Now we've made a huge progress in
terms of you know time time to market an average of about 30 days
a month. We've got some project teams that are ready to deliver
every two weeks. But again it comes down to Hey we incorporated
our corporate shared services into the mix and then we think we
we we instigated this concept of automation you know everything
so far as he wasn't even called them up so to begin with. How do
we automate it. You know if it doesn't make sense if it makes
sense. How do we automate it and that's when we come up with this
concept of that paved road so far as the pay really was. What
tools are we going to use that will help us expedite or automate
some of those controls that were holding us back. Right. So great
transformation. Tons of work behind the scenes but you get the
point here is we haven't finished. We're not done yet.


Bob Payne: [00:02:58] And lean, Lean never
sleeps.


Carlos Rojas: [00:03:03] Exactly exactly. So
right now for example one of our key you know tunnel vision eye
vision items that we're trying to chase is agile engineering
practices right. 


[00:03:15] How do we turn code that is maintainable how do we
create applications of 19 applications that are going to help us
with you know a cloud-ready approach not necessarily how do we go
about the methodology what tools that we use now but how do we
make that an engineer solution that is like the ultimate driving
machine for us. So we're trying to look for that not perfection
but that high performing team that will look into those proses
areas that we'll tap into the business for example that's not a
mission that's going to help us have more quality right
provisioning of environments that's going to help us get faster
at some point in time I think that we're going to be looking at
how do we do infrastructure as code so that we have a fully blown
solution where you build your own servers if you deploy your own
servers as development team and you're responsible from beginning
to end. 


Bob Payne: [00:04:11] Hundred percent audible
and nobody has the password to production exactly one per cent
cut off. It's not the engineer.


Carlos Rojas: [00:04:20] Exactly. But at least
you can know who what when and you're not having a checklist to
do it. It's all part of a creation of the solution right.
Software. 


Bob Payne: [00:04:30] Yep yep you know I think
it is. It's ironic and oddly a common theme and I don't know
whether it's me or just the folks that are coming up to talk on
the podcast today but almost every one of them we've talked a
little bit about governance and how how this drives you to be
more governable auditable reduces risk profile you know rather
than being a risky play to be able to deploy twice a week or once
every two weeks or it actually improves your risk profile
considerably makes you much more audibly compliant with the
controls. You say you have in place.


Carlos Rojas: [00:05:23] Yeah. You know it's
interesting you mention that when I started my journey at Fannie
Mae I ran a governance organization over SDLC and I was in charge
of connecting with at risk partners like legal audit you know
Architectural Review Boards change control boards and all of
that. And instead of fighting through the system what we did is
we partner up with all of the governance groups and we said you
know you can do it the way you're doing it. But what if we do it
this way and let's just compare let's explore let's figure out if
there's a better way and if there is a thought that if it doesn't
work on the exploration says that this doesn't make sense then we
just go back and keep doing it the way we used to do it. I guess
what the answer was always yeah that works better than the way we
do it and we get more data. Thank you. So they were actually
embracing it and promoting it for us. 


Bob Payne: [00:06:12] Yeah. I don't know if
you're familiar with Mark Schwartz from Citizenship and
Immigration Service. He's sort of a firebrand CIO and he said
when he first got there you know coming from Silicon Valley like
you don't tell the CIO it can't be done. But what he got to
government is people would say well you can't do that because of
the far no federal acquisition of election.


Bob Payne: [00:06:39] And he said first he was
angry and then he said I'm going to become the expert on the far
and and what he what he did was that same approach was like ah
here's what you wanted. You know here's the safety that you want
from this regulation. And here's how we do it better. Right. It's
sort of you know 


Carlos Rojas: [00:07:02] The approach works. You
know where you partner with the groups that are supposed to be
the roadblocks or supposed to be the longest pole in the tent. If
it works you know.


Bob Payne: [00:07:12] Well it's, They're there
to keep the organization safe safe.


Carlos Rojas: [00:07:19] Yeah you know if you
don't do it it goes against what you're trying to accomplish
anyway because you're not just trying to produce code faster you
want it to be reliable maintainable with quality compliant.
Right. So. So it's a matter of just you can argue whether the
legislation is right.


Bob Payne: [00:07:35] But you still need to
comply with it.


Carlos Rojas: [00:07:36] Exactly. But sometimes
the legislation or the government has the right intent. It's how
we approach it.


Bob Payne: [00:07:42] Yeah it usually does.


Carlos Rojas: [00:07:43] Exactly. So you know
hopefully for those teams that are struggling with you know the
governance aspect of this I'll also just take one thing at a
time.


Bob Payne: [00:07:53] Yeah.


Carlos Rojas: [00:07:54] Because you know I've
seen people trying to cover it all, oh we're going to fix all
these problems right now you can use your theory of constraint
was the biggest one. I just tried.


Bob Payne: [00:08:04] Protects the constraint,
concentrate on that one.


Carlos Rojas: [00:08:07] Yeah. Did you read the
goal.


Carlos Rojas: [00:08:10] Absolutely. That's what
I got that from.


Bob Payne: [00:08:12] Yeah the Hurby or the
Brian in the Phoenix project. Yeah. Put water where the fire is
and in small increments it pops up. Yeah great.


Bob Payne: [00:08:27] So what are you looking
forward to over the next couple of years getting accomplished.


Carlos Rojas: [00:08:33] So it's a phenomenal
question.


Bob Payne: [00:08:35] It is also a journey it's
not an end point.


Carlos Rojas: [00:08:37] It is a journey. It's
not done so I'll say maybe three things that we're looking at
right. So one is becoming a high performing team in terms of all
the agile engineering practices. So really just looking under the
hood and figuring out do we have the right automation and we have
the right you know called brunching do we have the right
techniques that are engineering techniques that will create a
great product. That's one. The second thing that we have started
working on is the AI machine learning so introducing much
learning to our software development lifecycle.


Bob Payne: [00:09:09] Okay 


Carlos Rojas: [00:09:09] Right so right now
we've been using some regression basic algorithms with machine
learning to be able to look at the data on the history of
releases that we have so that we can do two things one we can
predict when a release is going to fail interesting. And then the
second thing is if it fails you know we can start looking at the
deployment scripts. Figure out hey this might be that two or
three things that the reasons behind why that release is failing.
Right. So we're piloting some of that solution right now. And I
think that aspect is cloud. How do we turn these applications to
be called cloud ready right when we know in some cases we have
monolithic applications that have been around for 23 years.


Carlos Rojas: [00:09:49] How do you take that
and move it to the cloud and get the benefit of it not just a
lift and shift, but saying Hey lets triangle that application
that are more functionalities to micro services I think those are
the three components that we're going to be focusing on the next
two years.


Bob Payne: [00:10:02] Yeah the the predictive
analytics side that's very that's very interesting to me and I'm
super curious how that sounds complex but it is probably
something that I started while I had a sort of field career as
computer architecture free I so far as I was doing that and then
decided to go to procreate. So it doesn't sound that complex. I
think the interesting thing for me is that you need a reasonably
large dataset for that to be able to to really start to bear
fruit. But I'm excited to hear you guys would have it so. 


Carlos Rojas: [00:10:52] So we have a reasonable
amount of data on releases that go back you know years okay. But
interestingly enough when we started training our morals we use a
subset of six months. And when we started with that subset of
data we identified about 150 fields that were important and then
we had the model train for a few weeks and then we test that we
said let's just see that confidence level and we put real data
against that model and then our success rate was about 10
percent. It was really bad. It's no surprise right. So instead of
saying instead of saying let's get more data more fields more of
features we say let's take those hundred and fifty and gotten our
way down to 50 and then we have a smaller subset of fields and we
have a smaller subset of data for 3 months and then we trend the
model again.


Carlos Rojas: [00:11:56] And then now we can
tell you that we're you know successfully predicting with a 90
percent certainty certainty rate. So we went back to the model
with less data less data elements are more precise data set and
our success rate went up.


Bob Payne: [00:12:12] Well I mean look at you
know what we're starting to see genetic markers for you know in
biology that have a very large effect. So once you find the
targets I think I think that level of that that that is amazing
that you were able to how long did it take you to go from 10 to
90.


Carlos Rojas: [00:12:36] So it was about a 12
week project. But then thinking about thinking about the project
I'm thinking about the data said it took us about 6 weeks. So I
would say you know quarter a quarter and a half worth of you
know.


Bob Payne: [00:12:49] That's amazing, are you
using, what are you using back end engine?


Carlos Rojas: [00:12:52] So we're using
TenserFlow. It's an open source from Google, using Python to
write the scripts and to be quite honest with you because we're
exploring our capabilities are just basic you know we're not
building a beautiful website that people can access. We're just
enabling the developers to say hey gone check this page and then
you know pick your acid I.D. your application I.D. and some
information on the model is going to give you the answers. So
it's basic. Right. It's still in the exploratory mode. We've done
a couple of good successful use cases and now we're trying to
roll it out to the company.


Bob Payne: [00:13:28] I am I've been pleasantly
surprised at the resurgence of Python.


Bob Payne: [00:13:37] I was in some of the early
Python conferences Python back when they were here in Northern
Virginia and D.C. and now running into it in the large
corporations that they also have micro Python which runs on a
microcontroller. So you can you can you can do hardware
programming. You were always able to buy it you needed a heavy
heavy thing. Now they're doing you know arm processors isn't
super lightweight.


Carlos Rojas: [00:14:12] But the difference is
you know the user experience at the end of the day you know when
you started developing maybe you know in my case 20 years ago
with Cobalt or C++ or even Java. Right. You have to have some
sort of computer science background to be able to kind of like
and understand that and and do something with it. Nowadays with
Python, It's more of a English type of you know development where
you don't have to be computer science guru to learn it and apply
it so I think that's why it's being so helpful for the adoption
of those techniques. 


Bob Payne: [00:14:49] I mean it was, there was a
certain, well there's a whole philosophy around you know Python
and you know people people do argue whether they achieve that but
they wanted it to be an easy learnable right. You know part LAMDA
part object part you know and be able to drop down to the metal
pretty quickly with assembly or see you know the whole idea that
you could tie in those when you needed to if you needed to
program and see were able to easily tie it in with Python.


Carlos Rojas: [00:15:30] Exactly. And to be
quite honest with you we're leveraging some of the open source
you know libraries to execute on some of the you know projects
that we have. 


Carlos Rojas: [00:15:43] So it's not like we're
building something from scratch anymore right. So that's helpful
as well.


Bob Payne: [00:15:48] Yeah. Well great Carlos.
Thanks for coming in. It's a pleasure chatting with you and I
wish you guys all the well you don't need luck all the hard work
in the world. 


Carlos Rojas: [00:16:02] Thank you for having us
here, appreciate the opportunity. Thank you so much.


Bob Payne: [00:16:05] Thanks.