How Effective Discovery Supports a Successful Cloud Migration
John and Adam continue their discussion about cloud migration, this time focusing on discovery. Check out the previous episode here.
If you are going to move an entire enterprise to the cloud, there is a lot to figure out. But what exactly does discovery entail? And how can discovery be used to identify quick wins to start migrating systems to the cloud immediately? The also discuss how an integrated procurement component can increase success in federal cloud migrations.
Adam: Alright, welcome to DevCast! Today, John Janek and I are going to pick up our conversation where we left off last time. We were doing a discussion on cloud migration. Over the years, Dev Technology has done a number of cloud migrations to both AWS and Azure environments, and we’ve generated a lot of lessons learned. We put a process in place that we view as sort of a four-step process.
It isn’t necessarily a linear process, but you know–that agile feedback loop sort of process starting with working through IT Strategy which we spoke about at length last week. A Discovery Phase, which we hope to chat about a little bit today, and then Deliver and Operational phases of the migration process. And again, it’s not a linear approach, but something that continues to inform previous steps and helps you change and make course corrections as you go. You ready to kick off the conversation John?
John: Yeah, I’m excited. This is one of those opportunities where we really get to dive into something that we’ve been doing for a while and really talk about it at a technical level. So yeah, I’m looking forward to it.
Adam: Cool, cool. Well, so like I said we want to chat a little bit about the discovery process right? Because you started these cloud migration efforts. And well, frankly, if you’re going to move an entire enterprise, there’s a lot to figure out. You know what systems do you have? Where are they? Who’s the system owner? Do you have a configuration management database? What documentation exists on some of these legacy systems? There’s really a lot to unpack there wouldn’t you say?
John: Oh yeah, so let’s break that down. So everybody likes throwing out, so you need to do discovery right? You need to do a Discovery Sprint. You need to do a discovery phase, you need to take your pick, right. What’s really interesting, is that very few conversations involve what does discovery entail, and I think you hit on the big ones. How do we figure out what’s in our inventory list? And there’s those big categories that we can think of, right? I was on LinkedIn recently and somebody was posting about software inventories, and supply chain management, and believe it or not I think that’s part of your discovery too. Not only what’s on your network, but what are the things that tie into those things, right? What does your dependency network look like? All those things, because that’s your supply chain, right?
And if any one of those dependencies are altered, or interrupted, or suddenly unavailable, what is the impact on your existing systems, right? So there’s all these really interesting and intricate interdependencies. And so, I think there’s a conversation around both the work that you do to discover, and also the work that you should be focusing on automating in order to discover, right, because that component, as I mentioned in this LinkedIn post, I made a comment where like if you’re still doing software inventory by hand, chances are it’s out of date the moment you click save, right?
Adam: Absolutely that’s true. You bring up a lot of good points there, and I think, you know, you start sitting down with customers and talking to them about a discovery phase, and I think the initial reaction is “that sounds like a lot of wasted cycles.” Going through documentation, building reports, spreadsheets, blah blah blah blah blah. And it doesn’t sound like there’s a lot of value there, right? And I certainly understand that hesitation, right? I mean, I think, you are looking to find quick wins which we spoke about at length last time. Like how do you identify those quick wins, and how do you start moving systems immediately?
But it’s not to say that discovery should be pushed to the back burner because–let’s be entirely honest—you are going to do discovery whether you set aside time in the beginning of your effort to begin this process, or if you just do it as you go. Because some of the items that you’re going to have to discover include where is that system hosted, right? What are the memory and processing requirements for keeping that system running? So, you have to do discovery—whether that happens when you just move a system to the cloud and say, “Oh, Geez, the CPU is pegged at 99%, we’re going to have to, we’re going to have to, uh, you, know, use a bigger machine,” right? You know, so you are going to do discovery, but I think there’s a great value in doing it. Starting it at least upfront.
And as you know, John, we’ve kind of developed this what we call the Meet in the Middle Framework, right? It’s a top-down and a bottom-up discovery process, allowing us to look at not only those systemic level needs, right? What are those actual requirements of the system? Is there a configuration management around that system? You hit on automation, that’s a big one—automation maturity. As you’re doing that kind of bottom-up discovery, but we also view that top-down discovery as well. What are the data storage needs? How many system interfaces are there? What are the requirements of those? So you know, there are a lot of things to unpack here, and I think we’re probably going to only be able to glance the surface of this, but what aspect of discovery do you think is maybe one of the most important areas to focus on when you’re starting a migration?
John: Yeah, so that’s a really great question. So let’s start at the beginning, right? So last episode, we talked about this idea that you want to iterate, and you want to do those quick wins, right? So let’s assume a worst case scenario. We walk into an organization, like we want to get to the cloud, but we don’t trust our software inventory. By the way, for a government agency this would be mind-blowing, because FISMA requires reporting, and 300 exhibits, and all the different reporting for budgeting requires inventories, right? But as we all know when we’ve done these things, they’re all horribly out of date and they are never correct, and they are oftentimes skewed to meet certain bureaucratic requirements.
So let’s say you walk into an environment, and you’re just don’t trust the data. So how do you get to where you want to be? And I think the very first thing you need to do, right, and we talked about this last time is: “OK, what’s our strategy? Where are we trying to get to,” right? We talked about the different—well, we very briefly talked about the different buckets—then we talked about what does a normalized cloud state for an application look like?
And then what you could start doing is you can start saying, “OK, where do we start?” We start where you do your work today. Right? So often times in a discovery, you know, there will be a shadowing period, or a ride along, or a side by side, where you actually have a technologist engage with somebody in the business operations. It says “what’s your day look like” right?
And what that tells you, is these are the high priority applications that get used on a regular basis, and those—as we’ve talked about—may not be the first ones to move. But they’re important because it starts unpacking the discovery process.
In the in the meantime, you can start looking at what is our top-down perspective, right? That’s a very bottom-up user centric perspective. You can also go into the data center and run a systems map, right? Use some tools to get in there and understand what is running in that space. And again, that’s kind of a bottom-up. From a top-down perspective, you can take those data artifacts that have been published and have been filed and start learning from that perspective too. And you can ask the questions of leadership, of what things are important? So I think there’s an interesting discussion around where do you start? From my perspective, it’s always best to start with, well, what do you have that you trust? And who can you talk with? Who’s going to give you ground truth? Right?
Adam: Absolutely. I think those are some good points for where to view that starting point. It’s also easy to start looking to automation and tooling around that as well. And I know that’s something that Amazon has recognized that need for as well, to be able to perform. Kind of that deep discovery. Understanding the inventory, understanding the business case. And they recently purchased a company called TSO logic. To help perform some of these. And automate that holistic view of your portfolio management. So you know, even a large company like Amazon realizes, hey, look, if folks are going to move into our cloud environment, it’s very essential for them to understand what systems they have, what state they’re in, what are the operating requirements. Let’s not forget another important part of discovery: that is going to help inform that estimate of how much is this actually going to cost to run to migrate and run these systems in the cloud? You know, until you know what those system specifications are, you’re not really going to be able to come up with an informed estimate. And I’ll tell you what, it is very rare that you wind up talking to a CIO who says “hey, don’t worry about the cost. We’ll will pay whatever it is,” right? That’s a number that matters to probably all of our federal customers.
John: Yeah, I think that’s a really good point. We all like to think our agency pain points are unique. But although the people we serve in the mission we serve might be very unique, oftentimes there’re commonalities, especially in our technology and how we choose to employ it. So there’s so much we can learn from each other, right? And I think that’s a really key component to this whole conversation around discovery too, which is doing the environmental assessment, right? Who else has done this and what were the lessons learned out of their processes?
I think at this point, enough people have moved to the cloud that there are a lot of shared experiences that can be used to build a foundation of knowledge. What do you think?
Adam: Well, of course I mean, I think there are always going to be shared experiences that should inform how you execute your migration. But I also think like a lot of things in life, you know, those experiences of others depend on their unique circumstances, right? You know when my wife and I were having our first child, we got a lot of parenting advice from everybody. Some of it was solicited. Some of it was unsolicited, but the best piece of advice I got was “listen to all the advice, but do what works best for your family unit,” right? Because it might work for somebody else, but it doesn’t necessarily mean it’s going to work for you, and I really think the same is true in a lot of these major technology discussions, right there. There’s one way that this organization migrated to the cloud that was very successful for them. But constraints and factors within your own agency, for your own organization are going to be different, right? And you have to kind of adapt to that. That’s why we’ve created this methodology of agile around how we develop software, right? Because it’s not a strict science, there has to be some art and feedback and changing to that process as you go. So I think I think it is very important to understand how other folks have migrated to understand the best practices and allow that to inform how we move forward as a team as we’re migrating systems.
John: Yeah, and let’s talk about that a little bit ’cause I think it’s a really key component. You talked about discovery being this process—and you used a word there and I can’ t remember what it was—but it triggered a thought, so I’m going to kind of work with it and we’ll see where we go. You know one of the things that we talk about that fast iteration, right? Oh, I remember where we were going with this; I want to tie the whole idea of discovery to this idea of data, right? So I posted recently on LinkedIn—in case you can’t tell, I’m a big LinkedIn fan because it’s a way to have these kind of great conversations without all the noise that goes along with Twitter. So I tend to be a lot on LinkedIn, right? But it’s really interesting to me that we talk about discovery being this way to learn and to understand what’s happening in your environment. And I think specifically, there is a lot of data that gets generated out of good discovery that allows you to refine your questions. And this is something that gets missed a lot.
So Adam, you talked about how when you were a new parent people gave you a lot of advice, and the reality is, that’s all data coming at you, right? And you use that data to understand this is what the environment might look like. Understanding, of course, that your environment is going to be different. So use the data that’s given to you as a guidepost in understanding to develop questions. The discovery process again helps you understand your current existing environment.
And the more discovery you do, in the ways we talked about, by running tools in the data center to kind of understand what’s running right now. Where is it at? Who does it, right? Looking at the policies that are set in place, do you have governance components? Do you have CCB documentation right? What’s running, where is it running house? That’s all data. That’s all insight that allows you to define the questions that you need to ask and continue to refine on that.
I think the other thing you just commented on that was so important too, is that data helps you understand your constraints. And those constraints are what will help you get to a functional innovative, really, the outcome that you’re looking for. And those things all link together, and this is why, you asked in the beginning, why is discovery so important? Or at least that’s how I interpreted what you asked. And that’s one of the biggest reasons why. Because discovery gets you the data that gets you the insights you need in order to achieve the outcome that you want, right?
Adam: Right, I think that’s entirely true. You know we talked about the IT strategy phase. And much of that is sort of that assessment. The lay of the land; understanding at an enterprise level; what you have; understanding what your stakeholders expect from the process; getting a sense of operational requirements.
But when you start moving into discovery, things definitely start getting a little bit more technical. I should say a lot more technical, right? And they absolutely need to, because one of the outputs, in my mind, from the discovery process is being able to create a road map for your actual migration, right? Take that strategy, overlay what you’ve found in discovery, and you’re now going to have, you know, kind of all this great data that you talked about, that basically is going to guide you through how and when to migrate systems.
Because you know, there are going to be a lot of things that come out of discovery that are probably going to be unpleasant. You may find systems in your enterprise that you didn’t know existed. We’ve certainly seen that before. Machines that are hosting multiple applications–it’s probably like a physical server somewhere. And maybe they have strange requirements that they actually need to be in the same JVM. I’ve seen that strange stuff before as well. So there’s going to be a lot of stuff that comes out of discovery where you’re probably going to figure out where more cycles need to be spent. More discovery needs to be done.
And you’re going to find areas where your quick wins are going to be, because that’s going to be those systems that we talked about where hey, maybe there are very few system interfaces, it’s already running on the latest version of Red Hat Linux; and boy that’s easy to spin up—an EC2 instance in the cloud and just put our system over to that, you know. So you are going to be able to identify quick wins and you also are going to start being able to identify those projects where even more discovery is required.
John: Yeah, I think that’s a really good. Again, you know the data that you gain allows you to refine the questions to gain more data, right? There’s no stopping point on this journey. A lot of folks think that there’s a point at which you can come and say, “We’re done. We have done all the discovery that we need to do.”
And this goes right back to the beginning of the conversation. If you’re doing it right, you’re always discovering new things and you’re always innovating. That doesn’t mean in your current operating state, but what you’re doing is you’re finding “where should we go next?” And this is another interesting play into the whole supply chain. Decided the discussion too, because when you’re doing discovery properly, it’s not just what’s in your perimeter or in your security boundary, right? It’s also what systems, in environments and software do you tie too and have dependency upon in order to do your work as well, right?
It becomes very quickly this is huge, expansive discussion around what does it actually look like in order to be effective in my role? And you can’t do it all at once. You will literally spend all of your time doing it, so don’t. Make it part of your agile process, right?
Adam: Right again, I mean, I all this stuff does need to be done in a very agile way. Because if you just sit around and do this in a waterfall. Sort of a meticulously painstaking process of documentation prior to doing any execution. Well, you might be discovering things that aren’t useful, right? You know spending cycles doing something that—well, you know, actually just taking an action and trying to migrate this system, we would have discovered more than just SSH-ing into the box and taking a look at this and looking at that. You know, sometimes action is the best course of discovery, not just sitting around in a read-only mode, trying to identify things and put together documentation.
John: Yeah, I think that’s really interesting. Yeah, and I think that’s part of it, right? So we talked a lot about it last episode, about the innovation piece too, and how we’re constantly innovating based our constraints and stuff like that. Your discovery has constraints too right, so the very first thing you’re going to do, like we talked about, is look at those policy and guidance, the CCB documents, the CM documents. What kind of documentation have you produced? Is it accurate? Do you trust it? What kind of very broadly, we have some Red Hat Linux over here. We have some Microsoft over here. We have some Oracle over here, right? We’ve got some of these systems, and “oh hey look” like you said, “oh here’s a Drupal box.” Well that’s interesting, why don’t we just move it, right? It doesn’t have any dependencies. It kind of lives on its own. Wikis are great like that. People stand up these wikis and then they just live on their own.
But you find a system that you can move and then you move it. And that’s good discovery. Right? What is our quick win? What do we understand? What do we learn? And then how do we actually move it forward? And that’s good discovery, so I think that’s an important part of it too, right? There’s discovery and you can—it’s like analysis paralysis—you can very much get into this mode of we don’t understand everything yet. We have to understand more. And you can literally freeze up your entire organization that way, right? The answer is don’t do that. Start the discovery process and then keep iterating.
Adam: Yeah, absolutely you know, I think there’s something else that’s very interesting that comes out of discovery that we don’t talk about very often. And you know in the IT strategy portion of a migration plan, right? Part of it is kind of that assessment of your DevOps maturity, and some of that is the automation, but like we talked about in our previous segment, it has more to do with that organizational culture. And what’s interesting about discovery, even though it really isn’t about establishing a kind of this cross-functional culture, but this is really where you see how mature you are and gives you a great opportunity to increase that majority before you actually do work. By work, I mean migrating systems re-platforming, re-architecting, whatever that might look like further down the line.
But here’s where your development teams, your infrastructure folks, security folks, QA folks—really start coming together to understand some of these system level requirements, the interfaces, what does it mean to be successful to migrate the system? How do we migrate the system in? Hopefully you start getting all these folks in the room together during—you know, kind of what I call a low-impact phase, right? Because once you start moving systems, changing code, changing configuration, developing automation, and moving something to the cloud where it costs real money. You know it’s better to kind of create that collaboration, create that DevOps culture in this low-impact phase.
So I think there’s an advantage there that we don’t often recognize, but it’s something that I’ve seen that kind of happens naturally through this process as well. That’s a huge value add, because once you start changing configuration, changing code, well you really need to make sure that those teams are communicating and that they are collaborating. So even if you don’t have a high DevOps maturity moving into a cloud migration, I think you’ll find that is a fantastic opportunity to really push for that organizational alignment around specific tasks to achieve success here.
John: Totally agree. I absolutely agree with that. I think that in some ways it’s a control gate, right? In other words, if it takes two weeks to get everybody scheduled to get together to have a conversation around discovery, right? That’s not an auspicious start. You know what I mean? These are—and really great point Adam, right—that trigger to say “we’re all-in gang.” We’re going to do this, and we’re going to be collaborative and we’re going to be co-located, and integrated, and really work towards the outcomes. I don’t mean co-located necessarily physically, especially in this new environment we’re in. But you can be virtually co-located. You really can. There’s plenty of ways to work together, apart, right? So I don’t think these things are impossible to do in today’s environment. The point being that if you cannot get everybody in the room at the same time for conversation around, “how do we understand this environment better?” You might have some other problems you need to work through first, right?
Adam: Yeah, yeah. And it’s to be expected that you probably are going to have issues, right? You know, I think we live in an era here—in the federal space particularly—where how we create contracts for different teams to work together and how we incentivize that, and how the federal government and how the contract offices there manage these contracts and incentivize individuals in different teams to work together is very interesting. Pretty much still in its infancy, I would say, but a lot of the headway has and been made there. And John, I know in your previous life you’ve kind of served as a contract officer. I know this probably wasn’t something that we were planning on chatting about, but I’d be interested to start getting some of your thoughts about how we might consider advising our government customers to change contracting to incentivize different contract teams to work together for a common goal?
John: Yeah, that’s a really great question because I think you know the challenges, and this is where you really need strong leadership in your contracting organization. It’s an interesting conversation because there’s two pieces it. One, that often times the mission side and the contract side, right? People don’t understand, there’s two different lines of reporting in this arrangement. You know the contracting organization technically legally reports up through the procurement executive of an agency, right? And is completely outside the chain of command, necessarily, of a lot of the lower components. The program managers, even some of the more senior people in the organization. And it’s done that way on purpose in order to comply with federal law and some of the things that we want to do from a federal procurement perspective.
So to your point Adam, the very first thing agencies can do is to encourage and find ways for your procurement staff to integrate closely with the people doing the work, right? And this may sound completely counter intuitive because you know, oftentimes, procurement organizations try and shield their CEOs as much as possible, so that they have the necessary time to do the work they need to do. All very important. And the risk and the responsibility they bear is tremendous. So don’t get me wrong there. But at the same time, you know, there’s a lot of different kinds of warrants, there’s a lot of different kinds of ways to approach procurement.
The military and even the foreign service do this pretty well in that, what we do is, we say look, you know, not everybody needs to manage 300 million dollar portfolios. You know what I mean? And so it’s entirely possible to say to somebody, “We want you to do a lot of smaller activities very quickly and very well.” And so we’re only going to authorize you the ability to contract up to a quarter, $1,000,000, or $1,000,000 or $5,000,000 an in this way. It’s really interesting because what you do is you say you know your goal isn’t a 10-year contract. Your goal is complete work for specific need executed quickly, which is the very quintessence of agile, right?
And if you’re really talking about how do we reform contracting, it’s two parts, right? Bring the contractors closer to the work—to the contracting officers—integrate them. And what we’ve tried to do is we’ve tried to set up all these different layers of management in between. You know we have CEOs and then we have CORs and then we have GTM’s, right? It’s just created this awful mess of and, I’m sorry we have coders in between CORs and GTM’s, right? So there’s just awful mess of hierarchy that we’ve tried to interject between our contracting officers and the delivery.
And the reality is, it doesn’t work. It’s not appropriate. It’s not good. It’s not where it needs to be, and that’s been proven time and time and time again. So how do we move things faster? And you know you’ve got a lot of fantastic examples on how to do that. But the basis of it, you know you got a lot of work by USDS to kind of set up the new Digital Information Technology Professional Procurement Certifications, right. The Defense Acquisition University is doing a lot of with that now. But the reality is that until you start getting agencies who lean into it and say like, “hey, we’re going to take, for every GS-13 program manager, we’re going to stick a GS-13 procurement specialist with them at that level, with a warrant ready to spend.” Then we’re not going to make it, right.
So there’s my challenge for any govie that’s listening out there. Talk to your procurement executive about what does an integrated procurement component look like for your organization? And let’s bring those things together. Because we’ve spent a lot of time talking about discovery and what that looks like and how to breakdown the silos. And then we kind of took the bin towards culture. That whole cycle, really, if you want to be really philosophical about the whole discovery cycle–it starts all the way back with budgeting and the appropriation. And how are we going to pay for this? And who’s going to do the work? And how do we write the RFI and the RFP? By the way, I hate RFIs. We can make that whole segment. The RFP and all the different components right? Because discovery isn’t just about the technical discovery, it’s about understanding where you’ve been and where you want to go. So I’ll get off my soapbox.
Adam: No, I’m glad I asked you that question. I didn’t know that you were going to be able to give such a complete answer out in the fly, but that’s fantastic. And I definitely feel pretty strongly with in alignment with everything you said there. Just kind of thinking back to previous modernizations and migrations: The ones that wound up being the most successful had a lot of engagement from the contract officers or the CORs. They were right there in some of the meetings. They were actually sometimes asking the toughest questions. And ensuring that everyone was in agreement on what was discussed and what the timeline for a deliverable should be. And honestly, that’s fantastic.
Sometimes you do need–I don’t want to say an adult in the room–because you know us developers, us IT project managers are certainly capable of running projects, but I think when you were in the middle of a very challenging technical effort, it’s nice, to have someone who’s kind of trained to be an objective third party reviewing the situation as it’s unfolding. And I think there’s a lot of value to it. I think it protects both, obviously protects the government, helps protect the contractors, and I think ultimately it leaves everybody in a better position to achieve success so. Yeah, it’s very interesting little sidebar we had there.
John: Yeah, I think to your point. What is the stakeholders and perspectives right? This even is a discovery thing too. The more stakeholders and perspectives you have, the more data and observations you will be able to collect, and the better your outcome will be. So to your point—Gee, having that contracting officer or the COR who’s very well versed in acquisitions in the room, asking the hard questions made our outcome better. And there’s a part of me–it’s like, well, yeah. But I think it’s worth reiterating that that–yeah, because they bring a different perspective into the room. They bring a different perspective that’s different from the PM who’s looking along their bureaucratic line of sight. And I don’t mean that in a derogatory way, but rather they are in the organization. They are part of the delivery for the organization’s mission, and so they have a very specific perspective. They don’t share the contractor’s perspective. Who again has their own perspective. Instead they are sharing their perspective, which is “How do I spend money to the benefit of the taxpayer,” right? To fulfill a need of the US government. You know what I mean? So it’s these different perspectives that all come together to create the outcome that you’re looking for.
Adam: Yeah, absolutely. Again, it’s another data point. And going back to you know, really what we’re talking about here. That discovery phase? Yeah, that that contracting officer is another source of data, right? Their input and their perspective on the situation helps inform and drive decisions. And is absolutely essential to making sure things run smoothly through the course of a modernization migration effort. So yeah, it’s interesting that kind of everything that we talked about in discovery ultimately leads us to more data, right? And it doesn’t have to be in a pure and simple discovery phase. Again, we’re not doing waterfall here. You know, a lot of this discovery were talking about with a contracting officer that could honestly be happening through execution an operation at any phase, right? As all this discovery is right, it’s a very fluid process where hopefully always learning together and hopefully capturing this data. To inform and or verify our decisions. So yeah, I think that’s probably the main takeaway from our discovery conversation. John, wouldn’t you agree?
John: Oh yeah. Be in that learning mode, right? Maybe that’s how we should have started this conversation. There’s a lot that we can talk about with discovery, but if you’re not open to what you’re going to find, then you know–and that’s a really good perspective. All too often, people began a discovery phase with the end in mind. And I just think when you do that, you set yourself up for failure, be open, keep as many people in the loop as you can. And be integrated and you will come out with some amazing outcomes.
Adam: Alright, well I think that’s a great place to end it today. Thanks again, John. Another great DevCast in the books.
John: Yeah, absolutely. And thanks to you and Karen for all her help, and I think we’ve got some great guests scheduled in the coming weeks. So see everybody on DevCast soon.
Don’t forget to follow Dev on Twitter and on LinkedIn. We’ve got plenty of new positions coming open too, including all the discussions around data today. We’re looking for a data architect and we’ve got plenty of other opportunities to work with data. And with some of our really interesting work in our different client projects, so keep in touch, keep in tune, and look forward to talking with you again soon.
Adam: Thanks everyone.