Using Data Analytics to Improve the User Experience
Adam D’Angelo and John Janek are joined by special guest, John Bobosh of IQ solutions. John specializes in using data analytics for improving websites. John discusses how data analytics can be used to improve the user experience of website and applications by using data to identify areas for improvement.
You can listen to this episode, or read the transcript below, or listen on Apple PodCasts, Spotify, Sticher, Deezer, TuneIn, and more. And be sure to subscribe so you get new episodes when they are released.
Adam: Welcome to another DevCast my name is Adam D’Angelo. I’m joined by my Co-host John Janek. How you doing John?
John J: Yeah good. That’s me, I’m happy to be here excited. It’s evening, and it’s May, and were almost into June and we’re still in quarantine, so how about that.
Adam: We have a great guest tonight–another John–John Bobosh. Welcome John.
John B: Thanks for having me. I’m excited to be here.
Adam: Can you tell some of our Dev Technology listeners a little bit about yourself and what your background is?
John B: Sure. So I work at a company called IQ solutions. We typically focus on government health, so a lot of our clients are federal government health agencies. One of the things I do with clients is help sort through Analytics on websites and use analytics to come up with solutions for improving websites that we work on.
Adam: It’s a fun topic because John and I were chatting about this earlier today, just different ways to develop software, improve usability, improve the quality and data Analytics as a means to improve the usability is a very fascinating idea to me. It’s something that we’ve employed before on other projects doing things such as like AB testing to determine which kind of path is the happy path. The better path. What are some other ways that you’ve used data analytics to improve usability?
John B: So I tend to work on public facing websites and one of my favorite metrics to look at, I guess, just taking one step back from that even as first identifying what the purpose of the site is. So when the user comes to the site, there’s probably five or six things that we are looking for them to take or do before they leave. And then setting those up to track in Google Analytics. Those are just called goals in Google Analytics, and what I’m usually looking at is landing pages, so where they start on the website to goal completion, and what is the process, and then kind of taking a step back looking at pages where a lot of people start on the site and if they’re completing a goal while they’re there. And if we find that people aren’t completing goals, on like a highly traffic landing page, taking a look at what the user journey is an then recommending some changes to either content or how things are organized, or maybe even taking a step back and looking at the SEO of the whole thing like is the page indexed properly? Do we need to change some things with metadata? To improve the number of people that both come to the site and then complete the action that we’re looking for them to take.
Adam: A lot of our customers run Intranet private sites. Things that you wouldn’t be able to change your search engine optimization around and track using. I don’t think you could track using some of those Google tools, but maybe you could. What are some other ways that we could track data and kind of figuring out how to help our users complete the tasks they need to complete in the system.
John B: So I I think there’s like a similar approach that we can take, so I think just before this, you and I were chatting for a second about some processes that have to happen on the Internet. Maybe a form that needs to be filled out, or several steps that someone needs to take to send something on, maybe to another department or for feedback, whatever it may be, so you can set up in Google Analytics or any analytics service really, just goal tracking, and creating a final to see where in each step of the process are people falling off and then kind of looking at some of those steps to say, well, can we make this more streamlined? Is there something happening here that’s confusing to the user? and kind of just taking a look at that process where we see a big falloff from one step to the next?
John J: You know, John. One of the things is really fascinating to me. Is this whole idea of, you know, we’ve really entered into a phase where we’re doing kind of end to end digital transformation now. where we’re really embracing our end users, asking them to be part of the process, understanding through empathy where they’re coming from and how they’re going through this digital transaction and service delivery with us. And we’re using all that to base these things off, and I’m kind of curious when you talk about when you talk about analytics tools like Google. Analytics like Piwik right? Like open web analytics, right? We’ve got all these different sweets and for our government clients you know it’s not always going to be Google Analytics or use Google because it’s a public facing site and you want those broad data collection abilities right? Sometimes to your point Adam, you’ve got an internal site and those same principles apply, right? John, you just said that you know those same things can apply no matter what we’re looking at for a customer journey. I’m curious, is there a threshold where analytics don’t work? If I have a system that has, you know, several 100 users or somewhere or under 1000. Does it no longer apply, or can analytics provide insights really to almost any system? I’m kind of curious your thoughts on that.
John B: So I guess I have two thoughts on that because sometimes we may cross a privacy threshold where if there are two few users in the system, we may be able to start identifying individual users. So especially if there’s something behind a firewall or people can log in. So I think there’s some concerns there depending on what we’re tracking, but I think generally if you could anonymize the data for a lot of these things, there are probably still valuable things to glean. So, if we’re talking about Internet systems, you can see how many people are using phones or not using phones. Maybe people have switched more to Microsoft Teams or instant messenger, especially now that we’re all working from home. Maybe there’s a different system people are using, and I’m sure you could apply that data in some capacity, maybe reduce costs by reducing the number phone lines you have. Or maybe you need more connections for something or other with the data that you’ve got. That kind of getting to what you’re asking or what are you thinking there?
John J: Yeah yeah, absolutely. I’m wondering, so related to that, you talked about like in the in the journey, being able to figure out where the falloff looks like. So what happens next, right? So you see what happens where the people fall off and you’re like, OK, there’s a problem on this landing page or with this goal set. And you dig into it, and that triggers a whole other kind of iteration. What are you expecting this see right? How does that process work?
John B: So for for me, typically again kind of thinking more about the public facing websites. We’re going to look at the user journey from there, so at the step where people are not completing the action or leaving the site, leaving the funnel, whatever it may be. I think it’s kind of good to go to that point and the user journey user experience and say what’s happening and kind of just go through the experience yourself. So it’s like a heuristics review. You’re going to look to see what’s happening there, and hopefully you will be able to identify what’s happening. And if you can’t on your own, doing usability testing if possible. So again, if we’re talking about sensitive government systems, probably can’t really do some real usability testing, but if you can, it’s great you go observe the users, see what they’re doing, ask some questions. And then take that feedback, maybe take it to a designer or user experience person and have them published. Some solutions that we can then implement on the site.
Adam: Now let me ask you this, John, you know, are you getting involved in the early phases of the kind of pre-launch of a website, and by that I mean in the design phase, right? If you’re re envisioning it, let’s say a full-scale modernization project where you are looking to take a system that maybe existed on the mainframe or in older tech stack and the development team is supposed to be really just modernizing this, moving into may be moving into the cloud, moving into a modern technical architecture. Is there work for you to do in your mind before that system launches to make sure you’re developing the system right the first time?
John B: So I think the prep work can go in different directions depending on what the need is. So if you have data for how systems are currently used and your modernizing the system, there are probably certain things that you want to keep, and there’s probably things that you can get rid of so that can change your requirements. I think on websites and the like, it’s always good to look at the data and kind of see what’s working and what’s not. What are people looking at, what are things they’re not looking at and then trying to figure out why? Audience research is always very important at this stage. So if you can talk to the users and get a feel for what they need, that’s also great, because then you can apply that and build something that really speaks to them.
Adam: Yeah, I think that’s where we’ve had the most success is really doing some of that human to human interaction. Talking to our end users understanding how the system may or may not have been working well for them in the past. I’ll be honest, we really just have not had a lot of Analytics in the legacy systems, to help drive decision making moving forward. But here we are. We’ve moved a lot of systems forward over the last couple of years and now I think the challenge for us is how do we start saying OK, instead of giving you guys new features, let’s take a look at how this system is working for you today and using data figure out ways to improve the usability so you might have fewer button clicks to complete a task. Fewer screen switches to look up data or get the report you need, and I think that’s what the big problem that many of our teams are faced with today. Trying to move our customers forward from where they are right now and applying data to do that. So with that in mind, I actually don’t think I have a question to ask, I totally lost that.
John B: Totally fine I think one thing that comes to mind in what you said is this is a word that probably not everyone knows atavism. Are you guys familiar with the concept of atavism?
John J: Yep.
John B: So I feel like just to clarify, for people who don’t know, atavism is like a feature or something that has carried over from a previous system that is no longer really needed. I think examples of this in evolution are like humans still have a tailbone, but it’s not really needed for anything. It’s just always kind of been there. I feel like sometimes when we moved from like legacy systems to new systems, we carry some of those like atavist features with us and we don’t need them so I always feel it’s good to kind of take a look at what you were thinking, like do we need all these pieces? Maybe there are definitely things we need. Maybe there are definitely things that we don’t need to carry over because I know we’ve always done it that way. But why do we always do it that way?
John J: You know when next so I don’t know if you will remember next right? The company Steve Jobs founded after he was kicked out Apple and then came back and bought into when he went back to Apple. But next had a product called web objects. Do you remember that Adam? Web objects was really impressive platform but one of the reasons why it was super successful was because at the time it was one of the single platforms that you could go to that actually did screen scrapes of terminals, right? So you could do a terminal emulation session with the mainframe. And then do a screen scrape and pull all the data elements into the framework. So it was super fascinating that you had this product that was largely embraced because it provided this capability and backwards connection into the legacy in order to bring it forward. I think there’s a really interesting question around when can you use analytics to determine when to let go of the past? I mean, that’s my. That’s a really fascinating question from my perspective.
John B: I was reading something, I don’t know which company it was. I think it was like a big Fortune 500 company. They eliminated voicemail. And the reason was that no one used it, no one checked it and that’s like one of those things that make me think about what you were saying. There’s do we need that system still? When is the right time to give that up? Isn’t it better just to have someone send an email?
Adam: Yeah, that’s totally true and we’ve been doing a lot of internal to our own company. We have a lot of different cloud environments, a lot of different projects that we’ve started and built over the years and you know a lot of that stuff happened so organically that you don’t always know who’s using what, who is interested in that server or this server. Is anybody using that? Do they still work here? And we started to wrap a governance process around it, and when in doubt, we shut things off. And our motto is basically, look if the users are upset about it, we’re probably going to hear about it.
John B: I mean, assuming you can turn it back on, that’s great.
Adam: It’s a good assumption. Yes, yeah, We don’t delete everything, we just you know press stop on the server and see what happens for a couple of weeks. And if nobody, nobody complains, you know is clearly an unused feature, right? So why are you paying for it? Why is it become this burden to the organization, because all those little pieces do end up, you know, costing something right. And even if it’s a nominal in a dollar cost, it’s still something that has to be maintained and shepherd it in some way. So even those features, those remote features of a system that nobody is using, they are still in the code base, it is still something that a new developer is going to have to probably learn or understand that exists. And why is it there? If it’s not really providing value, and I think, you know, at the end of the day, you know. John, I don’t know if you listen to our podcast, but one of the things that you’ll hear John Janek and I talk about regularly is business value. So at the end of the day, if it’s not delivering business value, get rid of it, right? I mean, we’re not holding on to features here or technology for sentimental value. That’s not really our business.
John B: I think that makes total sense, and I feel like there’s lots of systems that I’ve inherited in certain ways that I’m like, why are we doing it this way still?
John J: Yeah, so really interesting question Adam, because you know as you’re talking about those things in my mind when I’m thinking about is, you know those are very reactive. We take that position because we don’t have the inside of the ability to say we have people coming to this service, right? As Adam said we literally turned certain servers off just to see if anybody would notice, and in some cases they did. And then most cases they didn’t and it’s been interesting from that perspective, because I think when you talk about Analytics, John, what you’re talking about is the ability to kind of get ahead of that problem a little bit, right? To see what those trends are. And before you get to the point where, hey, nobody’s been to the server in six months. Do you think we can just turn it off? You can see that trend well in advance and just think of the kind of, you know to your point Adam, the business value that you generate out of that by saying, You know we should turn this off now. We should be looking for what’s the seeing a huge drop off people aren’t using it. The capability set has either exceeded or been eliminated by the population. What where are they going right? So I think that’s a really interesting question for you John. When you see drop offs and there’s nothing immediately evident, do you ask yourself that question? Where are they going? You know, you see this on the Internet all the time where people stop coming to your site and start going to other sites ’cause they’re getting better answers somewhere else, right? So I’m kind of curious, how do you think about analytics in that way in order to be able to be insightful rather than reactive, right?
John B: So I think that’s really difficult. We try to figure those things out, but sometimes you don’t notice what’s happening until later, because maybe you’re not looking in the right place. Or maybe you didn’t realize it was happening. I think one of the things internally that we’ve been discussing is where people going for health information during the time of COVID. It a lot of the government clients I work with, they put up banners on their websites or made statements about coronavirus, and the data we’re seeing is that a lot of people didn’t look to many of these agencies, for that information. They really were looking to the CDC, basically because I guess the CDC is what’s shown to us on the news and things like that, and that’s where a lot of people went and we can see that in the data that we have, that’s in the top 500 coronavirus pages in the government right now. I would say almost 90% of them were for CDC, so maybe it isn’t always necessary to react. I guess I’m sort of saying. And I’m losing my train of thought a little bit here. So remind me again, John, where we going with us?
John J: So yeah, I was just curious, you know, rather than just turning servers off you know it, I think you’re exactly right, it is sometimes hard to see where you’re at, right, when you’re there in the moment, but when you see general trends, right, that are there formed over like you said that that analytics ability to look backwards and see what behavior looks like over an extended period of time. There’s a lot of value in that. And you can understand a little bit more about what to expect if you’ve had it. For instance, so coronavirus is a good example, right? You deal with a lot of health websites. I’m sure a lot of them expected to be sources of information during this particular period of time, and what you’re finding is that that’s not where people are going. So you know how do you? Well, I’m not going to get into that question, but I think there’s a question around going back to your user experience discussion and Adams business value discussion of making sure that you’re flagging this. How do you flag it that: hey, this was a good experiment. It’s time to iterate, you know, so this is the other thing that when I talk about a lot, is this idea of agile means always iterating towards value, and so if you do something and you don’t get immediate value out of it, what’s the pivot, right? What is what is the next thing that you can do to experiment to try again?
John B: So I mean, I, I think continuing to iterate is always good, so you talked about AB testing before. Sometimes you do an AB test and it either has a negative impact or no impact at all and that is OK. I think as long as you’re kind of trying to figure out ways to improve. That’s bound to happen, you know, not everything you try is going to succeed. I think just going back 1 second to talking about when to shut something off. I think for me when I see a website traffic declining, I tend to go look 1st at the website to say is something broken, is it not functioning? Because frequently when I see traffic tank to zero, it means that something is wrong, so I feel that’s always step one is like the troubleshooting. What happened? Is everything OK or has everyone actually shifted on?
Adam: I think that’s a very good indicator of content or quality not being there. We had an interesting case of that on a complete Internet system here in the federal government, and I probably can’t say anymore in terms of details about what it is than that. But we had it directly from the user’s mouth that they don’t like using the system. Instead, they track everything in Excel instead of this application. OK, so you know, we started talking about it. And was the issue training, you know? Do they need more training? Do they need to understand it? Is it features? Where the feature is not in the system? Well, it was a combination of a lot of things, but at the end of the day was just a bad system, right? The data they wanted wasn’t there at the system was difficult to use because it was developed. You know, maybe in the late 90s or early early 2000s, right? And folks are used to using systems that are much more user friendly these days. And if you don’t meet your users where they are, they’re going to find a way around it. And even if you put them, even in the federal government where they only have really one option, well, they proved us wrong, right? They had another option and their option was to not use the system and that was a huge huge problem. So at the end of the day we wound up redesigning using a lot of user centric design to build a new system for them that function as they wanted it to redesigning their business processes. But definitely very interesting about when to shut down the system, right?
John B: Yeah, I think that’s really interesting too you know. In my experience with work, even internally at any company, if there’s a system that’s really hard to use, people are probably just going to come up with their own solutions. And then you have that other problem. We gotta bring everyone back into using one solution in the future.
Adam: And that might be a bigger problem than you know than fixing the initial system. But if you don’t have the data to back that up right, if you don’t know your users are slowly leaving your system or not completing tasks, you don’t know until it’s too late that your system is not capable of meeting the mission needs, right? Yeah, so I think that’s a great takeaway for us John, as we think about designing and developing systems here in the federal spaces. Are we actually tracking data well enough about the usability of our systems? So often I think we probably take it for granted and you know our federal customers in there in IT shops probably as well, right? We assume well, what else are they going to do? But we do know that they can create their own skunkworks projects. I know that there are a number of systems that folks have developed that sit on servers hosted in somebody’s cubicle instead of in the proper data center farm or AWS, and I’m sure you saw a lot of that at the State Department, when you were there as well.
John J: Shadow IT is a thing, Yeah, but you know, it really gets back to this idea that data is a strategic asset. And John, I think what–I really appreciate this whole conversation–’cause one thing that you’ve really kind of brought together for me is this idea that data is a strategic asset, and understanding how people work through interactions and experiences is part of that overall data, right? We can’t just look at data from one perspective. Data is going to come to us from a lot of different perspectives, and it’s important to understand both the context and then the broad application, especially when it’s aggregated so that you haven’t you gave a really good example of where well, if your utilizations your analytics drop to zero suddenly—probably that’s an indicator of a system problem rather than a usability problem, and there’s no better discussion around that then like, yes, so why didn’t you get the notification from AWS in the cloud trail that the systems down, right? So I think there’s so much value in this conversation. Adam, you put it right on with this focus on data, right? Data is how we get to better solutioning no matter the approach. And there’s a lot of different components to how we’re going to use data, but the underpinning of all this is that data, right?
John B: I think one thing that you mentioned there too made me think of this–there there’s some data that’s not captured in basic Analytics, so what we get in a lot of the analytic systems is the quantitative data. But the qualitative data where, you know those are the pieces that that’s the users feelings like, why aren’t they using this? And that’s where the user testing, maybe even like a little site survey, something like that really is important because you don’t always know why people stop using a system. It could be like a really simple problem, or it could be something really complex. Or maybe people just don’t know about it, and you wouldn’t know about that if you’re not collecting that qualitative piece as well.
John J: Yeah, so we use a number of different communications platforms, and I’ll use a very specific example–we had a platform, shan’t be named right, but they do a phenomenal job of collecting back end diagnostics. To the point where if you have a problem with that platform, I can probably tell if it was your Wi-Fi or a problem with their system, right? It’s that good. What’s really interesting is that until I flipped a switch on their configuration control panel, there was no qualitative survey of how was your interaction with the environment, and there’s a whole line of thought and study around that too. You know, there’s a whole discussion of do you use a Likert scale of one to five, or do you just thumbs up, thumbs down it, and there was a whole body of research that was done by Netflix in a few years ago. They said, like look, people are bipolar in this in this discussion. Either they like it or they don’t like it. There’s really no other analytic that matters. Given an experience, right? Either they have a positive experience or they don’t have a positive experience. People don’t tend to go. Oh yeah, that was a three. You know, those are really arbitrary numbers, so it’s really interesting to hear you kind of talk about this. and I totally agree with it. That there’s an experience part. There’s a visceral human component. And then there’s also a very quantitative measurable bits and bytes component, and you have to have those two pieces together, right?
John B: Yeah, I was just thinking of this. It made me laugh a little bit I force fence riders on a Likert scale to pick one side or another. So I I give them an even number. So a scale of like one to four. So either you had a positive experience or a negative experience. None of this in the middle because that’s not helpful for us as getting feedback too. But you’re totally right. You could even simplify–that’s just the thumbs up or thumbs down.
Adam: I think that is a fantastic place to leave it for this evening. John Bobosh of IQ solutions thank you so much for joining us tonight. John, we really appreciate your time and your insights.
John B: Thank you for having me. This has been a lot of fun.
John J: Yeah, I gotta say this has been a total thumbs up for me. Adam, I don’t know about you, and I hope if you’re listening tonight you’re going to give us a thumbs up on whatever podcasts.
John B: Five stars.
John J: Yeah, absolutely five stars all the way. If there’s nothing in between. It’s 0 or 5.
Adam: Don’t tempt them, John.