Barriers to Data Driven Decision Making
Last week, I started my third quarter teaching a class entitled Data Driven Government as part of the Maxwell School at Syracuse University’s Executive MPA program. The first week focuses on both the need or argument for using data in policymaking, but also covers some of the barriers faced to doing so.
The class is made up of mid-career professionals who largely work in some level of government or non-profit organizations, though a few people from the private sector have taken the class before, too. They all work in very different circumstances, and their organizations have used data in their work to varying degrees.
My experiences working for the City of Syracuse and Syracuse University gave me good perspectives on the challenges around using data. Since moving to consulting, I’ve seen similar challenges for a variety of companies that have been my clients. The both fascinating, and not at all surprising, thing I notice from my students, is that their organizations largely experience similar types of issues. This, to me, shows that while some companies or organizations are further along in having the technologies and data architectures to enable the use of data, as well as the buy in from leadership or staff, it is not just government that lags behind. There are some governments doing really creative and innovative work with data (plenty of solutions here: https://datasmart.ash.harvard.edu/). There are also multi-million dollar companies that I’ve worked with that have the same challenges counting basic things as I experienced with the City.
From my first class, here are some of the challenges and barriers they mentioned experiencing in their own work:
Access to real time and updated data — We’ve likely all experienced sending or receiving an email with an Excel spreadsheet attached that included the most recent report or associated data that mattered to an organization. These reports can be meaningful, but they also are immediately stale. Someone has done work since that spreadsheet was emailed, and the reported outcomes are now out of date. This can be a major barrier to using more data within an organization because staff may get confused about which report is most up to date, or may feel like looking at up-to-date information is critical, and thus the outdated reports are not helpful. Giving expectations around where to retrieve the most recent data and how often it will be updated is an important step. Building out the infrastructure so data is accessible from a central location and is updated regularly is not an easy process, necessarily, but it does help to overcome this barrier. This is a great read on a modern approach to making this happen (https://a16z.com/2020/10/15/the-emerging-architectures-for-modern-data-infrastructure/?utm_campaign=The+Data+Science+Roundup).
Too much data, what to do with it — Some organizations are excellent at collecting data. With the City, we had departments who had collected data for more than a century. As the “smart city” and internet of things movement has expanded and data storage has gotten much cheaper, volume of data has exploded. Having the data is a necessary step toward using data for better decision making, but it also can become overwhelming. What is important? How do we manage it? Working with stakeholders who know their day-to-day challenges can help overcome this barrier. As a data analyst, you may have the skills to tell a good story using data, but you haven’t learned about the story to tell. Connecting the experts in subject matter with the experts in analysis is critical. It is also totally appropriate to question why certain data is being collected in the first place — it may be the case that the data is not actually needed and if collecting it is overly burdensome, then it is ok to not collect it anymore.
Staff afraid to share data — Staff who oversee programs may be protective of data for a couple of reasons. First, they may feel like it is their program and their data, and do not want to let go of control so someone else can see the data. Second, the data may shine light on challenges a program or policy is experiencing and the recognition of the flaws may be threatening to the staff member. Especially in government where a lot of data is or could be made public, this is potentially particularly threatening and then a major barrier to using data more broadly in an organization. Having buy in from the organization’s executive that ultimately sharing data helps improve the organization overall and that results of an analysis won’t immediately be used to punish an employee overseeing a struggling program is important. Of course, using data to inform program performance is important, and a staff member may need to rethink or improve how they are managing that program based on analyses done, but they should not immediately feel threatened that their job is at risk.
Departments don’t track/can’t track data — We saw this constantly with the City. First, some departments just didn’t track certain data — we wanted to know how many potholes the street repair department filled each year and couldn’t get an answer. It came down to the fact that data collection wasn’t the core part of their job, pothole filling was. Determining alternative methods of data collection in this case was important — I wrote about that here. Second, sometimes the reporting tool is not available. Staff may track their work on paper, or not at all. The chosen software system may also be subpar and there is no accountability if staff don’t track their work. This talk given by Taylor Murphy and Emilie Schario discusses in part why a data team needs to think about all the ways data are collected — the larger data product — to ensure staff can adequately track the data for which they are responsible.
Funders don’t invest in data capacity building — This issue is generally specific to non-profits and in some cases government. An exciting grant is available that will enable an organization to help a lot of people. They can hire staff to implement the program and purchase the supplies that they will need. They work 40 hours a week making the program run, but then they need to file a report with the funder with data and metrics about outcomes. There is no funding to do this, there are no tools provided to do this. It becomes an additional burden for organizations who may already be operating with a tight budget. There are not always grants available to help fund the overhead around data reporting (or even building the data architecture to streamline reporting). Making more funding available for these purposes could help increase the use of data at a non-profit level.
Hard to match up data from different sources — One department formats their addresses one way, another department formats their addresses another way. Joining those two datasets together is difficult because the naming conventions are different. That means, potentially, a Code Enforcement Department know about structural deficiencies in a property, and the Fire Department knows about history of fire incidents, but combining the two for a more holistic picture of the property is non-trivial. This limits the overarching strategic potential for data in an organization. This is where the less exciting, but critically important data governance investment comes into play. Standardizing how data is collected or formatted can be key to overcoming this burden.
Data quality is questionable — This happens for a number of reasons, though in class the students talked about how difficult-to-use software systems could cause data quality issues. Maybe entering data is tedious in this specific system, so staff take shortcuts. Maybe the system wasn’t designed to match the workflows used in the organization, so staff have created workarounds that may not be used uniformly or certainly would be challenging for an analyst to understand. In my case, I remember cases where some departments would use a notes field to enter a lot of information in non-standardized ways. It would make sense to an individual staff member, or maybe to that respective department, but to anyone else, it was confusing. Another department was so frustrated with their software system, they threatened to go back to using paper.
There are many barriers to using data for better decision making in an organization. The ones listed above came from students working mostly in non-profit and government organizations. My experiences are from those kinds of organizations as well. As I’ve observed in the last few months, though, all organizations and companies deal with these barriers in some way, which is both frustrating and reassuring. It feels like we should have figured these things out already, but no one has, so you likely aren’t far behind many other organizations or companies if you are struggling to figure out the path forward.
Do these barriers exist in your organization?
