Pt 1: How to select the right monitoring system for your organization

IDinsight’s work with governments and NGO partners often involves designing or setting up performance monitoring systems. These are systems that help our clients regularly collect and analyze real-time data on their programs to figure out what’s working and where more resources are needed to achieve social impact. Performance monitoring systems are similar to what some organizations call “impact measurement” software tools, which help them regularly collect data on their work and share this data with staff and key stakeholders to help them make informed decisions. While these tools often do not measure the actual impact of an organizations’ work, which would require a counterfactual, they do provide important ways for organizations to track key outcomes amongst their target groups and understand whether their programs are being implemented effectively.

There are currently a broad suite of “off-the shelf” impact measurement software tools out there, and it can be difficult to figure out which tool is right for an organization’s needs. Typically, these tools have several capabilities: they help you collect data (i.e. through surveys or data import), analyze and visualize the data, and share back that data in the form of reports and dashboards. When selecting the right tool for your organization, one option is to buy an “off-the-shelf” software like Impact Atlas or SocialSuite. Other options are to build a custom system yourself or to bring on a technical vendor to build the system for you. If you choose to build the system yourself, this will likely involve integrating other tools like SurveyMonkey, Google Analytics, and Tableau. Bringing on a technical vendor will enable you to build a fully-customized system for your needs. In order to determine which type of solution best fits an organization’s needs, it is critical to first determine your priorities and how the tool will inform those.

Drawing on our experience helping clients set up systems to collect data and manage their impact, we developed a framework to help other organizations identify their needs, which will then inform the type of monitoring system they might require. Below are key questions you should ask before you begin exploring different solutions. The answers to these questions are the “minimum product specifications” you can use to rapidly test different tools and select the best solution for your organization.

What data do you need to collect?

The first step is to determine what data an organization needs, how it will collect data from stakeholders and the types of data it will need to collect. It’s valuable to specifically consider what kind of data staff need to understand whether they are achieving their outputs and outcomes. For example, an education non-profit might want to collect both quantitative and qualitative data from students and teachers, and possibly administrators and parents. Depending on a school’s context, it might be necessary to collect data in-person using mobile phones or tablets, or it may work to send an online survey through email. The questions below will help think through an organization’s needs around primary data collection:

• What types of data do staff need to generate program insights, make effective decisions, and fulfill reporting needs? Is it mostly qualitative? Mostly quantitative?
• Who should be surveyed and how can you best reach them? For example, through mobile surveys, SMS, email, or in-person data collection via mobile apps or tablets? Do these tools need to be able to collect and store data in offline environments?
• Do you need a tool that supports case management, or for tracking individuals’ outcomes over-time?
• How often do you need to collect data, and on what timelines? If your stakeholders will be surveyed on a range of different timelines, you may want to consider a tool that can automate survey processes.
• How much do you want to customize how your surveys look? Do you need support for advanced form features such as skip logic, data type constraints, multiple languages, audio audits, and calculated fields?

How will administrative data will flow into the system?

In addition to primary data collection, another important consideration is the systems you use to track and store administrative data and how these will integrate with the Impact Measurement and Management solution. This involves thinking about how data from current systems — such as your Customer Relationship Management (CRM) system, grant management system, enrollment and application platforms, attendance trackers — will flow into the tool. Some questions to consider are:

• What administrative data do you need to import into the tool to support analysis? This data will look different for each organization, but could include things like enrollment data, attendance data, patient records, personally identifiable information (PII), household visit records, etc.
• What platforms do you use to track and store the administrative data that needs to be imported into the system? What are the constraints around how these platforms export data? Examples of constraints include supported file formats, or the structure of the data that can be exported. Identifying these constraints will help you understand whether certain platforms will be incompatible.
• How often does the administrative data need to be imported into the tool? If you need to import data often or from many different sources, it may become necessary to automate data ingestion into the tool to increase efficiency and minimize room for error.

What data do you need to analyze?

The next step is to determine an organization’s needs for data analysis, including the work involved in preparing the data for analysis and the types of analytical techniques needed:

• What features do you need for data cleaning and quality checks? Ideally, the tool should be able to prevent or flag data anomalies like duplicates and repeat values. Depending on the type of data you are collecting, you may want the ability to flag outliers and allow users to resolve potential data quality issues while maintaining the integrity of the raw data. Some organizations may also want to have a data owner inspect incoming data before it is approved for analysis.
• What types of analysis do you need to perform for the data you’ll collect? If you’re collecting a lot of qualitative data you may want features to support sentiment analysis. If you’re collecting a lot of quantitative data, you’ll need features to support quantitative analysis, such as the ability to transpose the data, create new variables, perform calculations on raw data, and merge data with external datasets.
• What are the levels across which you need to aggregate/disaggregate data that you collect? For example, if you are a funder with a portfolio of grantees conducting surveys both with your grantees and their beneficiaries, you’ll want to analyze data both at the portfolio level as well as at the program and beneficiary-level, and be able to ensure that beneficiaries are linked to relevant programs and grantees.

What data do you need to report?

After determining an organization’s needs related to data analysis, it’s valuable to then identify what needs to be reported. This involves thinking about various audiences and which reporting outputs will best resonate with the people you need to present insights to:

• What are the different audiences that you need to share data with? Defining these audiences will help you think through the next few questions.
• What kind of reporting outputs do you want to create for these various audiences? Some common examples are data visualizations, dashboards, data views, and pivot tables.
• For data visualizations, does your organization need specific types of data visualizations to communicate data effectively to your audiences? More sophisticated visualizations (e.g. heatmaps, word clouds, network diagrams, etc.) may require more specialized software.
• For dashboards, do you need dynamic or static reporting capabilities, or both? Static dashboards provide a snapshot of trends or data over a specific time period, whereas dynamic dashboards provide real-time data while allowing users to interact with the data (e.g. through filtering) to facilitate more advanced analysis.
• How much do you want to be able to customize the style and formatting for all of your reporting outputs?
• How will you share these reporting outputs with your team and partners? Do you need to share links that provide real-time data, or just static exports?
• What constraints might your different audiences face when viewing reporting outputs? Do you need to create dashboards or reports that work well on mobile devices or in areas with poor connectivity? Do you need support for multiple languages on your reporting interfaces?

Consider your team’s capabilities

Finally, it’s important that your team has the capabilities to use the tool on an ongoing basis:

• Does someone on your team have capabilities related to configuring data ecosystems? For example, skills in database management and setting up data flows? If your team has low in-house capacity, it will be important to consider tools or custom solution vendors who can work with you closely to configure your system.
• Does someone on your team have capabilities in data collection, management and analysis? For teams with lower capacity, you should select tools or custom solution vendors that have robust client support services, as well as invest resources into training staff or engaging an external partner to manage your system on an ongoing basis.

You can use your answers to the above questions to formulate your “minimum specifications” before you begin exploring different tools. In our next blog post, we share some key considerations for determining whether buying an “off-the-shelf” solution or custom-building your own is the right approach for you.