Pt 2: Should you build or buy your new monitoring system?

After you have successfully identified what your organization is looking for in an impact measurement tool, the next step is to consider whether to buy an “off-the-shelf” solution — such as SocialSuite or Impact Atlas — or to custom-build your own. “Off-the-shelf” solutions come with a package of features that enable data collection, analysis, and reporting, whereas building your own involves hiring a technical vendor to design a bespoke system for your organization’s needs. In general, “off-the-shelf” tools are often quick to configure and work well for teams with lower technical capacity for developing data systems. Hiring a vendor to custom-build your system will take more time and require more technical expertise from your team, while giving you the flexibility to design the exact features you need. In this post, we illustrate the key considerations to help you determine whether buying an “off-the-shelf” solution or custom-building your own is the right approach for you.

1. Time

The first thing to consider is how soon you need your system to be set up. Most “off-the-shelf” impact measurement tools can typically be configured in a short amount of time — you can expect to have a functional system set up within a few days or weeks from purchase.

On the other hand, custom-building a system with a technical vendor usually involves a much lengthier process. Typically, these vendors will start off with a design process to understand what data an organization needs, how it will collect data, and how this data needs to be analyzed and reported, as well as the roles that staff members will play to implement this work. The process of building the tool can take anywhere from a few months to half a year (or more), and often involves multiple iterations before you reach the final product. The length of this process will depend on the number of iterations you need, as well as the capabilities of the vendor.

2. Cost

Determining whether an “off-the-shelf” or “custom-built” solution is more cost-effective for your organization will depend on your minimum specifications and which key features you may be looking for. Platforms will often have monthly or annual subscription licensing models, which come with individual licenses, team licenses, or organizational licenses. Team and organizational licenses will set limits for how many individuals can use the system. In addition, subscription licensing options will include support hours for the vendor to help you configure the tool and train your staff to use it; otherwise, support hours are offered at an additional cost.

Hiring a technical vendor to custom-build your system will involve a larger up-front investment. You can also expect additional costs for support hours as your staff start to use the system, as well as additional costs for developing new features down the road where the need arises. However, by custom-building your system, you have the freedom to choose the number of individuals who can use the system without worrying about license purchases. This can be particularly valuable if the majority of your staff need to collect, use, or analyze data on a regular basis.

3. Internal capacity related to data systems

“Off-the-shelf” tools are typically more suitable for teams with minimal experience managing data systems. Since these tools are already built, you will have a chance to see what the software looks like and test it through a trial prior to making your selection. In addition, “off-the-shelf” tools often make it easy for staff without programming skills to re-configure the tool over time, particularly if they come with support services and training resources. This will be valuable when your organization’s data collection — analysis, or reporting needs change — for example if you want to collect data on a different set of indicators or integrate a series of new data sources.

Your team will need some degree of technical knowledge to effectively work with a vendor building a custom solution for your organization. This is because you will need to assess the quality of vendors prior to making your selection, as well as manage the vendor throughout the design and building process. It is also important that staff can proactively learn the skills required to use the system, particularly because vendors may not be as specialized as companies offering “off-the-shelf” solutions in client support and training services.

4. Features

“Off-the-shelf” solutions are typically designed to satisfy a broad set of use-cases, but may not represent the most elegant design solution for these use-cases. If you are seeking to set up a more advanced system or have specific preferences around how you want users to interact with it, you may find that you will need to compromise on some features or aspects of usability. It is particularly important to consider whether the built-in workflows (how users interact with the software tool) meet your needs, as well as whether you can request new features or set up integrations with other tools. For example, an “off-the-shelf” solution built on Salesforce will come with many options for integrations with other tools, whereas a custom software solution may not.

If you have strong preferences or need a specific set of advanced features or tailored workflows, then you should consider the custom-building approach. This is particularly true if you need to: a) collect and analyze large volumes of data (like high-frequency data, real-time data streams, or sensor data) that may require specialized data storage and processing tools, b) automate data quality checks during data ingestion and analysis, c) perform advanced analysis such as through econometric methods or predictive algorithms, d) set up highly customized workflows for your staff, or e) have specific aesthetic or content requirements (like advanced mapping functionalities) for data reports and visualizations. All of these features would typically only be possible with a custom solution. Of course, custom-building a solution involves a certain degree of risk too; you will have to work with a capable vendor to build features that meet your needs.

5. Data security

Finally, it is important to consider the data security features of “off-the-shelf” versus custom-built solutions. If your organization has specific requirements for where servers must be located, “off-the-shelf” solutions may not offer much flexibility here. Specialized features for data security — such as the ability to tightly control permissions and encrypt or de-identify Personally Identifiable Information (or PII) are also not guaranteed with these solutions.

In our next post, we share our experience with testing several “off-the-shelf” Impact Measurement tools for a client, The Incubation Network, as part of our work designing their performancing monitoring system.