Post by Adena Rissman
People want conservation programs, such as streambank protection, wildlife habitat restoration, and conservation easements, to work well and produce results. But the most common information about their performance—for example, how much money was spent or how many acres were managed—doesn’t really tell us what kind of progress the programs are making.
Given the broader movement by funders, governments, and scientists to improve the effectiveness and efficiency of conservation efforts by tracking performance metrics, what can we learn from the information conservation organizations provide? What measures—other than bucks and acres—are conservation organizations using to reassure their funders, bosses, and stakeholders that their work is actually working?
According to a study I recently co-authored, some organizations measure and report much more information about their performance than others. Unfortunately, performance measures don’t tell us that much about program effectiveness, unless they are combined with in-depth program evaluations.
My co-author, Bob Smail,* and I compared how four types of conservation organizations measure and report on the performance of their private-land conservation programs. The organizations, all of which have offices in Wisconsin, included two at the national level, the Forest Stewardship Council and the US Fish and Wildlife Service’s Partners for Fish and Wildlife, and two that work at a local level, county-level land and water conservation departments and nonprofit land trusts.
We found the amount and type of reported information varied among the organizations. The county conservation departments reported the most types of performance information, while local land trusts reported the fewest. In addition to information on dollars spent and acres conserved, the county conservation departments and the Forest Stewardship Council reported outcome indicators, such as how well they adhered to performance standards set by state agencies and multi-stakeholder groups, respectively.
Of the four organizations, only the county conservation departments tracked and reported the effects of their conservation efforts—even so, these were typically modeled, or estimated, effects; only rarely were they measured effects as compared to a piece of land without conservation measures.
Organizations must report their performance to other organizations within their network. For example, a land trust will inform its donors and members about their efforts, and a local government department must report their efforts to other departments and agencies at varying governmental levels, as well as multi-stakeholder boards.
Our study suggests that these governance networks influence the pattern and type of performance reporting. Figure 1, for example, shows the reporting network for the county conservation departments, and Figure 2 shows the network for the land trusts (click to enlarge figures). Here is a complete diagram of all four reporting networks.
The county conservation departments, for instance, report to many organizations, some of which have different goals—e.g., environmental versus agricultural industry representatives, who may seek additional performance information as they negotiate over the details of managing and implementing conservation programs. Also, county conservation departments must demonstrate compliance with water quality laws and regulations to federal and state agencies, hence their reliance on performance standards and quantitative models, which are common tools for ensuring clean water. Satisfying such diverse information needs may be one reason why the county departments reported the most information of the organizations we examined.
Overall, we found that most performance information actually cannot be used to evaluate effectiveness, because there is no way to tell what would have happened if the organization had not implemented the program.
So, for example, if water quality in a particular river declines, we can’t know if it’s because conservation practices were not effective where they were installed, or if they were effective, but there weren’t enough of them to make up for increasing nutrient pollution from other sources.
This creates a problem of “indeterminate causality,” or not knowing the cause and effect, which means the performance information cannot be used to evaluate how well the program is working. This is important because many people are working to reform government and nongovernmental programs by requiring additional performance measurement. So while performance measures can offer important information to help funders, managers, and stakeholders, unfortunately much of this information doesn’t provide direct evidence of a program’s efficacy.
Funders, agency leaders, multi-stakeholder groups, and other authorities can play important roles in encouraging conservation organizations to produce more and better performance measures. However, answering how well a conservation program is working may not require just “more data,” as is commonly suggested. In some cases, in-depth evaluations are needed to help uncover what is enabling or inhibiting program effectiveness. We also caution against performance management systems that count only what can be easily measured, like bucks and acres.
*Bob Smail is a former UW-Madison graduate student who now works for the Wisconsin Department of Natural Resources.