Last week I wrote about the process and reasons for adding data items to the annual return. This week I am writing about the opposite – removing data items – and the very good reason(s) why.
The Pentecostal Church of Canada (PAOC) has just released its data request for last year. It is shorter. Much shorter. How much shorter? Well, the 2022 form (and its precursors) stretched to 7 pages and asked 81 questions. The 2023 form is 2 pages and has 12 questions. That is a radical shift in the data gathering philosophy. I will say more about the data items below.
The ACC has made some changes to our data collection in the last few years, cutting two data areas from our form – but nothing on the scale. Our current data set is based on a survey from 2012 when we established what dioceses collected and what we might want to collect. In the 2000s we were collecting, or rather trying to collect a much larger data set. We were largely failing to get the returns we were looking for.
And that points us to the main reason that “less is more” when it comes to data collection. Fewer questions means that respondents are more likely to respond. More data forms are likely to be returned complete and the overall picture from the data will be more complete.
Of course the number of questions in a form is not the only factor in getting a good completion rate, but it is a key factor. When the ACC coupled appropriate questions with a good return mechanism and a person who was willing to nag the dioceses in order to get the returns, then we saw the data that we needed.
I know from correspondence with the PAOC that response rate is the motivating factor in the dramatic reduction of the data set. It should be noted that their church system requires pastors to make a return, at risk of not having their ministry credentials renewed! But even that requirement doesn’t get them the return rate they want. I very much hope that the reduction in data set will have the hoped for response.
Sacrificing items
There is another reason for not collecting data items, which may form a rationale for items which are sacrificed in order to get a better return rate. Some items are simply more important than others. That may be for a variety of reasons:
In the ACC
The simplest items to sacrifice are those for which there is no imperative. In 2016 we chose to discontinue collecting data on justice and servant ministries in the ACC. Nobody was asking us about this data and the responses form the dioceses demonstrated a wide variety of understandings of the data required. In other words we didn’t know what we were collecting or why.
In 2020 we stopped collecting data on those “engaged in formation and learning”. We would not argue that this is not significant, but again there was a wide variety of understandings of the number in the various dioceses. And of course the data from 2020 was affected by Covid!
In the PAOC for 2023
The first type of data that has disappeared is superfluous detail. If you know the church ID number then you probably don’t need the church to fill in all their contact info…. again! Similarly the churches are no being asked for the names and contact info of particular ministry team leaders.
There is no longer the repeated category of “discernment questions” which the churches were expected to fill out. It appears that these were questions which were required not so that the national office could have the answers, but so that the individual churches took time to reflect on what was happening in their community. These asked questions about whether the church was in growth or decline based on subjective assessment.
The churches are no longer being asked about their leadership development strategy. They are being asked about spiritual and missional vitality on the basis of activities which the church is running. Conversely there are no longer questions about engagement with national Pentecostal programs for mission in Canada and globally.
The national office is also not asking questions on the finances or administration of the local churches. The Pentecostal church is a fellowship with no structural oversight of the local church, so collecting administrative data may be seen as an overreach.
Finally questions on online engagement and new communities have been dropped,,alongside lookings at technology being used (and its suppliers).
I understand that some of these items will return in two years time when they run a “long form” return.
Less is more
So less is more – The shorter form and fewer questions will result in a more complete set of data for the items we are asking about. And that is the ultimate goal – a complete data set and hence a complete picture of what is happening in the church.
I believe that it is exactly those sources (churches or dioceses) who do not respond to data requests that we most need to hear from – and that are most likely to be in problems. That seemed to be the case when we looked at the charity form required by the Canada Revenue Agency. Failing churches often simply did not, or could not, return the form. I have suggested that church administrations use form completion as a metric in itself.
Comment on this edition of NumbersMatters
by joining the “Church Statistics” Facebook group,
or by emailing me – nelliot(a)national.anglican.ca
