Following the release of the second Everybody needs good neighbourhoods report, Stefan Noble, Director and Head of Research at OCSI, answered a few questions about the methodological advances that have been made since the original experimental study, and what the findings tell us about the effectiveness of resident-led neighbourhood regeneration programmes.
You can read the full report on the Local Trust website.
This report has a unique methodological approach that brings together quantitative and qualitative methods to really get a strong sense of areas that are statistically similar but have had different local conditions.
At OCSI, we used quantitative methods to identify these similar areas, and then our partners in this study, Shared Intelligence, used qualitative methods to identify how they might differ in important ways. From a methodological point of view, it’s an interesting synergy between both approaches.
First, one of the things we’re looking at much more is longitudinal data. We’re focused on core indicators that have consistent methodology over a 10-year period, which is pretty much the lifetime of the Big Local programme. We’re looking at those long-term differences in performance rather than just where they were at the current point in time.
Another key difference is that we’re looking really at the extent to which these differences are statistically significant, by developing confidence intervals around each of the indicators and doing a difference-in-differences approach.
We’re picking out not only what the trends and relative trends are looking like, but also whether those trends are actually statistically significant – when you consider the sample size of the areas in focus.
We’ve also boosted the sample size in this piece compared to the previous piece, which has enabled us to say more confidently whether these differences are significant in the statistical sense.
We’ve taken into consideration the sample sizes of each of the datasets that we’ve looked into and made sure that we’ve produced 95% confidence interval error limits around each of the data points.
Using that approach you can say with 95% confidence that any differences that you observe in the datasets are significant enough to take note of, and therefore your assertions can be a lot more confidently stated – because you are, through the magic of maths, able to determine those differences with 95% certainty. That’s the standard approach that people use to identify differences in that way.
That’s where Shared Intelligence’s qualitative work really came in. Their team did intensive research in those areas to make sure that they weren’t receiving any form of neighbourhood working. They had quite a long criteria list for what could count as that, because it can be a bit of a grey area. They also looked in the neighbouring areas around it to see whether they had any impact within that area.
There was a long tick list, a lot of conversations with various stakeholders, both council and community groups, and also a bit of on-the-ground work just to check. That’s what makes this piece quite unique – that level of qualitative work that’s gone into it to determine this. There were a lot of areas that didn’t make the cut because of that.
We identified quite a large potential – I think it was over 80 areas that we considered in total, and 29 that rigorously met the criteria for having no evidence of any neighbourhood working at all in them. So it was a multi-pronged approach with a lot of people doing desk-based work and on-the-ground work to determine this with quite a clear checklist.
Generally, the reality is that unemployment or wider worklessness has increased over the last 10 years for lots of reasons. The time points that we’re looking at were around the time of the height of the pandemic as well – and there’s impacts of austerity and other wider economic headwinds that you would expect to see leading to increases in child poverty and worklessness everywhere. But the fact that those increases are smaller in resident-led neighbourhood regeneration areas points to some of the impacts there. It may not be the only factor, but it certainly looks like they are achieving better outcomes.
It’s quite important to look at any area relative to other similar areas. That can negate some of those wider economic factors that might impact on a trend that’s been identified. It also might negate issues around people moving in and out of areas. People aren’t static, staying in one spot – so that’s why we’ve been quite careful to pitch all our analysis in relative terms rather than in absolute terms.
The rate of increase is significantly higher. When we use those statistical significance tests, you would expect some degree of error around what the rate change might look like, but the rate of change was so much faster in NBI areas.
Businesses increased by 42% (an increase of 141 units per 10,000 working-age residents) in Big Local areas over that 10-year period, compared to 22% (an increase of 104 units per 10,000) in benchmark areas. That difference is definitely significant because you might expect a bit of variation around that, but not a difference of that size.
So I think you could confidently say that both the improvements and the relative improvement are statistically significant. It’s not just a case that it’s starting from a lower base – there’s some real tangible difference there.
There’s definitely a thematic gap around health. A lot of the gaps relate to the lack of consistent data over time periods, and that’s generally a limitation that we’re working with throughout this. For example, it would have been interesting to look at levels of health and disability over time, but some of the key benefits datasets have gone through such changes over a 10-year period that you can’t get consistent measures.
Find out more about our research services
We are proud to have contributed towards the Everybody needs good neighbourhoods…
More
The Community Life Survey (CLS) is an annual survey commissioned by the…
More
We are proud to be working with Wiltshire Community Foundation to provide…
More
