At a recent conference in Montreal, individuals and organizations involved in access to justice across the country gathered to present current research, share initiatives, and build a strong foundation of knowledge. The conference was grounded in a people-centred approach to access to justice, shaped in part by the OECD Framework and Good Practice Principles for People-Centred Justice.[1] Within this approach there is a common understanding that if we are to improve access to justice we need to include, and in fact, elevate, the voices of those attempting to access justice, both in terms of their challenges, and their experiences. This is neither a radical nor new realization (in fact the NSRLP was founded in 2013 on research into SRLs’ experiences). Over the past decade, many researchers and access to justice activists have called for research that prioritizes the justice system users’ voice. However, despite this call, user-based research is not undertaken as often as it should be.
We continue to ask lawyers and judges and court clerks what they think would improve access to justice or how we ought to operationalize initiatives, which may provide useful insight, but remains, at best, a partial response to the access to justice crisis. Moreover, we struggle to admit that what the legal profession believes will improve access to justice might not be what justice system users need, want, or find practical. Justice system professionals see the problem from a very particular perspective: that of insiders who are trained and experienced in the operation of the law and the legal system as it is currently constituted. Additionally, while the legal profession is tasked with serving the public interest, lawyers are also generating a livelihood. Thus, without appearing too controversial, it is not difficult to see how the profession’s vision of what effective access might look like might, in some instances, be different from the average citizen’s perspective. This is not to say that justice system insiders should be excluded from the conversation about access to justice, or their perspectives deprioritized, but rather that they should not be the only voices heard.
However, beyond the issue of whose perspective is currently sought is a broader concern about how access to justice research is undertaken.
In recent years, we have seen a growth of legal research projects seeking to understand the average citizen’s legal needs, as well as where they turn to secure assistance with their legal problem. Thanks to these types of research projects we now have a greater sense of the nature of people’s legal problems and how they undertake addressing them (or not). To the latter point, this research has also highlighted the fact that many individuals either ‘lump’ their legal problems before even attempting to address them, or give up at some stage prior to resolution.
So, these are the things we know about justice system users. But what do we know about what is working for individuals trying to resolve their legal problems? In our modern age of instant information, we may assume that there are a significant number of individuals who understand that they have a legal issue, and seek to resolve it through a variety of options – we even have a sense of what those options entail. But we do not have a good sense of whether the access to justice resources or programs accessed by such individuals actually assist them with their legal issue. Understanding what works and for whom is crucial to a people-centred approach to access to justice. If we are to develop good resources, initiatives, and pathways, then we need to ask if these materials are indeed helping the people they are meant to serve.
Addressing this question is more complicated than it appears for two reasons: the difficulty of measuring outcomes, and the current perception of what constitutes rigorous and reliable research.
With respect to the first concern, there remains a tension in access to justice research over the measuring of legal outcomes. A variety of factors make it difficult to evaluate outcomes generated by an access to justice initiative. Essentially, the arguments are that there may be too many ‘unknowns’ to isolate the impact of the program, and an ethical concern in controlling for the impact of assistance (in effect the creation of a control group who will not be assisted so that the effectiveness of a program or initiative can be measured). Thus, this type of research is often relegated to customer-satisfaction surveys in which individuals are asked whether they were happy with the outcome. Alternatively, outcome-based research frames achievable outcomes in extremely narrow and sometimes unhelpful ways. The problems these pose are that individuals who feel as if they are drowning will be positive about any life-saving device, no matter how effective, and/or the impact of the program is not reflected in the articulation of the measurable outcome. Thus, self-reported customer satisfaction survey results in this context provide limited meaningful insight into the efficacy of an access to justice initiative.
The second concern relates to the nature of the research undertaken: namely, quantitative versus qualitative research. The tension between these two broad approaches takes on particular weight in the context of access to justice research. Quantitative methodologies are perceived as producing ‘hard data,’ using more rigorous methods, whereas qualitative methodologies invoke more skepticism about the nature of the data generated, what is learned from it, and how it might be deployed. Access to justice researchers continue to debate the rigour of these methodologies (as they should). However, this debate can sometimes result in a form of paralysis, whereby certain types of research are not undertaken, or discounted when they are undertaken. While both methodologies have something to contribute, the reality of large-scale quantitative research is that it may take significant time and resources to design and implement, and as such, is really only accessible to large institutional players who are able to effect access to surveyable populations. Moreover, there are times when it would be helpful if we were focused on understanding individuals’ experiences, rather than measuring the occurrence of a phenomenon; this is particularly the case when outcomes can have both positive and negative elements, and are, as such, complex. In this sense, quantitative research does not allow for a great deal of nuance. This is not to suggest that we do not require quantitative measurements in access to justice research – gathering updated data on the percentage of SRLs in courts is a good example of a much-needed measurement. But a focus on research projects that narrowly conceptualizes ‘rigour’ and does not include qualitative work such as interview and observational research means we are missing out on important insights into the impact of access to justice programming on the very individuals making use of such programming. The result is an absence of data on justice system users’ direct experiences with access to justice initiatives and resources, and lost opportunities to learn and improve on those programs that are working. This is significant when we also remember that many access to justice projects or initiatives are operating on very limited resources, where knowing what works and what doesn’t work becomes urgent.
In order to move access to justice research forward in a truly practical way, researchers, policy makers, and justice system insiders need to embrace the expansion of methodologies that are consistent with a people-centred approach. This is not a discounting of quantitative methodologies, but an acknowledgement that well-designed qualitative research methodologies deployed in the sociological and anthropological context provide a rich canvass of individuals’ experiences and perceptions. This type of research can offer important insight into various aspects of our social world – and these research methods can produce data far beyond that of a simplistic customer satisfaction survey. Where quantitative methodologies can produce data that shapes the scope of the problem, qualitative methodologies with their own internal dynamics allow researchers to better understand individuals’ experiences, and how such experiences impact their lives in both tangible and intangible ways. Moreover, research that explores individuals’ actual experience with a program or process may also identify consequences that are reflective of insiders’ blind spots. While this type of research engages a more complex and nuanced genre of data, it is no less rigorous than its quantitative cousin. Moreover, undertaking qualitative research within established ethical parameters allows us to prioritize the voices of those seeking access to justice, consistent with a people-centred focus.
The challenge for the legal establishment is accepting that rigour, within the context of qualitative, experienced-based research, may look different from its quantitative counterpart. We must be willing to live and grapple with complexity. If we do not, our understanding of the true scope of the access to justice landscape will remain surface level, at best.
______________
[1] See OECD (2021), OECD Framework and Good Practice Principles for People-Centred Justice, OECD Publishing, Paris, https://doi.org/10.1787/cdc3bde7-en.
The post A Little Rigour Can Go a Long Way… Reflecting on Research Methods in Access to Justice appeared first on Slaw.