The tl;dr: 

In the first of a series of resources, we explore the principles of the National Statement which guide every evaluation, market research, and social research project. There's examples, as well as a concise summary of merit.

Introduction

Ethical review, whether by a full committee or lower risk review must ensure that the activities uphold the principles of the National Statement:

• Research merit and integrity;
• Justice
• Beneficence
• Respect

When preparing a submission for ethical review, we ask you to identify how your proposed work has considered each of these principles in the design process. You can do this either by written response or by reference to documents such as your project plan. But it is important regardless to consider each of the principles and how they are met because it is an opportunity to ensure that your work is of value to all stakeholders.

This article is the first in a series of pieces designed to unpack each of the principles and support you in identifying how your project addresses them. We'll be considering each principle in order, starting with the principle of merit (integrity is a separate issue worth its own guidance).

The National Statement is clear on the primacy of this principle in the design and review process:

Unless proposed research has merit and the researchers who are to carry out the research have integrity, the involvement of human participants in the research cannot be ethically justifiable.

What is merit?

Merit in research and evaluation is complex in context and simple in principle. So simple, it can be summed up in a single quote:

[They] were so preoccupied with whether or not they could, they didn't stop to think if they should.

Ultimately, determining whether your research or evaluation has merit involves asking whether you should. It's all too easy to chase after a research or evaluation question because to the investigator it's "interesting" or "nice to know". But demonstrating merit means interrogating that research or evaluation question to decide whether the value obtained by pursuing it outweighs the potential risks that could occur. 

What guidance is there to determine merit? 

Under the National Statement (Section 1.1), evaluation/research that demonstrates merit is: 

1. justifiable by its potential benefit;
2. designed or developed using appropriate methods;
3. based on a thorough study of current literature;
4. designed to ensure that respect for participants is not compromised;
5. conducted or supervised by persons or teams with appropriate experience, qualifications and competence; and,
6. done using appropriate facilities and resources.

How do I demonstrate merit in my application? 

Given the above conditions, you can consider each of these as questions you must address in the design process, and incorporate your responses in your application. 

Is my proposed evaluation/research justifiable by its potential benefit?

The National Statement is helpful in answering this question by not just stating examples of benefits, but also advising on how to identify these:

[Potential benefits} ...may include its contribution to knowledge and understanding, to improved social welfare and individual wellbeing, and to the skill and expertise of researchers. What constitutes potential benefit and whether it justifies research often requires consultation with the relevant communities with whom the research takes place and/or that may be impacted by the research.

In a practical sense, meeting this first objective means being explicit about the objectives of the work and how by meeting these objectives there is likely to be a positive benefit to stakeholders. In practice this is often addressed in the first few pages of any research/evaluation plan. It's about answering why you are doing the work in the first place. The benefits can also be small, as long as the risks to stakeholders are smaller. 

A vital and sometimes overlooked element of this is a process of consultation with relevant stakeholders who may be involved and/or impacted by the activities. This is going to look different for each project, but a common example is the engagement of an advisory group that represents the stakeholders involved in the project to understand the benefits and risks, as well as the needs of those stakeholders. The higher the potential risk (and benefit), the more important it is to have this in place.

Is my proposed evaluation/research designed using appropriate methods? 

The next question to ask is whether the methods you are using are appropriate for achieving the aims of the work. Think of this in terms of the old saying "the right tool for the job". While it can be tempting to want to use the latest or favourite method, it's important to consider whether that method is appropriate to answer the evaluation/research questions. Note that we have said "appropriate" rather than "best". Questions can be answered using more than one method, and sometimes the "best" method is impractical in the context. What is important is demonstrating that the selected approach is likely to address the research/evaluation questions effectively and does not introduce unnecessary risks to stakeholders.

Is it based on a thorough study of current literature?

This is pretty straightforward. It's about checking that there is a justification for the work based on what has come before. It's harder to justify the benefits of research if that research has already been done, and it's also harder to justify the benefits of an approach if it is contrary to well-established approaches. It's not impossible, but in any case, justifying the benefit effectively means demonstrating how the current work builds on what has come before, whether it's by doing the work in a new context, testing the validity of assumptions made in previous work, or exploring a previously untested method in practice. In a practical sense, a research or evaluation plan with a context chapter that considers the existing research or policy landscape and considers how the proposed work fits into that landscape can help to address this question.

Now it is worth noting that there are times where there is little to no literature available (e.g. a completely new method or topic), or where the timeframe is too short to enable a thorough review (e.g. rapid response evaluation/research). This is allowed under the National Statement, but there needs to be explicit consideration of this (i.e. why it is new, or why a rapid response is needed). Again, the context chapter of a plan should address this.

Is it designed to ensure that respect for participants is not compromised?

Respect in ethical research and evaluation is so important it gets its own principle, and we're going to do a deep-dive on it in the near future. Respect is so important that it gets noted when it comes to merit. In answering this question it's worth skipping ahead in the National Statement to paragraphs 1.10 to 1.13 as these set out what respect for participants means.

In answering the question, it's about showing how all participants (and at Iris Ethics we take this to include all stakeholders) are respected through the aims of the work, the way it is carried out, and the results. It means having regard for the welfare, beliefs, privacy, and confidentiality of stakeholders, and importantly, giving credence to the idea that people should be empowered to make their own decisions and where this is not possible, protecting them as necessary. 

There are going to be multiple elements of an evaluation/research plan that demonstrate this, from the design process through to consent forms and data collection instruments. As an illustrative example, respect for participants in a survey project can be compromised by asking too many demographic-type questions (e.g. Gender, Age, Income, Ethnicity). There are sometimes good reasons to ask these, but often a set of demographic questions are thrown in at the end of a survey because it's "convenient" or a "nice to have". But unless you are using that data as part of the analysis and ensuring that privacy is respected (especially when considering the potential for re-identification), you should ask whether those questions need to be included given the risks and potential discomfort that answering those questions can cause respondents. In this scenario, a data analysis plan that sets out how each question will be analysed and addresses the research/evaluation questions can be very helpful in determining what is essential and what may compromise respect for stakeholders.

Does the team have the right qualifications and experience?

Having the right qualifications and experience in the context of the National Statement means showing that you have in place a team who collectively have the capability to deliver the proposed work ethically and effectively. This does consider to a limited extent individual qualifications and experience, but more often the team's competency is demonstrated through a grasp of the methods, the context, and the stakeholder considerations that factor into the work. It's why lived and living experience is central to so many projects, and why interpreting qualifications and experience in narrow terms such as requiring certain degrees or a publication history can exclude these perspectives.

It's okay if members of your team are learning on the job, as long as you can show there's people who can support them to learn while minimising the risk to participants. Having in place advisory groups and mentorship/coaching arrangements are great examples of this. After all, built capacity of stakeholders (including those doing the work) is a benefit of research and evaluation. And learning on the job extends to the principal investigators (a.k.a. research/evaluation leads). It is more important to demonstrate that as a team you can uphold the principles of the National Statement.

Demonstrating qualifications and experience in an application can be through the traditional CV method (or a link to your LinkedIn profile), but it can also be addressed in the research/evaluation plan by talking about the design and how the experience and knowledge of the team supports the delivery of the design.

Is it using appropriate facilities and resources?

This question is another case of being both complex and simple. Simple because in most cases it's easy to answer, complex because the right answer is going to be different for each project. For guidance, we need to go back to the conception of ethical research and evaluation having merit. 

Imagine the scenario where inappropriate facilities or resources would lead to the outcomes of a research/evaluation being significantly compromised or discredited. Then we could say that the research lacks merit because the outcomes have limited or no value. 

But what would make a resource or facility appropriate or inappropriate? That is the part that varies from project to project, and it's a great example of something where an expert determination might be helpful. As some simple examples, imagine running a focus group in a location that is inaccessible to most of the people you want to engage with, or not investing in participant recruitment materials or advertising. Either of these situations are likely to lead to highly skewed data, if any data are collected at all.

Thankfully, in most cases this is not the case because research and evaluation teams are intrinsically motivated to do credible, high-quality work. But the criterion is there both to catch those exceptions, and to give all researchers and evaluators pause for reflection.

Summary

Merit in research and evaluation deserves careful consideration. Research for research's sake isn't sufficient; we need to consider the impact of the activities on stakeholders and ensure that what we are doing is worth doing and worth doing well.

In future deep-dives, we'll look at the other principles, how these are applied, and how you can build them into your research and evaluation designs.