This section provides an in-depth discussion of common goals among refugee rights organizations, methods of monitoring and evaluating these goals, as well as complications that may arise.
Goal one: X number of refugee men, women, boys, and girls are better able to make informed decisions about their lives because they received quality information, advice, or representation through our legal services.
About this goal: This goal does not focus on any specific legal service. This goal is about the inherent value in refugee legal aid, regardless of whether or not a case has a successful outcome; it assumes that legal aid in and of itself will provide refugees with the information they need to better make decisions about their lives. For more information on conceptualizing the inherent value of legal aid, please see the Namati report on the success of legal aid organizations worldwide in achieving greater access to refugee rights through legal empowerment strategies.
What monitoring mechanisms can be used?
(1) Measure the number of clients reached.
You will need to configure a system to collect data about the number of clients served. Ideally, this information will be input to a database that allows you to disaggregate by sex, gender, and age. Being able to look at each subset of the population will be very useful later in the evaluation process to teach you about who you’re serving, and to provide insight into how to improve services to specifically reach underserved populations.
Additionally, think about including client dependents in your measure of the number of clients served. Typically legal aid offices consider immediate family members — those that will directly benefit from the information the client is receiving — as also served by the legal service. Including dependents in your measurements will give you a more accurate understanding of the true number of people reached by your legal aid program.
To better understand your workload, look at the total number of services rendered, in addition to looking at the total number of clients served. Legal services organizations often provide multiple services to a single family. Measuring by the number of services, separately from the number of clients, will give legal aid managers better insight into the actual workload.
(2) Measure the quality of legal services provided.
There are many ways to think about what it means to provide quality legal services. Each organization may find different monitoring mechanisms beneficial.
What are some methods of measuring quality of services?
Review Quality in Legal Aid
(1) Independent Review
An experienced practitioner from outside the organization reviews a case brief. It may be preferable to utilize this method less frequently, perhaps once per year. Issues of confidentiality may make this a less preferred method.
(2) Second Person Review
A supervisor or a peer reviews a colleague’s files.
This significantly improves the quality of work. If the organization’s workload grows to a point where it is impossible for a supervisor to review every file, consider permitting “experienced” peers to conduct reviews. During periods where workload grows fast, quality is often put at risk. If peer reviews are undesirable or impossible, consider weekly case reviews, where newly opened cases are discussed among the legal team.
(3) Client Satisfaction
Learn from clientele whether they believe legal aid services have been effective, and areas that could use improvement, through surveys, focus groups, complaint/suggestion boxes, or other feedback mechanisms.
This is an imperfect measure of quality. For example, you may provide accurate legal information but because the client was unhappy with the outcome, client satisfaction may appear unduly low. Nevertheless, client feedback can provide useful insight into how to adjust services.
Additional points of measurement that should form part of all quality measurements include:
(4) Gender & Age Sensitivity; Sensitivity to other vulnerable populations – Gender and age should be considered in all quality measurements. Assess whether the organization is successfully reaching different subsets of the population; assess men/women/boys/girls separately; assess the organization’s impact in subsets of the population with vulnerable characteristics; for example, torture survivors; people with disabilities; elderly; female head of households; minors.
(5) Adherence to Ethical Standards
Strict adherence to the Nairobi Code should be monitored.
(6) Adherence to the Legal Services Agreement
Strict adherence to the Legal Services Agreement should be monitored.
Why should numbers and quality be considered together?
This goal is about getting information to refugees that will help them to make informed decisions about their lives. While you should concern yourself with educating the highest number of refugees, this absolutely cannot be at the expense of quality. Link to PDF describing Project Management Triangle
Goal Two: Through legal services, an increased number of refugees are able to access refugee status so that they may enjoy their full range of rights under international law.
About this goal: This is an example of a substantive goal with a specific end goal. Here, legal aid is utilized to help an increased number of refugees to access legal status. It makes the assumption that legal status will assist refugees in accessing their rights. However, this is not always accurate in all contexts. In some cases, refugees that obtain status are concurrently safeguarded against deportation and detention, and granted access to the court system and to work rights. In other cases, refugees are not granted full Convention rights upon gaining status. Be cautious about setting such a goal before testing whether this is true in practice in your country. Perhaps your target population is routinely rejected, or presents a particular challenge to decisionmakers, which will not reflect the quality of your legal services but rather the situation in your region.
Compare case outcomes (success rates) at the legal aid office to the national rate of success. The best way to find national success rates is through the UNHCR Global Trends Report (see See Tab 8), which is released annually. Note that this comparison is an imperfect measurement of impact, because there is no way to identify the strength or weakness of the claims considered in arriving at that baseline. It nonetheless provides information that can teach you something about your work.
This information should be tracked in your database, or through some other collection mechanism (a spreadsheet, for example), with client information updated regularly. To track progress in each case, you should create a clear and agreed upon AAT Case Guide . A case guide is a document used to define success by case type (RSD admissibility, RSD eligibility, wage withholding, etc.). The case guide will define the way in which each case outcome should be monitored.
There are four general categories of case outcomes:
3) Not applicable
A case outcome is categorized as not applicable if, for example, a client decided to withdraw her case, or perhaps because the legal aid office lost contact with the client.
4) Other Favorable Outcome
A case outcome is categorized as other favorable outcome if the case was unsuccessful in one respect, but successful in some other respect. For example, in a land dispute claim, if a client only gets a portion of their land, this case could be categorized as other favorable outcome. The purpose for this category is to be able to organize cases where there is not a clear successful or unsuccessful outcome. For each case that falls into this category, it is essential that you clearly define what makes that case categorized as other favorable outcome.
It may be important to point out to donors how your organization conducts follow up, to demonstrate that you have an established procedure in place that protects quality. You should feel comfortable explaining that case outcomes are imperfect measurements of success, and are better served to inform strategies than to gauge impact.
Monitoring mechanisms for RSD case outcomes
It is helpful to track various RSD instances separately. Once a client has moved from first instance (RSD admissibility) to RSD eligibility, open a new case and track it separately. If your client is not granted status and you decide to appeal, open a new case and track that separately. This mechanism is used because clients arrive at legal aid offices at different stages in the RSD process. To efficiently track resources, it is important to understand at which stage in the RSD process the client solicited your services. Additionally, utilizing this mechanism will help you to understand the national process—whether admissibility is plenarily granted but eligibility is not, or vice versa. This will help your organization to understand where to spend legal aid resources, to focus legal aid efforts on the processes that require increased intervention. Monitoring will help you to learn these lessons and therefore inform your strategy.
Perhaps this will demonstrate that your organization’s rate of success is lower than the national average. This may indicate poor quality of legal aid. This demonstrates the importance of clear quality measurements—to ensure that quality is protected. Alternatively, it could indicate that your client population poses a particular challenge to decision makers and are thus routinely rejected. Perhaps you focused on this demographic or population precisely because of these challenges. Monitoring quality, case outcomes, and documenting external developments will help you to understand the root cause of challenges unique to your target population.
Challenges presented by this monitoring mechanism
Comparative statistics rarely categorize cases in the same way that legal aid organizations do. For example, first instance and appeals are combined, complicating an organization’s ability to make apt comparisons. Additionally, the Global Trends Report release dates are delayed, which means this comparison can only be analyzed in hindsight.
Case outcome monitoring requires clear follow-up processes. In many contexts, RSD decision makers release the case outcome directly to the refugee and it can be difficult to contact refugees after this point. Continued efforts to contact refugees can be resource intensive. However, such contact is required under the legal agreement. To mitigate this, be clear upon signing the legal agreement that the client has an obligation to contact you. Set clear guidelines within your organization to standardize the maximum number of client contact attempts before closing a case without a known outcome, in order to preserve resources.
Goal Three: Through legal services, refugees are increasingly able to access work opportunities and are increasingly protected from workplace violations.
About this goal: Similar to Goal #2, this is an example of a substantive aim with a specific end goal pursued through legal services. This is actually a sequential goal-combined in this section for organizational purposes: increased access to work and increased protection in the workplace. The first part of this goal–increased access to work opportunities–could be achieved as an effect of a successful RSD case, or through alternative means of achieving legal work permits (such as the peasant permit in Tanzania). The second portion of this goal–protection from workplace violations–is typically achieved by addressing wage withholding, workplace discrimination, and sexual abuse in the workplace, among other forms of exploitation in the workplace.
This goal should be conceptualized temporally—the first goal is centered on gaining access, and the second becomes a focus after this first portion has been secured, to ensure that work is dignified and legal.
Mechanisms for monitoring increased access to refugee work rights
Similar caveats and imperfect measurement capabilities present in goal #2 are present in this goal as well–nature of caseload, or of the political situation–which may affect your success rate. To mitigate this, you should implement other monitoring mechanisms to ensure quality of services before proceeding to evaluate the impact of contextual caveats.
To properly measure the impact of your organization’s work to increase refugees’ access to work rights, you will need to collect certain data from clients when you open the case, updating this information in your database over time. For example, upon opening the client’s case, you will find it helpful to ask a set of questions, i.e. What is your job? What is your salary?, and then ask again in six months.
Without an available national average baseline, this goal must be measured by improvements in your own ability to address work rights issues. You may decide to gather information for a full year, or perhaps even two to three years for big picture goals, before you can formulate a meaningful baseline to be used to evaluate your impact.