4. Evaluation Findings
- 4.1.1 Assumptions
Underlying the IRB's design of the new refugee determination system are core assumptions about the volume and nature of the claims, time to conduct key activities, and availability of members. The evaluation found that the assumptions underestimated the resources required to deliver the new system according to its legislated timelines and operational targets.
The volume of claims in the first two fiscal years has been substantially lower than the projection of 22,500 claims annually, but it is on the rise (up just over 25% for 2014‑15 to 13,216 claims). Key informants (external in particular) noted that the lower intake was to be expected; whenever there is refugee system reform, potential claimants will refrain from filing claims until they see how the system is working. The first two fiscal years indicate a system already experiencing some strain, as pending claims after fiscal year 2013‑14 were 2,217. Rather than reduce the pending claim inventory, 2014‑15 added 472 claims, to result in 2,689 claims pending at the end of two fiscal years.
Another key assumption is related to the complexity of the claims. The assumption was that 10% of claims would be “complex” and 90% of claims would be “regular.” This assumption affected the amount of time allocated to each type of claim, with the difference being almost two hours. To give an example of the potential effect of this assumption, if 30% of 2014‑15 claims decided on merit were complex, rather than the projected 10%, about 4,435 more hours of member time would be required to finalize these claims. This is the equivalent of about 3.5 FTEs (approximately 5% of the total FTE complement in 2014‑15).
Another assumption related to the complexity of the claims was that the percentage of total claims decided on merit would be 30% DCO claims and 70% non‑DCO claims. In reality, less than 10% of claims have been DCO claims. DCO claims have expedited timelines. Underlying the shorter timelines was also the belief that these claims could be ready for a hearing in those timelines and would require shorter hearings. Instead, approximately 90% of claims are non‑DCO claims, which are expected to have longer processing times.
Claims are more complicated than predicted as a high percentage of claimants have counsel, there are more ministerial interventions, and the claims have more documentation. These factors mean that RPD members have needed more time to prepare for the hearing and to deliver reasons. Both of these consequences have resource implications for the new refugee determination system.
Recommendation #1: The ability of the IRB to determine its human resource needs and set reasonable finalization targets for the RPD and RAD is complicated given the apparent differences between the working assumptions when the system was being developed and the reported experience two years later. The IRB should consider a limited exercise where members would track their time usage. That information could be used to update resource needs and inform training activities or other decisions related to how to distribute human resources to increase efficiency.
The evaluation found strains in the delivery of the new system. While the volume of cases is lower than anticipated, the RPD and the RAD are not achieving their finalization targets and, for the RPD in particular, an inventory of new claims is growing. The evaluation found that this is due to the challenges in keeping up with the pace and the fluctuations in intake, as well as accommodating the secondary intake of claims returned by the RAD or Federal Court, and applications for vacation or cessation.
Stress and burnout experienced by members was a recurring theme in the evaluation interviews given the pace and volume of the work. For RPD members, the main suggestion to improve efficiency and their work environment was to receive more administrative and adjudicative support. For RAD members, the main suggestion to improve their efficiency was the provision of RPD hearing transcripts.
Recommendation #2: The previous system had more extra‑hearing support for the decision‑making process by assisting members with triaging and preparing files for hearings. The IRB should consider a pilot project to determine if the provision of similar supports would be a cost-effective method of increasing the efficiency of the system in finalizing claims.
Recommendation #3: The IRB should consider a pilot project on the provision of RPD transcripts to the RAD to assess the efficiency gains the availability of transcripts would provide.
Key informants questioned the RPD's current adjudicative approach of deciding cases on one determinative issue in order to streamline and increase the efficiency of decision-making. From the perspective of RPD key informants, this adjudicative approach — while intended to save time and be more efficient — has made their decisions more vulnerable to being set aside or referred back to the RPD by the RAD. Most RAD interviewees also questioned whether the adjudicative approach supports the whole‑of‑board approach. The RAD review cannot focus solely on the one issue but must consider all issues raised on appeal. From the perspective of RAD key informants, the result of the RPD adjudicative approach is less efficient decision‑making, as the RAD has to spend more time on its decision and often has to refer the matter back to the RPD for it to consider other issues.
Recommendation #4: The RPD should review this adjudicative approach and whether it supports a more efficient refugee determination system by considering the approach from the perspective of the entire system and not just the RPD.
- 4.1.2 Impacts of other involved departments/agencies
The evaluation evidence identified several factors that are affecting the efficiency of the new system.
- Scheduling hearings - While having the Citizenship and Immigration Canada (CIC) and the Canada Border Services Agency (CBSA) schedule the initial hearing was intended to help the system meet the new legislated timelines, the IRB has found the system challenging. The hearing booking tool has less functionality than hoped, as it does not interface with NOVA, the IRB's case management system, which would enable more strategic booking of hearings. The result is that the RPD cannot easily take advantage of the efficiencies available through geographic specialization nor can it manage effectively and flexibly members' workload.
- Front‑end security screening (FESS) - The delays due to pending FESS are down substantially in the second full year of the new system's operations. Both the number of claims delayed and the length of the delay have declined. However, in 2014‑15 over 700 claims were delayed due to a pending FESS and still experienced about a four‑month delay on average.
- Information from portfolio organizations - The ability of RPD members to efficiently and accurately assess claims is supported by having all the available information. Difficulties in getting the following from CIC and CBSA were identified as having an impact on efficient decision‑making: identity documents, temporary resident visa applications, and responses to information requests.
- Fluctuating Federal Court jurisprudence - Key informants reported that Federal Court decisions on the appropriate standard of review for the RAD have had an operational impact by slowing down RAD decision-making.
Recommendation #5: The IRB, CIC, and the CBSA should engage in discussions on how to address issues related to scheduling hearings, FESS delays, and information‑sharing. In particular, the discussions should consider the development of a more robust/flexible scheduling tool, improvements to scheduling practices that gives the RPD more control, and more timely and responsive information-sharing. Any such discussions must be conducted in a manner that respects the institutional independence of the IRB.
- 4.1.3 Program support services
Prior to the reforms, the IRB had a single registry that served all divisions. In the reforms, the IRB established a separate registry for the RPD. The rationale for the separate RPD Registry was to support the RPD's ability to deliver on its legislated commitments. The belief was that with the new, aggressive timelines, the RPD needed to have more streamlined interactions and reporting with its registry services.
The Registry and Regional Support Services (RRSS) branch includes Common Services, which handles services such as mail, interpretation, transcriptions or recordings, and reception for all of the divisions, including the RPD. It also provides registry services for the RAD, Immigration Division and Immigration Appeal Division. This reporting structure has a clear separation of management of RAD members and registry staff.
Key informant opinion was mixed on whether the separate RPD Registry's benefits outweigh its challenges. The division largely rested on whether the key informants worked within the RPD or not.
RPD management expressed strong support for the separate RPD Registry and believes that it has generally achieved the objectives underlying the rationale for its creation. They commented on the sense of teamwork with the registry that exists now. They believe that the new structure has increased their ability to quickly and effectively address issues that arise concerning how registry processes might affect the work of the RPD.
Key informants, primarily from areas outside the RPD, pointed to challenges created by the separate RPD Registry. Their comments primarily concerned the effect of the separate registries on the potential for a “whole‑of‑board approach”. Under the single registry, staff could be reassigned across teams to respond to work demands, while with separate registries staff mobility is considered more limited. Furthermore, separate registries within the IRB create duplication of work at the organizational level (e.g., human resources, financial management) and each requires its own management structure, which adds to the cost of the IRB registry function.Note 6
Recommendation #6: For the RPD Registry, the main area for improvement is clarifying the roles for staff and more consistent application of those roles. This point is related to the members' need for more support. As the members seek assistance, they tend to expand the roles of registry staff. The IRB should consider as part of Recommendation #2, whether there should be a greater redesign of Registry roles or whether some other method is more suited to providing members with more administrative supports.
Key informants generally provided positive comments on the quality of Legal Services advice and reasons review. A concern expressed by key informants was whether Legal Services has sufficiently adjusted to the focus on efficiency and the need for members to meet tight timelines for making decisions. RPD members are now expected to make more oral decisions and more decisions that are based on one determinative issue. According to some key informants, Legal Services has not made a corresponding shift in its approach and considered how to support members in becoming more proficient at streamlined decisions that will withstand RAD and Federal Court scrutiny.
Many key informants praised the Research Directorate's country of origin information (COI) work producing and updating the National Documentations Packages (NDPs), which were noted as having an international reputation and being used by other countries. A few key informants suggested improvements to the NDPs. In particular, the length of some country reports were considered “unwieldy,” particularly for members new to handling a refugee claim from the country, who must digest potentially hundreds of pages to prepare for a hearing.
In the fast‑paced environment of the new system, members need timely information when making decisions. The evaluation findings indicate that the Research Directorate generally provides high quality information in its various research products. However, the Directorate is experiencing capacity issues as it adjusts to the timelines of the new system and the changing jurisprudential requirements placed on members.
Recommendation #7: The ability of the Research Directorate to respect the RPD timeline pressure requires the Directorate having the capacity to handle the requests in the tight timelines. In combination with recommendation #2, the IRB should consider undertaking an examination of what needs to be done for the Research Directorate to deliver more timely responses with more focused content to the RPD.
The IRB has captured substantial data on the work of the RPD and RAD. The Standards, Analysis, and Monitoring unit (SAM) provides a national picture of key metrics. The SAM reports were considered to provide valuable information, although there was some suggestion that they could include more analysis and more regional reporting. The RPD and RAD regional analysts carry out many tasks that vary by region and division reflecting uncertainty among key informants over how the analyst group can best be deployed to support the divisions.
Recommendation #8: The analyst role is reportedly becoming clearer. However, their duties appear to be quite broad and sometimes tailored more to suit the individual skill set of the analyst than to be based on a strategic decision about what work analysts should do. The IRB should review how the IRB's analytical function can be restructured to enhance its effectiveness.
The evaluation found much frustration among key informants (primarily among RPD members, but also among a few RAD members) related to monitoring member performance, as there is the perception that the performance of the system is based solely on individual member performance. According to key informants, the current performance metrics for members that focus on targets such as the number of hearings and decisions per week provide an incomplete picture of performance. The metrics do not take into account factors such as the complexity of cases, the need to adjourn or postpone a hearing due to factors outside the member's control, or the quality of the member's job performance.
Key informants questioned the individual performance targets. The original targets were not realistic, and while they were subsequently revised, they were still difficult to justify to members given the limited information available on how they were set.
Recommendation #9: The individual performance targets should be set in consultation with members and based on evidence of what is reasonable in the current system.
The IRB has created a number of products to inform parties about the new system as well as their rights and obligations. The evaluation findings indicate that parties are generally aware of these products, have used them, and have found them helpful, in particular: research products, the Chairperson's Guidelines, IRB guides, IRB forms, the Claimant's Important Instructions, and practice notices. The website postings of persuasive decisions or other decisions of public interest are used less than the other products, largely because few decisions are posted and the site does not appear updated. The Ready Tours and Information Sessions have been positively received, but there was little information on the level of uptake.
Recommendation #10: The RPD and the RAD should consider whether they should be more active in identifying and posting persuasive decisions or other decisions of public interest on their website. In addition, the RPD and the RAD should evaluate the Ready Tours and Information sessions to obtain a better idea of the uptake and satisfaction with these activities.
The evaluation found that, generally speaking, authorized representatives found the Basis of Claim (BOC) form to be easy to complete and to complement other forms that claimants complete during their eligibility interview. However, when asked whether the form allowed claimants to express their personal information in a clear and logical fashion, approximately one‑fifth of respondents strongly disagreed. Key informant comments reflected this sentiment, noting that the BOC form did not allow for a clear narrative and that claimants often simply attached a separate chronological narrative to the form.
Recommendation #11: The RPD should review the BOC form to determine if improvements are needed to allow for a clear narrative of the information relevant to the claim.
Initial RPD hearings are typically on time though there was a spike in delays in quarter 4 of 2014‑15. Since the reforms, over 80% of claims have had their initial hearings scheduled within the time limits. However, the results also show the tensions in the system: The timeliness of the scheduling of DCO hearings has improved while that of non-DCO hearings has declined. When a hearing cannot proceed on its scheduled date, a change to date and time (CDT) may be granted. CDTs due to FESS cause considerably long delays, but the volume and duration of FESS-related delays have decreased from 16% of claims in 2013‑14 to 8% of claims during 2014‑15. FESS‑related delays have declined particularly for non‑DCO claimants. With that said, from an overall system perspective, CDTs due to pending FESS pose a larger impact on the overall average processing time (18 days in 2013‑14 and seven days in 2014‑15) compared to CDTs for fairness and natural justice or operational limitations (two days and under‑one day respectively). Approximately half the delays associated with pending FESS occur prior to the IRB receiving the FESS, while the other half of delays occur after the FESS is received by the IRB, which reflects the difficulty of getting delayed cases back on the schedule.
Recommendation #12: Surges in intake are likely to continue to magnify the operational challenges under the new system and test RPD resilience. Delays in scheduling initial hearings resulting from such surges should be tracked and examined to understand the precise cause of delayed initial hearings during these periods. Equally important, the RPD should consider the downstream effects of such surges on member productivity, in an effort to more fully understand the division-wide impact of these surges.
While the RPD has approached its RPP timeliness targets, it still remains below them. In the current performance measure (80% of claims finalized within 90 days of being expected to proceed), the RPD came within three percentage points of meeting its target as 77% of claims were finalized on time. In the other RPP performance measure of median time to finalize claims, the RPD exceeded its target of 4 months (3.2 months in 2014–15).
Recommendation #13: In its performance measurement framework that became effective in fiscal year 2015‑16, the IRB conceptualizes FESS as external and outside its control. Yet, data shows that delays occur after FESS confirmation has been received. This part of the delay is due to the operational limitations of the RPD and the IRB should categorize it as such in a revised performance measure. Using this approach, the RPD will be able to provide a more complete assessment of the impact of reforms and the capacity issues it is facing.
While the above measures of timeliness show that the RPD is improving, they do not show the effect of claims not being finalized on time. The importance placed on scheduling the initial hearing means that secondary intake can remain unresolved for a period of time. The growing age of the new inventory reflects key informants' perceptions that while the system is making strides in meeting established targets, the current pace may not be sustainable with currently available resources. As a result, the system may revert to a pre‑reform‑like backlog of claims.
Recommendation #14: Monitoring and reporting on the backlog should be factored into the RPD's performance measures. The RPD should track and report on the volume and age of the inventory, and analyze and report on the nature of the cases in the inventory and the factors that contributed to these cases ending up as backlog. By including the backlog in its performance measures, the RPD will be able to provide a more complete assessment of the impact of reforms and the capacity issues it is facing.
The RAD is seeing a growing volume of finalizations over the regulatory/performance targets. Based on key informant interviews, the increase in finalizations beyond 90 days is largely due to two main issues: the fluctuating jurisprudence related to the scope of the RAD appeal that caused the RAD to take a hiatus in 2014 to allow appellants to make submissions on this issue; and the workload implications of the increasing volume,Note 7 coupled with relying on recordings rather than transcripts. Given the jurisprudential environment, the situation is too much in flux; no recommendations are offered related to the achievement of performance targets for the RAD. After the evaluation's period of review, the RAD continued to experience shortfalls in its member complement in comparison to growing intake due to pending government appointments and internal reallocation of GIC resources.
In 2013–14, only 3% (n=26) of the RAD's paper‑based appeals were finalized in excess of the regulatory target of 90 days. This changed drastically in 2014‑15 when 36% (n=677) of finalized paper‑based appeals exceeded the 90-day limit, dropping the RAD below its 80% performance target. On average, these in‑excess finalizations were finalized within 134 days — 44 days over the target.
Paper‑based appeals made up 99% of all appeals finalized during the evaluation period. Between 2013‑14 and 2014‑15, the RAD experienced a 45% increase in intake and came close to doubling the number of paper‑based appeals finalized per fiscal year (from 1,095 to 2,032). Reflecting key informant opinion that the RAD was working at capacity, the result of the higher intake was that the RAD took substantially longer in 2014‑15 to process appeals (from filing to finalization – 114 days) compared to 2013‑14 (74 days). Subsequent to the evaluation's period of review, the RAD has continued to face growing intake without a corresponding increase in capacity through either new government appointments or reassignments of members from the RPD legacy initiative.
Knowledge management tools are one way an organization can improve efficiency. For both RPD and RAD members, initial training was important: for RPD members because 80% of them were new; and for RAD members because the RAD was a new decision‑making body for refugee appeal determinations. Members who commented were very positive about the initial training provided to new members and found it to be thorough and effective training for their role. For the RPD, the average number of days of training per member in 2013‑14 was 15.8 days and 8.6 days in 2014‑15. This decline may reflect the reduced need to have training after most members had been serving for about one year, but it may also reflect the other demands placed on members. For the RAD, the average number of days of training per member in 2013‑14 was 14 days and 15.5 days in 2014‑15.
Interview findings showed that RPD and RAD members generally considered the training sessions to be useful, but they also found it difficult to accommodate training in their schedules without impacting their productivity. The training sessions covered a variety of topics and each region was able to determine what subjects they wanted to cover. Key informants commented that the RPD and RAD are proactive with training, and they stressed the importance of choosing training topics in consultation with members to ensure that they address identified needs. Suggestions for improvement included developing a database of professional development sessions so that they are more easily accessible after the session, and offering more sessions on writing concise decisions and holding focused hearings.
The RPD and RAD provide several types of tools to assist members with their work. In particular, members commented positively on available checklists and country discussion groups. Overall, most personnel believe they have the materials and equipment to do their work, although the percentage is lower than for the IRB as a whole.
- Training topics chosen in consultation with members;
- More stable VPN connection to support telework;
- A database of hearing recordings on the shared network to facilitate telework;
- Better search capability in the National Reasons Database;
- Access to a database of RAD decisions for RPD members; and
- Macros standardized nationally and available in French.
Recommendation #15: The RPD and the RAD should review the suggestions for the improvement of knowledge management tools raised in this evaluation and take action as warranted.