Immigration and Refugee Board of Canada
Integrated Case Management System
Technical Assessment

TABLE OF CONTENTS

• Restrictions and Limitations
• Executive Summary
• Introduction
• Summary of Key Observations and Conclusions
• Observations
• Appendix A - Resources Interviewed
• Appendix B - Documents Reviewed
• Appendix C - Software Engineering Best Practices
• Appendix D - BPMS Market Scan
• Appendix E - Management Response

Acronyms

Acronym	Description
BPM	Business Process Management
BPMS	Business Process Management System
CIC	Citizenship and Immigration Canada
CMS	Case Management System
COTS	commercial off-the-shelf
DOS	disk operating system
EA	Electronic Architecture
EDM	Electronic Document Management
ePIF	Electronic Personal Information Form
IAD	Immigration Appeal Division
ICMS	Integrated Case Management System
ID	Immigration Division
IRB	Immigration and Refugee Board of Canada
IT	information technology
KPMG	Klynveld Peat Marwick Goerdeler
PIF	Personal Information Form
RPD	Refugee Protection Division
RUP	rational unified process
SDLC	software development life cycle
SQL	Selected Query Language
STAR	System for Tracking Appeals and Refugee Claims

Restrictions and Limitations

Restrictions

• This report and the comments and conclusions expressed herein are valid only in the context of the whole report. Selected comments or conclusions should not be examined outside the context of the report in its entirety.
• We have indicated within this report the sources of the information provided, we have not sought to independently verify those sources unless otherwise noted within the report.

Limitations

• This was a rapid and high level technical assessment conducted over a short period of time. We have done our best to cover all technical aspects of the Integrated Case Management System (ICMS) application and the project that led to its creation, however, it is possible that certain relevant documents were missed or an important resource not interviewed which could have a material impact on our observations and/or conclusions. This assessment should be interpreted in light of the cross-section of Immigration and Refugee Board of Canada (IRB) stakeholders we have interviewed, documents we have reviewed and time and budget that was available for the assessment.
• Further investigation may be needed before proceeding with the proposed actions as some fundamental questions have remained unanswered in this Assessment.
• The goal of our Assessment was not to analyze the overall project management of ICMS or the business requirements and business modeling pertaining to ICMS. However, in the course of ascertaining whether the ICMS functional specifications provided sufficient detail for guiding the system's architecture and that governance ensured these specifications were successfully implemented, we observed a number project management and business related issues. Some of these have been documented but would require further investigation prior to proceeding with any action in their regard.

Executive Summary

This report documents the results of the findings and observations of a rapid and high-level technical assessment that KPMG LLP was contracted by the IRB to undertake, regarding its partially implemented ICMS. The project was initiated to ascertain the technical viability of the system to sustain ongoing modifications to meet operational requirements and to perform more effectively.

• The high level objectives of the assessment included the following:
  • Conducting a high-level technical assessment of ICMS' overall design (architecture) to ascertain the technical viability of the system to sustain ongoing modifications to meet operational requirements and to perform more effectively
  • Providing guidance on the key priorities and actions for consideration and execution to help the IRB gain the technical functionality it requires to more efficiently and effectively conduct its case management processes
• The following items were deemed to be out of scope for this study:
  • Review of the ICMS project budget, costs and/or other financial aspects
  • Review of the ICMS project plans, release and iteration schedules and their content
  • Review of problem requests, change requests and/or other functional issues
  • Review of the ICMS project code or code reviews
• The project was conducted in three phases:
  • Phase 1 - Confirmation of project objectives and preliminary review of system documentation
  • Phase 2 - ICMS assessment, which involved conducting a system walk-through, interviewing with key stakeholders from both Operations and information technology (IT), and further review of documentation provided
  • Phase 3 - Analysis of documentation, information received in interviews, high-level market scan of Business Process Management System (BPMS) architectures and preparation of a final report

Based on interviews with key stakeholders, a system walkthrough and a review of the key documents provided, the following are our key observations and conclusions:

• The ICMS architecture and technology appears to be based on current technologies and on a flexible, process-centric architecture. Based on our high-level market scan, it appears that this architecture is consistent with recommended architectures for Case Management Systems (CMSs).
• ICMS' process-driven characteristics appear to be sufficiently robust but they appear not to address all of the IRB's major tribunal management business requirements.
• Business users are dissatisfied with the ICMS (Release 4) functionality and have reported numerous issues. These are generally of two types:
  • Problem reports / feature enhancements - Given sufficient time, effort and resources, the current ICMS software appears to be flexible enough to resolve these issues
  • Business process changes - Although ICMS has a process-centric architecture which should allow for changes in business processes, the ICMS processes as designed and implemented do not appear to be flexible enough to accommodate the large number of business process changes, and the speed at which they should be implemented in order to make ICMS operable.

To address these key observations and conclusions, the following key actions are suggested:

• Engage tribunal management and business end users in:
  • The definition and governance of ICMS functional specifications
  • The alignment of these functional specifications with the process reengineering efforts related to the integrated tribunal process
• Consider changing the mode of ICMS operation from the current "maintenance mode" back to a "software development mode" via a defined and funded project.
• Assess in greater detail, the existing ICMS architecture and its components to determine which sub-systems could deliver short-term business value.
• Consider changing the current Software Development Lifecycle (SDLC) methodology to increase agility and reduce iteration delays.

As a result of having undertaken a rapid and high-level assessment, further investigation may be needed before proceeding with the proposed actions as fundamental questions have remained unanswered in this assessment.

This report is organized into the following sections:

• Introduction - provides the objectives and scope of the study, our approach as well as some context for the ICMS project.
• Summary of key observations and conclusions - summarizes the results of the assessment.
• Observations - provide a more in depth description of our observations which formed the basis of our key observations and conclusions.
• Appendices - provide background information including stakeholders interviewed and documentation reviewed.

Introduction

Objectives and Scope of Engagement

• KPMG LLP was contracted by the IRB to undertake a rapid and high-level technical assessment of its partially implemented ICMS. The project was initiated to ascertain the technical viability of the system to sustain ongoing modifications to meet operational requirements and to perform more effectively.
• The high level objectives of the project include the following:
  • Conducting a high-level technical assessment of ICMS' overall design (architecture) to ascertain the technical viability of the system to sustain ongoing modifications to meet operational requirements and to perform more effectively
  • Providing guidance on the key priorities and actions for consideration and execution to help the IRB gain the technical functionality it requires to more efficiently and effectively conduct its case management processes
• The following items were deemed to be out of scope of for this study:
  • Review of the ICMS project budget, costs and/or other financial aspects
  • Review of the ICMS project plans, release and iteration schedules and their content
  • Review of problem requests, change requests and/or other functional issues
  • Review of the ICMS project code or code reviews

Approach and Methodology

Phase I - Initiation and Preliminary Documentation Review

As a first step in this Phase, we met with the Project Authority to:

• Reaffirm the objectives and scope of the engagement
• Reaffirm and finalize the proposed approach and related timetable
• Discuss communication with the stakeholders to be engaged throughout the engagement
• Establish points of contact for the scheduling of interviews and/or working sessions
• Obtain copies of key background documents related to:
  • Overall project scope and objectives
  • Business requirements, both functional and non-functional including use cases, scenarios, business model;
  • Technical documentation including analysis model, design model, deployment model, data model, entity-relationship model, system/enterprise/application architecture;
  • Application design, guidelines and standards including coding standards, coding templates, etc.
  • Project management documents including project plan, iteration plans, risk plans, change control process, issue lists, and project organization;
  • Test strategy, plan, cases, code, data and results for unit, integration, acceptance and performance tests
  • Operation guides, e.g. operator's guide, monitoring, disaster recovery, etc.
• Not all of the above documents were available for review, however, a preliminary review of the key documentation was undertaken to provide context and focus to an on-site assessment exercise.

Phase I began on May 12, 2008 and was concluded on May 14, 2008

Phase II - ICMS Assessment

During this Phase we undertook the following activities:

• Reviewed available documentation related to the design and architecture of ICMS and other relevant and available technical documentation.
• Conducted a system walkthrough and interviews with 13 key ICMS personnel including key executives, ICMS project members and other key stakeholders, to review:
  • Current ICMS functionality related to case management tasks and activities
  • Overall architecture of the system and how the various components were designed to interact with each other
  • How ICMS menus, features, processes, and options are designed to ensure adherence to business rules and practices
  • System releases and the key functionalities included in each release
  • Outstanding approved changes
  • Major technical issues encountered and how they have been addressed

Phase II began on May 15, 2008, and was concluded on May 21, 2008

Phase III - Reporting of Observations and Recommendations

Upon completion of Phase II, the results of our technical assessment and recommendations were included in a final report. The final assessment report includes the following items:

• An assessment of the robustness of the architecture of ICMS and the extent to which the components within interact according to established web-based software design standards.
• An assessment of whether the architecture is flexible enough to accept the major modifications currently planned to provide an operational CMS that will meet the IRB's minimum case management business requirements.
• Guidance on the key priorities and actions for consideration and execution to help the IRB gain the technical functionality it requires to more efficiently and effectively conduct its case management processes.
• Key technical risks the IRB may be exposed to through the continued use of ICMS in its present form.
• Other items which we deemed appropriate within the context of this assessment to bring to the attention of the IRB.

Phase III began on May 23, 2008, and concluded on June 13, 2008.

ICMS Project Context

• ICMS was a multi-year project to gradually replace the existing legacy System for Tracking Appeals and Refugee Claims (STAR) application with an ICMS that would support IRB's divisions: the Refugee Protection Division (RPD), the Immigration Division (ID), the Immigration Appeal Division (IAD), and the Refugee Appeal Division (does not exist yet).
• STAR is a Disk Operating System (DOS)-based Clipper application for case tracking that dates back to 1990. Users find the application non ergonomic. Its technology is outdated and cannot be extended to support new requirements.
• ICMS business users reside in three regional offices: Montreal, Toronto and Vancouver. ICMS was deployed nationally however, only 6,000 cases were entered into the system in 2007. No new cases have been added to the system since.
• Based on our interviews and documentation review, it is our understanding that the IRB runs about 90,000 cases a year with the largest and most complex handled by the RPD. This Division handles approximately 30,000 new cases yearly. IRB has a backlog of 50,000 cases to process and the backlog is growing.

History

• IRB has been working on the introduction of a CMS application since 1996. The timeline below presents the major milestones over the past 12 years:
  • 1989: creation of STAR application
  • 1996-2002: development of a CMS to replace STAR using Sustain (project cancelled in 2002)
  • 2002-03: definition of ICMS vision, project definition and Treasury Board funding for the deployment in 3 stages to the following IRB divisions:
    • Stage 1 = RPD
    • Stage 2 = IAD
    • Stage 3 = ID
  • 2004-2007: development of the ICMS application for RPD (Stage 1) with Preliminary Release 1, 2 and 3
  • April 2007: official release of ICMS (Release 4) to production for use in the RPD, new referrals were no longer entered into the STAR application, project team dismantled
  • August 2007: handling of new cases in ICMS was stopped, all new referrals were once again entered into STAR for processing
  • 2007-2008: ICMS maintenance and releases 4.1, 4.2, 4.3, 4.4 were developed and released to end users
  • As of May 2008, the ICMS application is deployed but is not used to handle new cases. As existing cases within ICMS are processed, data is extracted and recaptured within STAR and other systems for processing
• It appears that there has been limited resource continuity throughout the project from both IRB Management, and development and operations team resources.
  • For example, since the inception of ICMS, the IRB had: three Chairpersons, two Executive Directors, two Deputy Chairpersons of the RPD, five DG Operations, and four Directors Organizational Information Management
• The Development project team, which, according to stakeholders interviewed, was mainly comprised of external contractors, was almost completely disbanded at project end in April 2007.
• Few resources with ICMS project experience remain within the current IT and Operations organizations to provide the much needed guidance over the project's legacy (code, documentation, process, etc.).

Summary of Key Observations and Conclusions

These conclusions are based on the assessment outlined in the Observations Section of this report

Robustness of ICMS Architecture

Based on the information gathered through interviews with key stakeholders and a review of relevant technical documentation we observed that ICMS architecture and technology is:

• Based on current technologies and on commercial off-the-shelf (COTS) software packages integrated with limited customization to allow for future evolution and enhancements
• Based on a flexible, process-centric architecture which, given our high level market scan, appears to be consistent with recommended architectures for CMSs. Our high level market scan of BPMS engines reports that ICMS' core Business Process Management (BPM) technology, i.e. Handysoft Bizflow v10, is ranked among the leading human-centric BPMS tools (see Appendix D for details).
• Meeting some business requirements particularly in the following key areas:
  • Electronic Document Management (EDM) - The Hummingbird application and its related systems appear to provide required functionality
  • Reporting - The Cognos application reportedly provides reports and templates to answer the analysis needs of business users

How robust is the process-driven architecture?

• ICMS is a process-centric application, where a defined case management process determines the tasks that must be accomplished and the work items that IRB resources must perform. Process centricity is acceptable provided ICMS can generate and regulate all of the major activities expected to be performed by tribunal management.
• At the other end of the spectrum, a data-centric application would store and track multiple types of data related to case management. It is an event-neutral application with minimal process-oriented capabilities. Its most simple expression can be found in STAR or Adjudication Tracking System, but also in subsystems of ICMS (e.g. document management, scheduling, etc.).
• Based on our observations, ICMS' process-driven characteristics appear to be sufficiently robust to incorporate business processes such as those found within tribunal management. However, ICMS as designed today does not appear to address the IRB's major tribunal management requirements.

In the Observations section of this report, additional information is provided related to process-centric and data-centric systems and their impact on system architecture.

Flexibility of ICMS Architecture

Based on the information gathered through interviews with key stakeholders and a review of relevant technical documentation we observed that:

• Business users are dissatisfied with the ICMS Release 4 functionality and have reported numerous issues with the application, namely:
  • Performance: Response time is too slow on certain screens, flow between screens is too long, etc.
  • Ergonomics: The user interface is not intuitive. Web pages are usually longer than screen size and require a fair amount of scrolling to get to appropriate data entry fields or required information, and there is no anchoring of scroll bars. Search and filters to help control information displayed are absent or limited.
  • User roles: Appear to be inflexible and do not reflect organizational realities. This requires some users to have multiple user identifications in order to perform their jobs.
  • Data integrity: Users report not being able to find cases and work items when moving between steps in the process. Work items (which is how a case file is accessed) appear after days or weeks of delay.
  • Lack of Error Correction: The system currently does not support errors in the process steps. If a file is sent on to the next step in the process, there is no way to retrieve it if it was sent in error.
  • Other: New fields and bulk printing required, need to reduce/simplify the number of tasks, a new scheduling interface has been requested, need to include National Documentation Package documentation, increased reporting capabilities are required, etc.
• To date, the issues that have been identified as two main types:
  • Problem Reports and feature enhancements: These are very specific changes that lend themselves to very specific solutions (e.g. anchor scroll bars on screens).
  • Business process changes: These are more complex and costly as they require profound changes to the processes currently implemented within ICMS. These issues are defined in more general terms than problem reports and enhancements (e.g. need for error detection/recovery, system creates too many tasks, work item allocation is not efficient and creates resource allocation problems, need for more flexible roles, etc.).

Analysis of Flexibility

• The current ICMS software appears to be flexible enough to resolve issues related to the majority of the problem reports and enhancements identified by the IRB to date given sufficient time, effort and resources.
• Although ICMS has a process-centric architecture that should allow for changes in business processes, the ICMS processes as designed and implemented do not appear to be sufficiently accommodating to adapt to the large number of business process changes and the speed at which they should be implemented in order to make ICMS operable.
  • There are many indications that business process rules and procedures are not well incorporated into the ICMS design and do not reflect a business focused design
  • Business processes do not appear to be well defined, stable or definitive

Key Challenges and  Risks

Below are some key high-level challenges and risks that may hinder successful implementation of recommendations furnished in this report and the IRB's ability to move towards a version of ICMS that meets their business needs:

• Continuing to focus on the current process-centric design approach without getting business-end user involvement, understanding and acceptance would continue to produce a system which does not meet the IRB's business requirements.
• The IRB's ability to move to a more agile and iterative development cycle that minimizes time delays between iterations and involves business users.
• Inconsistent leadership and resource turnover that could undermine efforts to put ICMS back on track.
• Unresolved communications issues between and amongst business-end users, Operations, Development and Maintenance teams.
• The continued push by the regions to develop localized custom solutions to answer pressing business needs (e.g. case scheduling) that may not conform to enterprise-wide standards and requirements.

Key Suggested Actions for Consideration

Below are suggested actions for the IRB's consideration and execution. They are provided based on the information gathered through the course of this rapid high-level Assessment. As this Assessment was completed with time, scope and budget constraints, further investigation may be needed before proceeding with some of the proposed actions.

• Engage tribunal management and business end users in the definition and governance of ICMS functional specifications and the alignment of such specifications with the process reengineering efforts related to integrated tribunal processes. For example, the following actions should be considered:
  • Determine overall business architecture that reflects the IRB's caseflow management
  • Determine how best to leverage ICMS's existing solution (Release 4) using a newly defined business architecture as a means of addressing existing gaps
  • Assess the IRB business process maturity to determine its limits and the speed at which process changes can be made
  • Prepare a business case and project charter to get the ICMS initiative back on track
• Consider changing the mode of ICMS operation from the current "maintenance mode" back to a "software development mode" via a defined and funded project where:
  • Business end users, business analysts, systems analysts, developers, testers and other ICMS resources are united in a single, risk-sharing team with all members pushing towards common short, mid and long term goals
  • Dedicated, full-time business end users are trained on business process analysis and modeling, and have the authority to implement process changes in ICMS
  • Outside expertise can be acquired where required

From a technical perspective, as a minimum, the following actions should be defined and actioned as key priorities:

• Assess in greater detail the existing ICMS architecture and components to:
  • Determine, based on the IRB's business architecture, whether the IRB should continue with the current ICMS architecture or rededicate ICMS into a hybrid application, partly process-centric and partly data-centric
  • Determine which of the existing ICMS subsystems can be deployed rapidly to deliver business value as data-centric applications, outside the current process-centric architecture
• Consider changing the current SDLC methodology to increase agility and reduce iteration delays:
  • Move to an iterative approach with shorter development cycles and greater business user involvement than today; if possible, explore a more agile approach which could pair business end users with system developers
  • Take into consideration the unique features of the BPM lifecycle, which delegates certain activities to business end users (e.g. business modeling and changes in the BPMS tool directly)

Observations that Require Further Investigation

Our observations regarding process-centricity and data-centricity have led us to ask a number of questions that could not be answered within the scope of the current Assessment. These should be considered prior to moving forward with additional work on ICMS. Our questions include the following:

• Can ICMS deliver most of its business benefits by implementing only its data-centric subsystems?
• Can ICMS incorporate, over time, some process-centric subsystems that would deliver additional benefits over its data-centric subsystems?
• Have business process re-engineering and process harmonization been documented by business users and regions? Have they been approved and endorsed by them?
• Has a transition plan been developed to pace the IRB's case management and processes towards the vision it laid out for itself in 2003, and that served as the basis for the ICMS project?
• Do the IRB business and technology resources have the maturity to embrace all the human, process and technological changes that are inevitable with transformation projects as big as those envisioned with ICMS in its current form? Can ICMS be modified to spread the changes it introduces over a longer period of time?

OBSERVATIONS

Overview

The observations outlined in the following pages were gathered as a result of interviews with key stakeholders (see Appendix A), a system walkthrough and a review of relevant documentation provided (see Appendix B). In addition we have applied known software system engineering methodologies (see Appendix C) and performed a high-level market scan of BPMS software.

Although we have done our best to cover all aspects of the ICMS application and the project that led to its creation, it is possible that certain key documents were missed or resources unavailable that may have a material impact on our observations.

Our observations have been organized as follows:

1. Business Modeling and Requirements
2. Architecture, Analysis and Design
   • This section also includes further information on process-centric and data-centric systems
3. Implementation, Test, Deployment and Environments
4. Project, Configuration and Change Management

For each of these main topics, we organize our observations around the following lenses:

• What Was Done Well - within the context of the ICMS project
• Constraints/Challenges/Issues - that were observed and would impact actions taken in the future
• Areas Requiring Further Investigation - theses are areas we were not able to explore during our assessment but which may merit further investigation in building an action plan for the future

I. Business Modeling and Requirements

Although the goal of our assessment was not to analyze the business requirements and business modeling pertaining to ICMS, software engineering best practices require us to validate that ICMS functional specifications provided sufficient details for guiding the system's architecture and that governance ensured that those specifications were successfully implemented. Information gathered with this in mind has allowed us to make the following observations.

What Was Done Well

Detailed use cases have been documented

Constraints/Challenges/Issues

• Although business requirements appear to have been gathered and documented ('Use Cases - ICMS - Release 3'), our interviews with end users, trainers and business analysts have identified fundamental issues with ICMS. There appears to be a disconnect between business user requirements and expectations and the system delivered:
  • Demonstrated examples include the following: lacking bulk printing, search and filter missing, unusable scheduling (Montreal region developed custom application)
  • Only the "happy" path of use cases have been documented, which could lead to the lack of "error recovery", as mentioned by end users during interviews
• It is not clear if business architecture and business requirements were approved by end users and have been implemented as documented (possible governance issue).
• ICMS, as delivered, does not appear to reflect the business process reality of the IRB.
• User interface mockups and/or ICMS storyboards do not appear to have been reviewed with business users prior to the Release 4 implementation.
• The reasons for the disconnect between business user requirements and expectations and the system delivered will need to be further explored and confirmed before moving forward constructively to the next phases of ICMS.
• Of the system changes that have been identified, there does not appear to be an overall high level view of the changes and whether they will in fact resolve all of the business process requirements.

Areas Requiring Further Investigation

• Business architecture was not reviewed - an approved business architecture appears to be lacking.
• Business process analysis, modeling, and translation into ICMS feature list documentation has not been made available to cross reference with ICMS R4 implemented feature list.
• Although a high level overview of the caseflow business process within the IRB exists (RPD overview process, Integrated Case Management training guide), we were not able to verify if business "owners" were involved in developing or approving the process.

II. Architecture, Analysis and Design

What Was Done Well

• Overall ICMS application architecture appears to be state of the art, even leading edge in certain respects.
• Based on our high-level market scan, the BPMS architecture appears to be consistent with recommended architectures for CMSs
• The ICMS BPM engine (Handysoft Bizflow v10) is ranked among the leading human-centric BPMS tools (see Appendix D).
• Web services are leveraged to decouple components and provide reusable components.
• Security concerns have been factored in to the system architecture. It appears that security constraints/requirements play an important role in key architecture decisions which impact performance, environments, and software development iteration cycle duration.
• Key architecture principles have favored "buy" over "build":
  • COTS have been leveraged in agreement with the "buy principle". The final solution integrates multiple components which remain lightly coupled, thus allowing increased flexibility regarding future product updates and the overall evolution of ICMS application: Hummingbird for EDM, Cognos for reporting and Business Intelligence, HandySoft BizFlow for BPM and user interactions.
  • The reviewed architecture indicates that COTS software customizations have been kept to a minimum to support future product updates.

Constraints/Challenges/Issues

• ICMS has a process-centric architecture, which relies on data-centric subsystems. The pages that follow provide further analysis and observations regarding this key issue.
• Our rapid BPMS market scan has shown that case management is recognized as having "special properties" when it comes to process automation. In particular, the reliance on "process queues and rules" is raised as an issue that also appears to be an issue for ICMS.
• BPMS engines allow for easier process modifications than traditional software architecture. However, business end users must be allowed to make changes to the process themselves in rapid iterations without requiring technical experts. We have not been able to validate that this is in place at IRB nor that it is possible.

Areas Requiring Further Investigation

• Interfaces with other systems (either internal or external) have not been reviewed. This area appears to be a key component of ICMS as the IRB must interact with systems at Citizenship and Immigration Canada (CIC) and other external organizations.
• The reasons for Electronic Personal Information Form (ePIF) not working are not clear and should be investigated further.
• Security constraints appear to have been important in the definition of ICMS architecture. Further analysis is required to determine if reduced security constraints can help simplify ICMS architecture and facilitate software development.

ICMS Comprised of Two Major Sub Systems

We observed that ICMS is comprised of  two systems: a data-centric and a process-centric system.

1. Data-centric subsystem - At its core, ICMS has a collection of information storage and management subsystems:
   • Multiple applications store and manage data essential to manage case files over their lifecycle
   • ICMS data-centric applications include:
     • Hummingbird as the document management system
     • Cognos as the reporting system
     • SQL server as the database management system
     • ePIF as the Personal Information Form (PIF) management system
2. Process-centric subsystem -  The process management and automation system:
   • Provides a uniform interface for data and applications
   • Manages the business process and business rules
   • Supports human activities and can automate certain activities
   • Monitors activities and reports real-time status on case priorities and delays
   • Handysoft BizFlow is the process-centric application in ICMS.

ICMS Data Centric System

Observations

• ICMS appears to suffer from data integrity issues that are not acceptable for business users given ICMS' expected role in providing reliable case management data.
• Multiple systems exists but, based on our interviews and system walkthrough, do not offer all of the required functionality and complex interfaces (e.g. reporting and scheduling systems).
• Reportedly, ICMS does not provide adequate tools for business users to perform their work efficiently or effectively (e.g. scheduling).

ICMS Process Centric System

Observations

• A BPMS system generally would require stable, well-known business processes, harmonized across all business users of the organization.
• The combinations, permutations and exceptions in the IRB tribunal management business process have not been fully implemented nor do they seem to have been defined by business users in the regions.
• Making changes to business processes requires business users to have the tools, knowledge and agility to change the processes themselves rapidly.
  • Business users appear to have had little involvement in the development of the current system processes

What Architecture is Needed to Achieve IRB Goals?

Our observations led us to question whether both process-centric and data-centric systems were necessary in order to meet IRB's goals for ICMS. ICMS Release 4 project plan (see extract to the right) was used as a basis to provide the following analysis of whether a process-centric BPMS architecture is required to deliver the IRB's goals and business objectives.

• The business improvement goals, as expressed in the Chairperson's mandated direction and purpose of the ICMS project, focus on the electronic capture and exchange of case file information.
• Only the "business process re-engineering" and "process harmonization across regions" objectives mandate the process-centric components of ICMS.
• It appears that the rapid implementation and deployment of the ICMS data-centric components could provide most of the business benefits and that the process-centric system could be postponed to a later implementation phase.
• The following data-centric sub-systems could have potential to bring short term benefits to the business end users:
  • EDM
  • ePIF
  • Electronic scheduling
  • Reporting

III. Implementation, Test, Deployment and Environment

What Was Done Well

• A great deal of documentation exists within the project's SharePoint intranet site.
• Based on the sample documents we reviewed, the documentation quality appears good, with details and appropriate content. This is especially true for the conceptual documentation prepared at the early stages of the project. However, our interviews have indicated that some documentation is incomplete in the later stages of the project (2006-2007) and logical and physical documentation has been neglected due to delivery time constraints.
• Sufficient number of environments appear to exist to support proper parallel development and the release of ICMS.

Constraints/Challenges/Issues

• Large numbers of environments currently exist to support ICMS; this large number of environments appears to increase the release cycle duration and be prone to introducing errors into the process.
• It appears that there is an absence of a sound documentation map.
• Basic documents such as technical overviews remain a work in progress and seem to have been created only recently.
• Documentation usage appears limited due to people not knowing what documents exist, where they are located, and/or what they contain. This may be due in part to the reported high turn-over of project resources.

Areas Requiring Further Investigation

• Code review was not undertaken and thus ICMS software code cannot be assessed at this time.
• There is a question as to whether or not a  move to the latest version of BizFlow (version 11) could provide features in bizcoves that could answer rapidly to pressing Change Requests and Problem Reports (e.g. search and filters, improved performance).
• Determining whether the number of environments and their complexity could be reduced.
• The effort related to documentation update, standardization of use and general cleanup.

IV. Project, Configuration and Change Management

What Was Done Well

• It appears that efforts have been made to plan and execute the development project using software engineering best practices (Microsoft Solutions Framework) and a structured software development approach.
• Project plans with structured development teams and processes appear to have been in place during the course of the project (2004-2007).

Constraints/Challenges/Issues

• The project appears to have had inconsistent leadership/ownership since its inception.
• The system appears to have been developed without the benefit of the business users iterative input and continuous approval of requirements, specifications, storyboards, processes, etc. Iterative development approaches (see rational unified process (RUP) in Appendix C) recommend that users provide constant feedback throughout the development cycle, and that change management disciplines be addressed continuously from project inception. This does not appear to have been the case for the development of ICMS.
  • It seems to indicate that business end users are involved only indirectly (via their business analyst representatives). Interviews substantiate, that in fact, end users were only involved in the User Acceptance Test late in the project (October 2006). A significant number of issues were reported at this time but many could not be addressed as the project end date was April 2007.
• Communication between Operations and IT needs to be improved. There appears to have been a general lack of understanding of the IRB business process by the project team which may have contributed to such things as: not including error detection/recovery in the system; and developing a scheduling module without an on-line calendar. Users communicate their requirements but may not understand all of the technical elements that they must specify for inclusion in the system. Technical programmers seem to have developed ICMS based on requirements provided without the benefit of testing technical solutions with business users to ensure they were in fact meeting the intended business requirements.
• ICMS project plans appear to follow a traditional "waterfall" approach where all functionality is delivered at once rather than an iterative approach where smaller pieces of functionality are delivered over a period of time with the objective of refining the functionality throughout the project:
  • During the project development (2004-2007), iterations do not appear to exist between major releases of the project (R1 to R4)
  • During maintenance (2007-today), iterations do exist but with significant delay (e.g. 6-9 months)
• In April 2007, the dedicated ICMS development project was dismantled and resources with intimate knowledge of ICMS were dispersed. Today accountability for the system is fragmented among different groups, thus making it more difficult to find dedicated resources and contributing to communication challenges. For example:
  • IT is responsible for the ICMS development team, infrastructure and deployment issues, security issues
  • Operations handles business analysis and end user training
  • Regions (multiple) have end users
  • Steering Committee provides governance

Areas Requiring Further Investigation

• Existing project plans and detailed list of Change Requests and Problem Reports.
• Validation  that project plan, governance and guidelines have been followed.
• The role of end users in ICMS specifications and their involvement during the project.

---

Appendix A - Resources Interviewed

The following key ICMS stakeholders and project team were interviewed on May 15 and 16, 2008:

1. Brigitte Desmeules - Director, Organisational Information Management
2. Bill Couch - A/Manager, Operational Support Systems
3. Jennifer Waymark - ICMS trainer
4. Penelope Routier - Business Analyst
5. François Thinel - Regional Coordinator / User representative
6. Lucie Loignon - Director, Information Systems
7. Bernie Vaz - Application Architect
8. Andrée Forget - Systems Analyst
9. Petar Vrzic - Development Lead
10. Dan Thanh Nguyen - Database
11. Troy MacFarlane - Cognos
12. Susan Siu - IRB Operations General Manager
13. Onnig Beylerian - A/Chief Audit Executive and Head of Evaluation

---

Appendix B - Documents Reviewed

ICMS Documents Reviewed

1. ICMS Overview (PowerPoint)
2. ICMS Implementation Overview (PowerPoint)
3. ICMS Technical Overview (PowerPoint)
4. ICMS R4 Technical Overview, v0.4, May 12, 2008
5. ICMS presentation - Bizflow Overview
6. ICMS R4 Logical Environment Diagram
7. Conceptual Design - ICMS - Release 2, v0.8, 26aug2004, status final
8. Conceptual Design - ICMS - Release 3, v0.4, 16aug2004, status final
9. ICMS R4 Technical Roadmap
10. ICMS R2-R3 Logical Environment Diagram
11. ICMS R4 EasyButton
12. BizFlow 10 Overview
13. Systems Development Standards
14. ICMS R4 Solution
15. ICMS R4 Big Picture, not versioned (created 8sept2005)
16. Release 4 Project Plan, v0.3, 26sept2005
17. Notes - RPD overview process, not versioned (created May 2004)
18. Use Cases - ICMS - Release 3, v1.0, 17aug2004, status final

Non-ICMS Documentation and Research

1. IRB - Case Flow Process Overview - May 12
2. BPTrends BPMS report 2007 on Bizflow v10
3. The Forrester Wave^TM: Human-Centric Business Process Management Suites, Q1 2006
4. Gartner Magic Quadrant for BPMSs, 2006
5. Q2'08 BPMS watch ratings, BPM Institute
6. The Process Audit, Michael Hammer, Harvard Business Review, April 2007
7. Getting Past the First BPM Project: Developing a Repeatable BPM Delivery Capability, BPTrends 2006

---

Appendix C - Software Engineering Best Practices

Our technical assessment is based on 3 leading software engineering and architecture best practices:

• RUP: widely known and used, RUP has been used as the basis to analyze the ICMS project and its deliverables. Specifically, RUP’s 9 major disciplines have been used in our assessment. RUP has been used in numerous large multi-year software development projects and encompasses technical, business and human aspects of large projects.

  RUP has been used in numerous large multi-year software development projects and encompasses technical, business and human aspects of large projects.

  We use RUP as a reference to ensure all aspects of the ICMS project have been covered.

• BPM enabled Application Development: BPTrends' Derek Miers has proposed an iterative approach for BPM projects in his 2006 paper "Getting Past the First BPM Project: Developing a Repeatable BPM Delivery Capability".
• Electronic Architecture (EA): EA frameworks and methodologies recommend that 6 major aspects be analyzed when looking at systems and their integration into the enterprise: business, data, information systems (applications), technology infrastructure, security and governance.

---

Appendix D - BPMS Market Scan

As part of the technical assessment, we have performed a rapid scan of leading technology research firms to evaluate ICMS core system, its BPMS engine, Bizflow v10 from the company Handysoft.

• Other ICMS applications (Hummingbird, Cognos, etc.) have not been reviewed due to lack of time.

Forrester Research (in its latest review of human-centric BPMS Q1'06) is the only firm that reviews Handysoft product

• It positions Bizflow as a strong performer
• It indicates a small market presence
• Low scores are given for its analysis & optimization features
• High scores are given for its product architecture and integration features.

BPM Institute, in its Q2'08 BPMS watch ratings, although it does not review the Bizflow product, it does provide insight on "special features" and "unique properties" of case management products. In particular, we note:

• It stresses that "Case management represents a very different style of human-centric BPM, in which the work is not routed to queues of task performers based on rules defined in advance, but in which users collaborate at runtime in a more ad hoc fashion."
• It allocates 50% of the total score for its case management products to collaboration.

BPTrends has reviewed Bizflow v10. It lists and reviews product features but does not provide any comparative research or qualitative comments, although certain statements are worth noting:

• "But perhaps the most impressive feature in BizFlow is the highly flexible and configurable user interface." (Bizcoves)
• Gartner does not review Handysoft Bizflow in its latest Magic Quadrant for BPMSs, 2006.

---

Appendix E - Management Response

In August 2007, the IRB responded to ICMS user concerns by diverting new RPD referrals to STAR to provide time to analyze and resolve systems problems. RPD files had by then reached the verification of the PIF stage.

The ICMS team, comprising regional coordinators, business analysts and system analysts conducted blitzes and simulations to test all the file processing functionalities, from referral to decision point, in order to pinpoint specific issues.

In February 2008, the ICMS team met with users to confirm the changes required to improve the system. The team drew up a list of problematic functionalities. The list meant that ICMS required additional adjustments before it could be fully functional in the operational context and work. These adjustments will take place over time.

Consequently, the Chairperson mandated the Audit and Evaluation Committee to commission a technical assessment of ICMS and directed the ICMS team to examine options to simplify the system. It was in this context that KPMG LLP conducted the ICMS technical assessment.

The IRB Audit and Evaluation Committee has expressed its appreciation for the expert analysis by KPMG LLP for its thorough review and recommendations.

The IRB has appointed Lucie Loignon, a project director, to assess the resources required and to determine the best organizational structure to address the renewal strategy. Her team will be working closely with users and coordinators to ensure that priorities are identified and changes implemented as swiftly as possible. Ms Loignon will soon be testing a new approach to increase the flexibility and timelines associated with the development and implementation cycles, while maintaining the current production mode of ICMS.

As the strategy takes shape, specific responses to the recommendations contained in the ICMS Technical Assessment will be added to this management response.