Getting a grip on IT Project Portfolios: Design and validation of an approach for the evaluation of IT project portfolio management

Organizations use projects to achieve their strategic objectives and increase their competitive advantage. IT Project portfolios (ITPPs) are essential for achieving strategy as they link strategy and operational projects. However, organizations often struggle with effectively managing and balancing portfolios of IT projects. Moreover, available resources for these IT projects are scarce. IT project port- folio management (ITPPM) has been implemented in most large organizations to manage a portfolio of IT projects and their interdependencies to achieve portfolio value.
Some authors indicated that ITPPM can be complex. Frameworks can be used to assess the conformity to standard practices. However, organizations find it challenging to use existing standards and frameworks to suit their context and requirements. Given the diversity of actual ITPPM contexts and the ever-present dynamics, developing a “one version fits all” ITPPM design seems impossible. ITPPM processes and roles must fit the organization and its context.
Many organizations have installed a form of ITPPM, but they are unsure about the quality of their ITPPM. An evaluation of ITPPM can enable organizations to identify, discuss, and implement potential ITPPM improvement areas.
Scientific literature lacks a complete and validated ITPPM reference model and approach for evaluating the design of ITPPM in an organization. Moreover, scientific studies have not yet resulted in an approach to a periodic evaluation of the operation of ITPPM. Also missing are complete and validated reference models for the structured evaluation of the operation of ITPPM.
This research gap prompted the central research objective of this thesis:
design and validate an approach for the evaluation of IT project portfolio management.
The research objectives of the first study were to design and validate an ITPPM reference model (RO-1) and an approach (RO-2) for evaluating the design of ITPPM in organizations in terms of processes, roles, responsibilities, and authorities. The reference model (i.e., RACI-matrix) indicates which roles perform which ITPPM activities in an organization and what responsibilities and authorities these roles generally have according to the literature. The evaluation approach includes a pragmatic approach that supports a structured evaluation and discussion of the local design of ITPPM, possibly leading to potential areas for improvement in the design of ITPPM in an organization.
To compose the components of the prototype ITPPM reference model, a SLR was firstly conducted to search the academic literature for ITPPM processes, roles, responsibilities, and authorities Secondly, a RACI matrix was created in which the reference roles from the literature were placed on the horizontal axis, and all processes and activities were placed on the vertical axis. The boxes in the RACI matrix indicated whether the roles involved in ITPPM were responsible, account- able, consulted or informed for the ITPPM activity in question.
During a multiple-case study, the designed ITPPM reference model and evaluation approach proved appropriate for systematically evaluating and discussing previous ITPPM design choices in an organization and identifying areas for improvement. It prompted practitioners to reflect and discuss previous ITPPM design choices within the organization.
The research objective of the second study (RO-3) was to design and validate a reference model of ITPPM health factors for evaluating the current status of ITPPM. A reference model of health factors is a set of criteria relevant to a peri- odic evaluation of the health status of ITPPM at a particular moment in time. The factors in the reference model can be considered as underlying dimensions by which the health of ITPPM can be evaluated and discussed among stakeholders.
To arrive at a reference model of ITPPM health factors, a SLR was firstly conducted to identify factors that might be appropriate to evaluate the health of ITPPM. Secondly, a classification of the potential factors identified from the literature into a clear and uniform set was needed to arrive at a useful set of factors. After an open card sorting session, a reference model of categorized potential health factors was available.
The results of a multiple-case study showed that all health factors manifested themselves in at least one of the five participating case organizations. In addition, practitioners were able to provide credible evidence for the usefulness of the health factors.
The research objective of the third study (RO-4) was to design and validate a reference model of ITPPM risk factors for evaluating the future status of ITPPM. A reference model of risk factors is a set of factors relevant to a periodic evaluation of the risks of ITPPM (i.e., the future health status of ITPPM). The factors in the reference model can be considered as underlying dimensions by which the risk of ITPPM can be evaluated and discussed among stakeholders.
To arrive at a reference model of ITPPM risk factors, a SLR was firstly conducted to identify factors that might be appropriate to evaluate the risks of ITPPM. Secondly, classifying all potential risks into a clear and uniform set was needed to arrive at a relevant set of factors.
The results of the multiple case study showed that all risk factors manifested themselves in at least three of the four participating case organizations and by at least ten of the twenty-four practitioners. In addition, practitioners could provide arguments for the relevance of the risks.
The research objective of the fourth study (RO-5) was to design and validate a reference model of relationships between ITPPM risk and ITPPM health. The reference model includes a matrix of identified and validated relationships between (the impact of) risk and health factors. The connection between ITPPM risk and ITPPM health is that risks that are not mitigated can negatively affect the health of ITPPM.
To realize a reference model of relationships between ITPPM risk and ITPPM health, firstly, risks that may impact ITPPM health were identified based on an analysis of previous research at the Open University and recent scientific literature. Secondly, a prototype reference model (i.e., a matrix) was designed based on this analysis.
The results of the multiple-case study showed that each risk was experienced in practice and impacted one or more health factors. Practitioners did not validate six relationships in the reference model. More case studies are needed to reach firmer conclusions about the relevance of these six relationships.
The research objective of the fifth study (RO-6) was to design and validate an approach for evaluating ITPPM concerning its current and future status. The approach consists of process steps, guidelines, and supporting tooling.
To realize an approach for evaluating ITPPM concerning its current and future status, a prototype for ITPPM health evaluation was firstly designed. Secondly, to test the quality of the design, actual user experiences of practitioners with the initial prototype approach during ITPPM health evaluation in their organizations were collected. The evaluation of this preliminary study revealed areas of improvement for the approach. Thirdly, the prototype approach was redesigned into a prototype approach for ITPPM health and ITPPM risk evaluation based on these areas of improvement and related work.
During a multiple-case study, the designed approach proved useful and easy to use for ITPPM evaluation. It encouraged practitioners to discuss their perspectives on the current and future status of the organization’s ITPPM and to define ITPPM improvement actions.
In the reflective interviews, practitioners generally indicated that the approach had proven its value by engaging in the right conversation during the evaluation with input from practitioners with different perspectives on the current and future ITPPM status. Moreover, they mentioned that the group discussions resulted in concrete ITPPM improvement points for the organization, ultimately contributing to its effectiveness and efficiency. The main disadvantage was that the evaluation approach was extensive and time-consuming. Practitioners also emphasized the importance of a neutral and experienced moderator.
In summary, using the reference models and approaches designed and validated in the studies, ITPPM can be evaluated from three perspectives:
• the structure of steering ITPPM;
• the current state of design and operation of ITPPM;
• the future state of design and operation of ITPPM.
In conclusion, this thesis has provided ITPPM reference models and approaches for using the reference models to evaluate the design and operation of ITPPM. These generic reference models and evaluation approaches can be contextualized and applied by organizations and with them ITPPM issues can be identified that need improvement. Organizations can get a better grip on their ITPP by using the ITPPM evaluation approach.