A Model and Best Practices for Risk Transformation

Increasing globalization and the associated business transformation mean that enterprises are now complex networks dependent on both the nodes within other organizations and the nodes within the same organization.¹ This business transformation includes outsourcing, offshoring, restructuring, mergers and acquisitions, and value chain optimization. It also creates new technology risk such as increasing third-party providers, complex service interconnections and challenges of emerging markets; other problems such as intellectual property, liability and data sovereignty issues also occur.²

Board members and executives believe that risk frameworks, processes and structures are no longer giving them the level of assurance they need. They see an increase in the speed and effect of risk events and a reduction in their ability to identify and tackle new risk.³ Cost pressures from boards feeling they are spending too much time and money running risk processes⁴ with limited effectiveness means risk organizations are being driven to optimize headcount, rationalize infrastructure and improve operating efficiency of their risk processes and technologies.⁵ The majority of risk managers see efficiency as the strongest driver of future investment in risk management.⁶

According to one report, technology is now involved in more than half of critical operational risk, and cybersecurity spending, for example, is growing at three times the rate of the technology being secured.⁷ As risk increases, control increases as well to mitigate the risk; controls become more prescriptive (e.g., Payment Card Industry Data Security Standard [PCI DSS]) and more dependent on the predictive model used.⁸

From a risk practitioner perspective, the following can be seen: fragmented risk governance and policy frameworks, a hodgepodge of nonscalable point solutions, spreadsheets and manual processes giving multiple views of risk and control,⁹ a shortage of qualified resources, and unclear business and technology ownership and accountabilities.¹⁰ The disparate systems cause confusion due to duplicated, opaque and contradicting information and cause waste due to overlapping and inconsistent strategies, policies, processes and systems that must be maintained.¹¹

Clearly, a transformation of the risk-management function is needed alongside business transformation.

A Model for Risk Transformation

A formal definition of “risk transformation” does not exist. Loosely, it is the continual evolution of an organization’s risk function, systems and processes. So, delving deeper into the risk management business model to examine risk transformation fully is required.

Risk management is a “value shop”; that is, a process of value creation for knowledge-intensive industries that involves five primary activities: problem finding, problem solving, choice, implementation, and follow up and control¹² (figure 1). Embracing the value shop model represents a potentially more useful approach for planning risk transformation than perpetuating a backward-looking risk function¹³ focused on tactical improvements to the linear Committee of Sponsoring Organizations of the Treadway Commission (COSO) risk activities of objective setting, event identification, risk assessment, information and communication, and risk response.

The ISACA Business Model for Information Security (BMIS) is a useful framework for articulating the levers that can be adjusted within a risk-transformation program.¹⁴ These levers include organizational design and strategy, people, process and technology and their interconnections, including governing, culture, enabling and support, emergence (or continuous improvement), human factors, and architecture (figure 2).

A risk transformation journey (figure 3) would then be to assess the current state of the risk-transformation program and decide which capabilities related to the BMIS levers require what degree of enhancement to transform each of the value shop activities.

Most organizations are focused on risk identification and rating, with half developing dashboards, fewer improving governance, and only a third addressing skills and culture.¹⁵

Governance, Organization and Enablement

Governance involves “evaluating” and “directing” the risk management plans, ensuring resources are used responsibly¹⁶ and “monitoring” the contribution of risk management in achieving the enterprise strategy.¹⁷ Risk and strategy are linked through risk appetite.¹⁸

The organizational design defines how the organization implements its strategy,¹⁹ including where final responsibility for risk management lies, what the mandate and value add of the risk function can be, and how the board of directors (BoD) influences risk-related decisions. It also defines the role, structure and staffing of the risk organization and various risk committees.²⁰

Successfully transforming risk governance and risk organization requires direct BoD and executive management sponsorship and direct chief risk officer leadership.²¹ It also requires both defining the roles of various management levels within the risk process and strengthening and streamlining risk committee structures.²²

Some questions to ask include:

• How effective is the governance of risk management and where does it need to improve?
• What risk factors are fundamental to the enterprise strategy?
• What is the risk to the strategy and the risk of the strategy?

Governance is essential to the transformation of the risk value shop “problem finding” activity because it decides the accountability and triggers for changes in risk models and what human intervention is required within automated processes. Risk governance is also necessary to decide whether analytically determined risk needs to be cascaded upward and how data quality errors and model risk should be addressed.²³

Current risk-decision structures and processes (the “choice” value shop activity) have developed organically without standardized process flows or a clearly defined end state, and they will need to be redesigned before automation can occur.²⁴ The risk appetite in many organizations is not explicitly defined or used to determine a strategic approach to risk,²⁵ so it will need to be defined and articulated.

The risk-appetite statement should include the board’s attitude to risk, the business and risk environment, and the organizational risk culture and value proposition for the risk function.²⁶ The end state of the “choice” activity is risk integrated with the corporate strategy and business management weighing risk/return implications and potential risk trade-offs in their strategic and operational decisions.²⁷

The role of a transformed risk function is to make business functions more self-sufficient and help embed risk management in regular operational processes.²⁸ The governance oversight of this “execution” value shop activity is establishing risk within the functional model and operating plan (resources, budget, goals) for each business function (resources, budget, maturity goals) and mapping risk goals to their strategic business goals.²⁹ Defining and enforcing an enterprise metrics framework that includes risk metrics is an important governance “control/evaluation” value shop activity. Challenges will exist in achieving consensus on the most important metrics to measure or standardize and how metrics relate to each other.³⁰

To ensure the right people receive the right information and are empowered to make risk-aware decisions,³¹ policies, standards and guidelines must be designed and promulgated. These execution-enabling documents need to ensure roles and responsibilities are well articulated and both flexible to changing business objectives and easy to follow at all levels of the organization.³²

Architecture, Process and Technology

Building the right technology and data architectures and the corresponding processes, tools and applications are key enablers for risk transformation,³³ with the most cost-effective transformation measures being simplification, standardization and digitization.³⁴ In terms of the risk value shop model, the main role of technology is to deliver the right information to the right people in a timely manner to identify and resolve risk-related issues and distill information in ways that help the right people predict, prevent, detect, manage and report the risk associated with decisions.³⁵

Most risk reference architectures focus on the risk data flows and not the complete set of capabilities needed for risk transformation, including business activity management (BAM), business intelligence (BI), business process management (BPM), and governance, risk and compliance (GRC).

The author developed and has been working with the reference architecture shown in figure 4 for the past decade. The architecture addresses the BPM and GRC “reactive” capabilities needed for the “execute” and “evaluation” value shop activities and the well-architected BAM and BI “proactive” capabilities needed for “problem finding,” “problem solving” and “choice” activities.³⁶

Many organizations are struggling with backward-looking technologies and processes that focus on their current risk profile rather than on identifying their optimal risk profile for the future. These organizations need deep insight into root causes, indirect effects and early warning signals based on new rating methodologies using unique and innovative data sources.³⁷

Current processes and technologies for problem finding and problem solving that increase control measurability and risk responsiveness include:

• Sensors and agents³⁸ such as social media listening platforms, horizon scanning and early warning systems³⁹
• Integrated data warehouses⁴⁰ and big data technologies such as Hadoop⁴¹
• High-frequency reporting⁴² and continuous control monitoring⁴³
• Crowdsourcing risk information⁴⁴
• Visual data discovery, optical character recognition⁴⁵ and natural language processing⁴⁶
• Scenario planning and stress testing⁴⁷
• Risk analytics and machine learning

Fast automated access to accurate data through a business activity management (BAM) platform and an integrated data warehouse are prerequisites for the strategic use of advanced analytics. A BAM platform with appropriate sensors and agents answers questions using a set of analysis techniques that include baseline, threshold monitoring, correlation, root cause, impact analysis and predictions.⁴⁸

Continuous control monitoring allows testing on a full population and offers near “absolute assurance” as opposed to reasonable assurance using traditional audit and assurance methods.⁴⁹ In addition to cost reductions through improved efficiency and effectiveness, other benefits of continuous control monitoring include increased test coverage (through greater sampling and the ability to do more with the same or less labor), improved timeliness of testing, reduced risk velocity, potentially reduced remediation cost, greater visibility (when included in a GRC solution), improved consistency and the ability to identify trends.⁵⁰

Risk analytics is the use of mathematical methods and tools⁵¹ to facilitate the risk value shop activities. These tools have evolved from “what did happen” data warehouse reporting to “why something happened” online analytical processing (OLAP) tools. The tools now comprise “what will happen” machine learning and artificial intelligence (AI). The key elements of risk analytics today are high-frequency reporting based on high-quality data, risk analysis drilling down to detailed data, risk modeling and optimization to predict events, continuous monitoring, real-time alerting, and automated response mechanisms.⁵²

Machine learning algorithms can be classified⁵³ as shown in figure 5.

In figure 5, specific machine learning algorithms can be selected for further consideration, based on the purpose required. In the “problem finding” and “problem solving” value shop activities, analytics and machine learning can be used to:

• Discover processes from logs⁵⁴
• Detect process inconsistencies between different groups or applications⁵⁵
• Perform rule-based checking (such as service level agreements [SLAs])⁵⁶
• Identify trends and make forecasts⁵⁷
• Predict the likelihood and effect of an event⁵⁸
• Classify data and build decision tree
• Visualize data⁵⁹
• Discover key risk indicators (KRIs)⁶⁰
• Analyze stresses and scenarios⁶¹
• Identify typical problems or common solutions⁶²
• Generate treatment options for the risk of action, inaction, overreaction and underreaction⁶³

Current processes and technologies for the “choice” value shop activity include:

• Machine learning (as described previously)
• Simulation models⁶⁴
• Stochastic optimization models
• AI
• Business process management⁶⁵

The benefit of analytical measures for decision-making is that they can remove the overconfidence and anchoring biases in manual decision-making.⁶⁶

However, due to evolving understanding of what information should trigger action, the use of analytics for decision-making should also be iterative.⁶⁷

As mentioned in the architecture discussion, the BI-provided awareness and analysis needs to be combined with the BPM to provide the business rules, process models and process orchestration needed to make decisions and execute them. BI then enhances BPM to make decisions more efficient⁶⁸ and more repeatable, scalable, traceable and accurate.⁶⁹

Current processes and technologies for the “execute” and “evaluate” value shop activities include:

• Machine learning
• BPM
• Case management through an integrated GRC⁷⁰

A majority of organizations have structured only part of their risk-reporting process and do not have predefined escalation mechanisms in place.⁷¹ An integrated GRC approach improves effectiveness by reducing complexity, facilitates standardization of processes and taxonomy, allows for the rationalization of controls, and enables a movement from tactical to strategic activities through automation.⁷²

Automating controls reduces the cost of the control (resources, time and effort) and the cost of assurance, which leads to reduced risk through increased control effectiveness and coverage.⁷³ An advanced stage of automation is “enterprise valuation control,”⁷⁴ where business performance is assessed and process risk vs. value is calculated at near real time during process execution. Beyond this stage is an adaptive IT system that will allow a movement from “security by design” to an adaptive threat, security and risk model.⁷⁵

In determining a risk-transformation road map, a number of technology implementation options exist,⁷⁶ including:

• Using a full-stack vendor and implementing technologies based on their road map—if the organization is prepared for potential lock-in
• Picking a best-of-breed solution that includes BI, BPM, GRC and an application development platform—if the organization can invest in the necessary skills
• Picking a BI platform only—if the organization has sufficient data management and integration tools and skills

People, Culture and Human Factors

A key challenge to risk transformation is developing an enterprisewide risk culture⁷⁷ to increase risk awareness and accountability.⁷⁸ However, most organizations are struggling with how to assess their culture and how to change it, let alone establishing the needed change teams, action owners and milestone for risk cultural change.⁷⁹ Culture is a pattern of shared experiences; expected behaviors, beliefs, assumptions and attitudes; and written and unwritten ways of doing things. Culture is both emergent and learned, and it is created from both internal and external factors.⁸⁰ Cultural change is most successful⁸¹ when it incorporates:

• Role modeling and “accompaniment”
• Developing talent and skills⁸²
• Creating “social capital”
• Implementing formal mechanisms such as well-defined roles⁸³ and KPIs

Social capital comprises three dimensions:⁸⁴

• Structural—Developing network ties where people trust each other and commit to shared activities
• Cognitive—Establishing a shared taxonomy and evolving informal codes of practice
• Relational—Identifying as a group with associated trust and obligations

To transform the risk value shop “problem solving” activity, analytical skills that span the organization⁸⁵ need to be quickly developed in the already changing workforce.⁸⁶ These skills include mining structured and unstructured data, identifying data risk scenarios, working with databases, applying statistical methods and advanced analytics, leveraging analytics in risk assessments, and visually presenting complex data analysis.⁸⁷

The pooling of risk data is crucial to collecting statistically significant data for predictive models, and willing collaboration is needed to drive this sharing of data across all business lines.⁸⁸ The creation of a “single view of risk” through collaboration will also enable risk and controls to be treated holistically, thereby increasing effectiveness and efficiency.⁸⁹

Recruiting and nurturing talent with an innovative “test and learn” mind-set and creating a culture of innovation are also necessary to transform the risk value shop “problem solving” activity.⁹⁰ Innovations may include tapping into gamification—that is, harnessing game-playing principles such as competition to motivate development of out-of-the-box risk treatment options,⁹¹ or harnessing the power of crowdsourcing by hosting operational analysis/risk analysis hackathons.⁹²

The risk value shop “choice” activity includes determining and articulating the risk philosophy and risk appetite, setting risk objectives, and making decisions between alternatives. The required cultural change includes explicitly tasking business and technology process owners with both organizational objectives and risk management responsibilities and giving them a support infrastructure⁹³ such as a community of practice (CoP) or center of excellence (CoE).⁹⁴ These constructs require talent with critical-thinking skills and hands-on experience in technology, business and risk in order to act as a thought partner and guide strategic decisions.⁹⁵

A center of excellence⁹⁶ transfers best practices and learning derived from bottom-up benchmarking. A center of excellence has four key elements:

• Authorization—Assisting in aligning resources with strategic objectives
• Standards—Establishing standard tools, templates and methodologies
• Education—Providing training and education to all concerned with risk management
• Readiness—Establishing whether an organizational unit is ready to use the required methodologies

A role of the transformed central risk function (as CoE) in the “choice” activity is to cascade down (and explain) the risk-appetite statement, risk-approval delegations and risk-appetite measurement metrics. These metrics are risk, financial and operational indicators linking risk appetite and business performance at each level of the organization.⁹⁷ A consistent and global application of risk management disciplines provides several benefits: greater insight into key risk drivers and indicators; increases in operational efficiency, service improvements and higher customer satisfaction; decreased risk exposures and improved strategic decision-making.⁹⁸

One cognitive change that needs to occur to transform the risk culture around the “choice” activity is leading operational staff to see risk decisions as their own business decisions and not the responsibility of a central risk function.⁹⁹ Another cognitive change is to have a forward-looking strategic approach (“What risk/return trade-offs need to be made in the coming year?”) rather than a backward-looking approach (“How did the organization perform over the last year?”).¹⁰⁰

Emergence and Continuous Improvement

Risk transformation needs a dynamic interconnection between people and processes that will continuously develop and evolve¹⁰¹ through feedback loops, process-improvement initiatives and consideration of emerging issues in risk management.¹⁰² Some of these issues include:

• Decreasing data reliability as data availability increases—creating risk from business decisions based on that data¹⁰³
• Investigating the potential for flawed or misused AI algorithms—creating risk from business decisions using those algorithms¹⁰⁴
• Expanding cyberthreat surface as AI systems replace more vital business functions¹⁰⁵
• Making the optimistic assumption that systems will identify and manage all risk—emerging risk may be overlooked¹⁰⁶
• Displacing roles due to risk automation¹⁰⁷
• Teaching skills needed to ensure data quality is understood and managed
• Developing skills needed to ensure the appropriate and ethical application of automation and analytics¹⁰⁸

Endnotes

¹ Ray, B.; C. Apte; K. McAuliffe; L. Deleris; “Harnessing Uncertainty: The Future of Risk Analytics,” IBM Research Report, IBM T. J. Watson Research Center, 14 April 2008, http://domino.research.ibm.com/library/cyberdig.nsf/papers/B910FD442135744585257434005349F4
² PricewaterhouseCoopers, “Risk in Review: Global Risk in the Information Age,” USA, 2013
³ PricewaterhouseCoopers, “Black Swans Turn Grey: The Transformation of Risk,” USA, January 2012, http://www.pwc.co.uk/assets/pdf/risk-practices-black-swans-turn-grey-the-transformation-of-the-risk-landscape.pdf
⁴ Ibid.
⁵ Deloitte, “Risk Transformation: Aligning Risk and the Pursuit of Shareholder Value,” USA, 2014, http://www2.deloitte.com/content/dam/Deloitte/global/Documents/Financial-Services/gx-fsi-us-grc-RiskTransformation-in-Financial-2013-10.pdf
⁶ Vohradsky, D.; “Towards a Single View of Risk,” TCS Bancs Magazine, 2011, http://www.scribd.com/document/325043567/BaNCS-Newsletter-12th-Edition-06-2011
⁷ Bevan, O.; S. Ganguly; P. Kaminski; C. Rezak; The Ghost in the Machine: Managing Technology Risk, McKinsey & Company, July 2016, http://www.mckinsey.com/business-functions/risk/our-insights/the-ghost-in-the-machine-managing-technology-risk
⁸ Cordery, C.; M. Woods; P. M. Collier; “Value Chain to Value Cycle: The Role of Risk Management and ICT,” SSRN Electronic Journal, August 2010, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1761661
⁹ Op cit Vohradsky
¹⁰ Accenture, “Managing Successful Finance and Risk Transformations in the Insurance Industry,” Accenture, 2014, http://www.accenture.com/t00010101T000000__w__/pl-pl/_acnmedia/Accenture/Conversion-Assets/DotCom/Documents/Local/pl-pl/PDF/Accenture-Managing-Successful-Insurance.pdf
¹¹ Op cit Vohradsky
¹² Op cit Cordery et al.
¹³ Bongiovanni, C.; L. Pancaldi; U. Stegemann; G. Taglioni, Transforming Enterprise Risk Management for Value in the Insurance Industry, McKinsey & Company, July 2016, http://www.mckinsey.com/business-functions/risk/our-insights/transforming-enterprise-risk-management-for-value-in-the-insurance-industry
¹⁴ ISACA, Business Model for Information Security, USA, 2010, xhrj.yutb.net/Knowledge-Center/BMIS/Pages/Business-Model-for-Information-Security.aspx
¹⁵ Op cit PricewaterhouseCoopers, 2013
¹⁶ Op cit ISACA
¹⁷ Standards Australia, “AS 8015-2005: Australian Standard for Corporate Governance of Information and Communication Technology,” Standards Australia, 2005, http://www.standards.org.au/standards-catalogue/sa-snz/communication/it-030
¹⁸ Op cit PricewaterhouseCoopers, 2012
¹⁹ Op cit ISACA
²⁰ McNish, R.; A. Schlosser; F. Selandari; U. Stegemann; J. Vorhoit; “Getting to ERM: A Roadmap for Banks and Other Financial Institutions,” McKinsey Working Papers on Risk, March 2013, http://www.mckinsey.com/business-functions/risk/our-insights/getting-to-erm-a-road-map-for-banks-and-other-financial-institutions
²¹ Op cit Bongiovanni et al.
²² Op cit McNish et al.
²³ Op cit Ray et al.
²⁴ Ganguly, S.; H. Harreis; B. Margolis; K. Rowshankish; Digital Risk: Transforming Risk Management for the 2020s, McKinsey & Company, February 2017, http://www.mckinsey.com/business-functions/risk/our-insights/digital-risk-transforming-risk-management-for-the-2020s
²⁵ Op cit PricewaterhouseCoopers, 2012
²⁶ Ibid.
²⁷ Op cit Bongiovanni et al.
²⁸ Economist Intelligence Unit, “Beyond Box Ticking: A New Era in Risk Governance,” The Economist, United Kingdom, 2009, http://graphics.eiu.com/marketing/pdf/Beyondboxticking.pdf
²⁹ Pironti, J. P.; “Key Considerations When Evaluating ISRM Programs and Capabilities,” ISACA Journal, vol. 2, 2011, http://xhrj.yutb.net/archives
³⁰ Bakshi, S.; “Performance Measurement Metrics for IT Governance,” ISACA Journal, vol. 6, 2016, http://xhrj.yutb.net/archives
³¹ Op cit Deloitte
³² Op cit ISACA
³³ Op cit Ganguly et al.
³⁴ Harle, P.; A. Havas; H. Samandari; The Future of Bank Risk Management, McKinsey & Company, July 2016, http://www.mckinsey.com/business-functions/risk/our-insights/the-future-of-bank-risk-management
³⁵ Rosenfelder, S.; “Risk Analytics: A Journey in Risk Transformation,” presentation to the Institute of Business Analytics, Indiana University, Kelley School of Business, Indiana, USA, 8 October 2014
³⁶ Op cit Pironti
³⁷ Op cit McNish et al.
³⁸ Op cit Rosser
³⁹ Op cit PricewaterhouseCoopers (2013)
⁴⁰ Ibid.
⁴¹ Kress, R.; D. Hildebrand; “How Analytics Will Transform Internal Audit,” ISACA Journal, vol. 2, 2017, http://xhrj.yutb.net/Journal/archives
⁴² Op cit Rosenfelder
⁴³ Vohradsky, D.; “A Practical Approach to Continuous Control Monitoring,” ISACA Journal, vol. 2, 2015, http://xhrj.yutb.net/archives
⁴⁴ Op cit Harle et al.
⁴⁵ Raphael, J.; “Rethinking the Audit,” Journal of Accountancy, April 2017, http://www.journalofaccountancy.com/issues/2017/apr/rethinking-the-audit.html
⁴⁶ Sood, A. K.; M. Rinehart; “Data Science as a Tool for Cloud Security: Cloud Generation Visibility, Detection and Protection,” ISACA Journal, vol. 4, 2016, http://xhrj.yutb.net/archives
⁴⁷ Op cit PricewaterhouseCoopers (2014)
⁴⁸ Op cit Bongiovanni et al.
⁴⁹ Caron, F.; J. Vanthienen; “Applications of Business Process Analytics and Mining for Internal Control,” ISACA Journal, vol. 4, 2012, http://xhrj.yutb.net/Journal/archives
⁵⁰ Op cit Vohradsky, 2015
⁵¹ Op cit Ray et al.
⁵² Op cit Rosenfelder
⁵³ Brownlee, J.; “A Tour of Machine Learning Algorithms,” http://machinelearningmastery.com/a-tour-of-machine-learning-algorithms
⁵⁴ Ferreira, D. R.; M. Mira da Silva; “Using Process Mining for ITIL Assessment: A Case Study With Incident Management,” Bournemouth University, Dorset, England, 2008, http://web.ist.utl.pt/diogo.ferreira/papers/ferreira08using.pdf
⁵⁵ Op cit Caron and Vanthienen
⁵⁶ Ibid.
⁵⁷ Op cit Ferreira and Mira da Silva
⁵⁸ Op cit Ray et al.
⁵⁹ Op cit Caron and Vanthienen
⁶⁰ Op cit Ray et al.
⁶¹ Op cit Rosenfelder
⁶² Op cit Ferreira and Mira da Silva
⁶³ Op cit PricewaterhouseCoopers, 2013
⁶⁴ Op cit Ray et al.
⁶⁵ Op cit Rosser
⁶⁶ Op cit Harle et al.
⁶⁷ Op cit Rosser
⁶⁸ Op cit Ganguly et al.
⁶⁹ Op cit Rosser
⁷⁰ Op cit Vohradsky, 2015
⁷¹ Op cit Bongiovanni et al.
⁷² Op cit Vohradsky, 2011
⁷³ Dutta, A.; D. Dopp; “A Framework for Estimating ROI of Automated Internal Controls,” ISACA Journal, vol. 5, 2011, http://xhrj.yutb.net/archives
⁷⁴ Op cit Rosser
⁷⁵ Wohlgemuth, S.; “Resilience as a New Enforcement Model for IT Security Based on Usage Control,” IEEE Security and Privacy Workshops, 2014, http://ieeexplore.ieee.org/document/6957281
⁷⁶ Op cit Rosser
⁷⁷ Op cit Accenture
⁷⁸ Op cit PricewaterhouseCoopers, 2012
⁷⁹ Op cit McNish et al.
⁸⁰ Op cit ISACA
⁸¹ Aiken, C.; G. Bhatnager; S. Kieler; J. Lavoie; J. Malan; M. Rennie; How Do I Create a Distinctive Performance Culture, McKinsey & Company, 2009
⁸² Op cit Accenture
⁸³ Op cit Rosenfelder
⁸⁴ Walker, D. H. T.; D. Christenson; “Knowledge Wisdom and Networks: A Project Management Centre of Excellence Example,” The Learning Organisation, 2005, http://pmcoe.ca/wp-content/uploads/2015/06/TLO-COE-Journal-Article.pdf
⁸⁵ Op cit Raphael
⁸⁶ Op cit PricewaterhouseCoopers, 2013
⁸⁷ Op cit Raphael
⁸⁸ Op cit Ray et al.
⁸⁹ Op cit Vohradsky, 2011
⁹⁰ Op cit Ganguly et al.
⁹¹ EY, “The Upside of Disruption: Megatrends Shaping 2016 and Beyond,” EYGM Ltd., 2017
⁹² Op cit Raphael
⁹³ Op cit Deloitte
⁹⁴ Op cit Walker and Christenson
⁹⁵ Op cit Bevan et al.
⁹⁶ Op cit Walker and Christenson
⁹⁷ Op cit McNish et al.
⁹⁸ Op cit Ray et al.
⁹⁹ Op cit PricewaterhouseCoopers, 2012
¹⁰⁰ Op cit McNish et al.
¹⁰¹ Op cit PricewaterhouseCoopers, 2012
¹⁰² Op cit ISACA
¹⁰³ Op cit Ray et al.
¹⁰⁴ Zongo, P.; “The Automation Conundrum,” ISACA Journal, vol. 1, 2017, http://xhrj.yutb.net/archives
¹⁰⁵ Ibid.
¹⁰⁶ Op cit Cordery et al.
¹⁰⁷ Op cit Zongo
¹⁰⁸ Op cit Harle et al.