Validation of National Performance Data
A large Commonwealth government agency needed to validate the content and manner of processing of national Key Performance Indicators comprising two key data sets used across government and the primary health sector to monitor activities and inform policy development. This processing occurs through extraction of data from multiple clinical information systems and its transfer to a secure data portal.
The KPIs measure national performance against a range of critical health factors, and their calculation requires the manipulation of hundreds of variables. The agency required an independent and objective review of the accuracy of the data processing that is performed to produce these KPIs.
Doll Martin Associates conducted a detailed analysis of the definitions and algorithms used for the KPIs, and developed a Control Data Set that was input to each clinical information system to check the integrity of the data extraction, transformation and output reporting process. The testing against the Control Data Set included a data-point by data-point comparative analysis of the outputs from the clinical information systems as well as an interrogation of the logic used to calculate each indicator. This process required a detailed understanding of the purpose and structure of performance indicators, the role of clinical information systems and their data collection context, and the impact of data quality on the results. The Control Data Set was subsequently used in the following data collection period to validate output from the existing and an additional clinical information system.
The Control Data Set approach demonstrated that an extensive validation of the KPI reporting functions could be achieved, and that it enabled the reporting systems to be tested and evaluated much more extensively and precisely than had been possible to date. The Control Data Set was complemented by the development of governance and procedure instructions for its ongoing support.
Policy and Procedure Compliance Review for NSW Health
The NSW Ministry of Health commissioned an independent review of a selected service provided by hospitals in NSW, to assess adherence to legislation, NSW Health policies, procedures and advice provided by the Ministry of Health, and to identify where improvements could be made.
Based on the National and State standards provided, DMA conducted background research, and collaborated with the client to identify and contact key stakeholders and develop a consultation plan.
A computerised tool was developed to enable the efficient collection of both quantitative and qualitative data for sample cases at a number of large hospitals across the State.
Onsite at each hospital DMA confirmed and clarified with stakeholders their existing local processes, prior to conducting a retrospective data and document review of the relevant paper and electronic records. The review included onsite and follow-up discussion with local stakeholders and the client.
The final deliverable from the review was a well-received report assessing specific practices at each of the local hospitals, and providing comparative findings at the district and State levels.
The review provided independent assurance to the Ministry of general satisfactory policy compliance, and made recommendations for continuing monitoring and improvement of practices in delivery of the service.
Conducting a broad review of business cases for
Fire & Rescue NSW
The Program Management Office at Fire & Rescue NSW recently implemented a new procedure for capital funding requests, to provide more efficient and effective planning and allocation of the limited pool of capital funds. The procedure involved Directorates putting forward preliminary business cases to the Executive Leadership Team for consideration. Selected proposals were developed into full business cases for resubmission to the Executive Leadership Team for consideration for funding in 2018/19.
The Program Management Office commissioned Doll Martin Associates to assist the Directorates to review their draft business cases. It was felt that an external party well placed to provide objective advice and act as a mentor to the business case writers.
As draft business cases were received, our consultants quickly reviewed them and provided written feedback, followed by a short Q&A session. The writers were then able to revise their cases for final submission to the Executive Leadership Team.
Sixteen business cases received were for a range of subject matters, from fire stations and fleet to IT systems and specialised equipment. Funding requested varied from tens of thousands to multi-million dollars. The quality and detail in the business cases also varied, with some writers having the advantage of the Program Management Office’s training, or prior experience in developing business cases.
The reception to the reviews was very positive about the review process and responsive to suggested enhancements. One writer said that it was ‘the first time I have received thorough and constructive feedback’ about their business case and expressed their gratitude.
The review process also had the advantage that it empowered the writers. Rather than outsourcing the writing of business cases to a third party, the process sought to improve the skills of the authors who could apply lessons learnt to future business cases.
The Program Management Office also benefited as justification for expenditure was clearer, a wider range of options were analysed and the overall standard of the final business cases improved significantly. In addition, recommendations were made for improvements to the procedure and the templates.
‘I have found our interaction with Doll Martin and each member of the team to be extremely positive. The feedback we are getting from the business case writers has also been positive. I am very pleased by the service we have been getting and I think it has enhanced our internal process.’
Michael Taylor, Director Program Management Office
Engaging with the challenges of today's aged care sector
This project, for a major aged care services body, was initiated as a convergence of factors was placing the aged care sector under growing pressure and creating an environment where the risk of medication errors was increasing. At the same time as the workforce is contracting and aging, the proportion of high care patients and rates of polypharmacy are increasing, and up to a third of all patients admitted to hospital from residential aged care are admitted as a result of adverse medication events, many potentially preventable. The sector also wanted to identify the potential benefits that participation in the National Electronic Health Record could bring to the aged care sector.
The business case was informed by interviews and workshops with key stakeholders across Australia held to inform solution design, feasible options and the scope of costs, risks and benefits. The business case included a full financial appraisal and sensitivity analysis, Risk Management Plan and Benefits Realisation Register.
A readiness assessment was performed to assess current information and business processes, define future processes and develop gap closure strategies to address information gaps and ensure facilities could participate in eHealth initiatives. This informed the deployment approach and mechanisms to promote adoption of systems to enable participation in improved integration with the health sector.
The project involved an assessment of the policy and process impacts that the introduction of eHealth would bring to the aged care sector, considering issues such as privacy, consent, authorised representatives, family access, liability, interaction between aged care processes such as ACAT assessments and eHealth systems.
A communication strategy was developed that identified all stakeholders and the communication materials and tactics appropriate for each group. A training plan and materials were developed to address both the business process and information system changes that staff needed to implement.
Improved data quality for improved analytics
A large state government agency wanted to establish a program to monitor and improve the quality of the data used to measure performance of services across the state and determine funding allocations. Data was sourced from a number of systems, with variability in configuration, integration and local business processes. Methods of monitoring data quality and resolving issues that were identified were highly manual, and there was no way of making the reliability of data visible to users to inform decision making. The department needed an assessment of their current processes and a strategy and plan to implement more effective data quality controls.
The project involved an assessment of the strategic priorities for the agency and an assessment of the organisation’s information assets to identify the highest risk data to be the initial focus of the program. This involved the development of a methodology for the agency to review and update the quality program priorities in subsequent years. A data profiling tool was developed to support the risk assessment process. Existing processes to manage data quality were reviewed, a consolidated view of the current state documented and gaps identified to inform the forward program.
We developed a data quality plan that focused on transitioning from the current labour intensive methods to an analytics approach. An operational model was also developed to identify the governance and processes needed to maintain the program and monitor the outcomes that were being achieved. A methodology for scoring the reliability of individual data and performance indictors so that a quality scorecard could be linked to dashboards was also developed.
Leading a competitive market engagement process
A large volunteer based agency needed to replace its aging online infrastructure with a new online system for staff and customers within an organisational environment with diverse constraints, including time-intensive seasonal community activities, prioritised funds and tightly committed support resources.
Key issues facing the project included the need to identify and engage potential market suppliers, provide rigouros project estimates, test solution requirements, and engage confirm supplier cababiltiies all the while ensuring that the key internal business units were kept informed and were able to provide their valuble input into key decisions.
We established an approach that addressed these issues while maintaining cost efficiency and procurement probity. We presented the agency with two alternative procurement approaches. One was to issue a Request for Quotation (RFQ), in which a formal tender was to be developed and issued to vendors. The other was to conduct a Competitive Dialogue Process, where a preliminary statement of requirements is issued to pre-selected suppliers, seeking an indication on feasibility, their capability and interest. This informative process is iterative and culminates in the receipt of formal responses
The client elected to proceed with the Competitive Dialogue Process. and it provided opportunity to informally engage selected market suppliers to test requirements and identify current, emerging and innovative approaches. The agency saw value in the earlier confirmation of scope, budget, cost estimates which better informed interrelated business activities, such as the development of the business case and benefited from the improved shared understanding and cultural fit between potential suppliers and client through collaboration. The competitive dialogue process ultimately decreases negotiation timeframes, despite additional dialogue processes, due to improved overall project understanding by all parties.
The project saw highly engaged agency stakeholders and market suppliers and an improved project scope, informed by discussions and feedback. We were able to elicit quality procurement responses and comparable, competitively priced solutions for the agency which help deliver high quality procurement outcomes and efficient use of agency resources. The agency felt comfortable with this model of procurement, as the competitive dialogue process allows successful respondents to ‘hit the ground running’.
Development of an analytics architecture to support enhanced business intelligence
A human services agency needed assistance to develop a strategy to manage a large number of data collections required for evaluation of a number of priority Government programs as well as reporting, monitoring and planning, and to develop an analytics architecture. Data is sourced from a range of systems as well as manual data collections. An implementation roadmap for warehousing the relevant data collections was also required.
The data collection approach that was in use was very complex and there was considerable duplication across programs. The first step in the process was a review of the data collections and an assessment of the feasibility of rationalising and standardising these. Several options to go forward arose out of this review and these were analysed in light of stakeholder feedback on the constraints and capability of the department and the emphasis on change for the areas of real need.
Once an option was decided upon, we were able to provide a roadmap and resourcing plan for the agency, which involved reaching their desired goal state through an iterative process, aligning with departmental and government strategies at each step. This put the agency on a steady path to enhanced analytics capability while still ensuring that they had the flexibility to take advantage of the inevitable technological advances in the analytics space.
In total, the project delivered an assessment of the maturity of the current analytics capability, systems, tools and infrastructure and the goal state and of the source systems in use and the challenges with extracting data from these into the warehouse. It ensured strategic alignment between the broader agency initiatives and the programs under review. It provided a gap analysis between the current warehouse and the future capacity of this to meet the identified data requirements.
The development and evaluation of options to meet analytics requirements, with an assessment of dependencies, constraints, risks and costs was a transparent mechanism to decide on and develop the proposed architecture for the future state. This architecture was supported by a implementation roadmap which identified the activities to transition to this architecture from the current state. Finally, despite the large nature of the project, costing and resource requirements to implement the recommended option were given so that the project was given the best opportunity to proceeded without major administrative roadblocks or complications.
A privacy impact assessment for a multi-channel government customer management system
A large government department was developing a new customer management system, which was to act as a centralised point for staff, customers and third-party organisations. Each of these stakeholder groups would have an entry point to the system which was to act as a key platform for both the department's internal and customer-facing functions. Before finalising the development of the system, the department asked us to consider the system's compliance with the government privacy principles and inform them of the best way to tackle their areas of risk.
This project was to achieve two broad objectives: to satisfy governmental requirements when it comes to the collection of personal information through the system; and to satisfy customer expectations of how their personal information was to be stored and used. Complete transparency in the collection, use and storage arrangements was key to both of these, and therefore became the guiding principle for the project.
As privacy touches a variety of points in the development process, complexities arose as to where the privacy risk could be managed. What are the specific legislative requirements for the department? Were they addressed in the contract with the vendors? Can they be solved by changes in design? What was being communicated to both staff and customers about the collection of personal information? We were able to work through these complexities in a methodical and rigorous manner, guided by government privacy requirements but informed too by existing business structures and feedback from key stakeholders. The process was not a box ticking exercise, but an exploration of what enhanced risk minimisation and customer information protection practices should result from the new system's implementation - protecting both customers and the staff.
Over the course of the project, specific areas of risk for this particular department began to make themselves clear. The hosting of the system storage off-shore, the direct access customers would have into the system and the moderation of what they could upload there were some concerning issues that the Doll Martin team were able to address.
The end result of the project was a privacy impact assessment that gave the department the confidence that they not only complied with government privacy regulations, but that the new customer management system would be a privacy-positive move for the way they did business.