9.1 GENERAL

9.1.1 This CHAPTER describes the process of implementing a compliant management control system from the time that the requirement is indicated in a Request for Tender (RFT) until the contractor's systems have been accepted by Defence. The type of review may vary depending on the type of work to be undertaken and the scope may depend on whether the contractor's systems have been previously accepted.

9.1.2 Scope. When a contractor is required to demonstrate compliance with the criteria for the first time, the Demonstration Review process is followed. In other cases, the Subsequent Application Review (SAR) process may be followed. The objective of the Demonstration Review is for the contractor to demonstrate that its systems meet the Criteria and are being applied to the contract. The objective of a SAR is to ensure that, on a new contract, the contractor is properly and effectively using an already accepted system. It is not the purpose of the SAR to reassess the contractor's previously accepted system. Hence, the scope of a SAR is significantly less although the same procedural steps may be followed.

9.1.3 SAR. A SAR will be normally be performed in lieu of a Demonstration review when:

a. a contractor or subcontractor is using a management control system which has been previously accepted; and

b. surveillance by Defence confirms that the accepted system is being, or has recently been operated as required in the relevant contract.

A SAR is not normally appropriate when an accepted system is being transferred from one facility to another. In this case, a Demonstration Review will normally be conducted to ensure that the entire system has been transferred, and properly implemented.

9.1.4 Types of Review. The management control systems used during development and production may be significantly different. Hence it may be appropriate to conduct separate reviews for development and production phases or contracts. Location, and hence, system operators and users, different interfacing systems such as Manufacturing Resource Planning systems, and procedural differences are some of the elements which affect the need for separate reviews. However, depending upon circumstances, the contractor may request separate or simultaneous reviews of the systems proposed for both development and production contracts (or a contractor may have one system which embraces development and production phases).

9.1.5 Selection of Review Type. In determining the type of CSCSC review (development or production) to be accomplished, the following issues should be evaluated by the Review Director in consultation with the Project Authority and the contractor:

a. The primary basis of the contract (development or production) should be considered, but it should not override other considerations.

b. If the manufacturing effort in the contract is not true manufacturing (ie., model shop work) and there is no major difference in the way cost data are collected from the method used for the engineering effort, then the CSCSC review can be based on the application of a development system.

c. If the majority of discrete effort in the contract is identified as either engineering or manufacturing, then the identification of the CSCSC review as development or production should be self-evident.

d. If there is little or no manufacturing effort (eg, contracts for long-lead items, engineering services, or production planning), the contractor can apply either an accepted development or an accepted production system.

9.1.6 Contractual Basis. Where it has been determined that the contractor for a specific project must have management control systems meeting the Criteria, it will usually be indicated in the RFT. The subsequent contract(s) will require that the processes in this guide be followed and the requirements are summarised in ANNEX B of this document. In exceptional circumstances, the Criteria could be applied to an existing contract.

9.1.7 Criteria Check-List. The check-list and guidance in this document are used by Defence for evaluation of contractors' proposals for the implementation of a compliant management control system. They may also serve to provide useful guidance to contractors in preparing descriptions of the management control systems. Contractors are encouraged to follow the criteria check-list when preparing System Descriptions to aid assessment by Defence.

9.1.8 Phases of Review. The phases of a typical review cycle include evaluations prior to contract award (paragraph 9.2 et seq), an Implementation Visit (IV) after contract award (paragraph 9.4.1), Baseline Review at about 60 to 120 days after contract award (paragraph 9.4.2), Readiness Assessment (RA) (paragraph 9.4.3) and the formal Review leading to acceptance (paragraphs 9.4.6 et seq.). A redemonstration may be scheduled where necessary to examine any changes made by the contractor to rectify deficiencies. After acceptance, the review process continues through on-going surveillance (paragraphs 9.5.1 et seq. and CHAPTER 12).

9.1.9 Proprietary and Commercially­Sensitive Information. Extreme care must be exercised during the review process to avoid improper or inadvertent disclosure of proprietary or commercially-sensitive information (see CHAPTER 7).

9.2 PRE-CONTRACT ACTIVITIES

9.2.1 Request for Tender. When it is determined that a contract will require management control systems that meet the Criteria, appropriate clauses will be included in the RFT. Sample clauses appear in ANNEX A.

9.2.2 Tender Submissions. Where required by the RFT, each tenderer's proposal includes a description of the management control systems to be used. Normally, the contractor would be expected to propose use of existing systems provided that they meet the Criteria. Defence does not require existing systems to be changed except where this is necessary to achieve compliance with the Criteria.

9.2.3 System Descriptions. The description of the contractor's management control system must be presented in sufficient detail to show how compliance with the Criteria will be achieved and to facilitate review and surveillance. While the contractor's System Description is not required to follow the evaluation/demonstration review checklist (ANNEX D), it must address all items in the check-list. Specifically, it shall:

a. describe the management systems and their application in all major functional cost areas such as engineering, manufacturing and tooling, as related to the development of the work breakdown structure, planning, budgeting, scheduling, work authorisation, cost accumulation, measurement and reporting of cost and schedule performance, variance analysis and baseline control;

b. describe compliance with each of the Criteria and correlate checklist items with applicable portions of the System Description, preferably by cross-referencing appropriate elements in the description of systems with the items in the checklist (ANNEX D); and

c. describe the proposed procedure for administration of the Criteria as applied to sub-contractors.

9.2.4 Form of System Description. A contractor may elect to keep the CSCS System Description general, and rely on cross-referencing to internal procedures or policy manuals for a discussion of the details. In this case, the procedures and policy documents are to be referenced in, and considered a part of, the System Description. In specific instances, only portions of the referenced documents may be Criteria related from a control and revision stand-point. In these cases, the contractor and Defence may develop control procedures that will permit changes to the non-Criteria-related portions of those documents without prior formal approval. This will require the contractor to identify, in the System Description, those specific sections or portions of the internal documents that are related to the contractor's CSCS and require prior Defence approval of any change or deletion.

9.2.5 Management Systems Software. Formal identification of third party commercial management systems software is not required in contractor's system descriptions, either for initial tender responses, validation, or for later reviews. The specific mention of software is not necessary, but it is necessary to clearly identify and describe sub systems being used to meet the Criteria. The accepted System Description and procedures must adequately describe what sub-systems exist at the time of acceptance. For example, this includes inputs, outputs, files, cost account and work package formats, earned value techniques and interfaces among sub-systems. However, mentioning the name of such software in the System Description, when the intent is to clarify and describe the capabilities as mentioned above (and thereby reduce the amount of additional content needed in the Systems Description), is permissible.

9.2.6 Previously Accepted Systems. A contractor proposing to use performance management control systems previously accepted may satisfy the criteria requirement in the RFT by citing in the proposal the Memorandum of Understanding (MOU) (paragraph 51 and Annex G) or notification of acceptance.

9.2.7 Evaluation Review. Normally, for a new contract, a CSCSC evaluation is undertaken by the Review Director as part of the tender evaluation process. The evaluation is basically an analysis of the contractor's proposed management control systems so as to determine the probability of the systems meeting the Criteria. The review will include use of applicable parts of the check-list (ANNEX D). If a contractor has proposed to use a previously accepted system, the Review Director will confirm that the previous acceptance was of an appropriate type (Development/production) that compliance with all relevant Criteria was demonstrated and that a current MOU exists.

9.2.8 On-site Examination. An on-site examination of potential contractor's proposed systems will not usually be required during the evaluation review. However, when any aspect of the system is not clearly understood, an on-site examination of that part may be necessary to clarify the contractor's intent. Any such review will be coordinated with other relevant Defence authorities including the Tender Evaluation Board and Project Authority. Care shall be exercised during the entire evaluation review process to ensure that the contractor and Defence have the same understanding of the system described in the contractor's tender. Data examples, using actual data in the case of existing systems, may be required to illustrate systems procedures and data flow.

9.2.9 Coordination. If the potential contractor's proposed system is in use under an existing contract with Defence, coordination with the relevant Project Authority should be maintained during the evaluation review process. If it is necessary to review plans and reports of the other contract, concurrence of that Project Authority will be obtained.

9.2.10 Evaluation Report. Following the CSCSC evaluation review, a report will be prepared which addresses the extent to which each prospective contractor's System Description in the tender adequately describes compliance with the Criteria. Any deficiencies will be described as to their nature, extent and perceived impacts. This forms part of the tender evaluation report on which source selection is based.

9.3 CONTRACTS

9.3.1 CSCSC Contractual Provisions. Where a contract requires CSCSC compliance, reference will be made to this Standard to the effect that the processes described herein are to be followed. Sample contract clauses are in ANNEX A. They require, inter alia, that:

a. the contractor shall use and demonstrate management control systems which meet the Criteria;

b. the contractor shall obtain prior approval of changes affecting the accepted management System Description before incorporation, if required by a MOU;

c. Defence shall have access to pertinent records and data associated with the management control systems; and

d. the Criteria shall be applied to selected sub-contractors as required by the contract.

9.3.2 Review of Subcontractors' Systems. Review and acceptance of subcontractors' management control systems is normally the responsibility of the prime contractor. This may be performed by Defence in coordination with the prime contractor when requested by either the prime contractor or subcontractor. It will normally be subject to equitable adjustment between the prime contractor and Defence and the Demonstration Team will include representatives from the prime contractor. Such review and acceptance will be accomplished in accordance with the procedures in this document and may lead to Defence validation of the subcontractors' systems.

9.3.3 Subcontract System Acceptance. In the event that a prime contractor reviews and accepts a management control system, the prime contractor should provide the subcontractor a written statement that documents the acceptance. Such acceptance will not constitute Defence acceptance and does not apply to other contracts or subcontracts on other Defence programs.

9.3.4 Prior Acceptance. Instances will occur where contractors' proposed management control systems were accepted by Defence in another contract of the same type (ie, development or production) at the same facility. When that system remains under Defence surveillance, the contractor will not be required to undergo a full Demonstration Review for a new contract unless significant modifications have been made to the previously accepted systems, or surveillance reveals that the accepted systems have not been operated as required by contract. Where a full Demonstration Review is not required, a Subsequent Application Review (SAR) is conducted.

9.4 REVIEW TEAMS

9.4.1 The Commonwealth will provide a team of personnel to conduct on-site reviews of the contractor's management control systems. The purpose of these reviews is to verify that the contractor is operating systems which meet the Criteria.

9.4.2 Team Composition. A Review Director will normally be appointed by FASCEP. Normally this will be the Director of Project Management Systems or his nominated representative who in turn will appoint a Team Chief. Where appropriate, other agencies will be requested to identify representatives with appropriate qualifications to serve as team members. Exceptionally, non-Defence personnel may be included in Review Teams. The team includes:

a. Review Director. The Review Director is in charge of the review process with responsibility for overall planning and the conduct of Reviews, visits and assessments. The Review Director furnishes policy guidance and interpretation of the Criteria as required by the Team Chief.

b. Team Chief. The Team Chief is responsible to the Review Director for the conduct of the Review, the selection of team members and the supervision of team efforts in reviewing the application of the Criteria by a specific contractors CSCSC.

c. Team Members. Teams normally will be selected from suitably qualified personnel in Defence. Where possible they will include representatives from the applicable project office and should include those personnel who will be engaged in surveillance after acceptance. Members will be administratively responsible to the Team Chief during the period of the review.

9.4.3 Team Function. The team is responsible for a rigorous assessment of the contractor's compliance with the Criteria or, in the case of a Subsequent Application Review, application of a previously accepted system. Such assessment includes review of all management control techniques used by all organisational elements which perform work on the contract.

9.4.4 Team Organisation. Members will be responsible to the Team Chief for the completion of their assignments. To the extent possible, the Team Chief assigns tasks consistent with background qualifications of team members. However, the Team Chief will retain the prerogative to select and use any professional skills and methods considered necessary to accomplish an assignment adequately.

9.4.5 Members will be full-time participants during CSCSC reviews. However, the team may be augmented with functional specialists to assist in specific aspects of a review. Team size and types of expertise of members will be determined by the requirements; for example, the type of review, contract size, contractor characteristics, and project characteristics.

9.4.6 Qualifications of Members. Members should be appropriately trained (see below) and possess one or more of the following qualifications:

a. knowledge of the technical content of the project or contract;

b. knowledge of the principal engineering design and test requirements of the contract under review;

c. general industrial engineering/production control background;

d. accounting/auditing knowledge;

e. project planning and control experience;

f. management analysis and/or cost/price analysis experience; and

g. contract negotiation or administration experience;

9.4.7 Training Qualification. All members should receive specialised training, dealing with management control systems concepts, performance requirements, and interpretation, before participating as team members. Formal training in CSCS, and satisfactory completion of a Review Team Members' training course organised by Defence (or equivalent USA DoD training) are required. Such training should be supplemented by additional instruction to ensure the fullest comprehension of the task to be performed during the review. On-the-job training will be provided, when feasible, to enlarge upon background experience and classroom training for members not having previously participated in a review.

9.4.8 Review Techniques. ANNEX D provides a checklist for use by the team members in the examination of the contractor's CSCS. The checklist includes a restatement of the Criteria followed by specific questions or areas to be addressed by the Review Team. To clarify checklist items, formats should be developed by the contractor as illustrations prior to or during the review. ANNEX E provides typical formats for displaying team findings and supporting conclusions drawn.

9.5 DEMONSTRATION PROCESS

9.5.1 Implementation Visit (IV). As soon as possible after contract award, preferably within 30 days, representatives of the review team should visit the contractor's plant to examine the contractor's plans for implementing compliant management control systems. This visit provides an early dialogue with the contractor on the review process. The contractor will be expected to make presentations to explain system design. The IV team will examine selected documents and procedures proposed by the contractor and identify areas of non-compliance and potential problems. During the visit, the schedule for the readiness assessment and full-scale review will be developed.

9.5.2 Baseline Reviews. A baseline review is normally conducted by the Review Director in conjunction with the Project Authority about 60 to 120 days after the effective date of the contract. The purpose of these reviews is to confirm satisfactory progress in extending the CWBS (see ANNEX A) and the allocation of budgets to higher level CWBS elements and cost accounts as the basis for performance measurement and reporting.

9.5.3 Readiness Assessment (RA). The readiness assessment is a visit, usually of four or five days duration, by core members of the Review Team to the contractor's facility. It is normally conducted one reporting cycle (one month) before the formal review to assess the contractor's progress and to determine its readiness to demonstrate fully integrated and compliant management control system. It normally follows the same pattern as the formal review and assists both team and contractor preparation for the full-scale formal review. Any discrepancies should be identified for correction by the contractor.

9.5.4 System Description Examination. The contractor should have current written descriptions of the management control systems. Copies of these will be forwarded to the Review Director before the formal review. In the case of a Demonstration, the Review Team will examine the System Description to assess whether it appears to describe systems that, if properly implemented, should comply with the Criteria.

Formal Review

9.5.5 Preparation. The formal Demonstration Review or SAR should be conducted as soon as possible after contract award. The timing is normally subject to satisfactory progress during the Baseline Reviews and RA. Prior to the review, it is desirable (but not mandatory) that:

a. the contractor has developed schedules and a complete performance measurement baseline for the work to be performed under the contract;

b. the contractor has completed at least two complete monthly accounting periods of performance against baseline budgets and schedules, and has submitted reports required by the contract, including where applicable, Cost Performance Reports (CPR) for these periods;

c. each subcontractor required to comply with the Criteria or to provide Cost Schedule Status Reports (CSSR), has submitted at least one set of reports to the prime contractor; and,

d. obvious significant deficiencies in the contractor's management control system operation have been identified and corrected.

9.5.6 Procedure. It is the contractor's responsibility to demonstrate compliance with the Criteria. However the review normally follows a standard pattern to facilitate the process. The Review Team examines working papers and documents to ascertain compliance with the System Description and Criteria and documents its findings. To facilitate this, the contractor will be required to make available budgeting, work authorisation, accounting and other functional documents which apply to the systems being reviewed. The contractor will also make available all appropriate internal planning and control documentation required for a comprehensive analysis of the adequacy of the system in relation to the criteria and the work under contract. All documentation must be current and accurate.

9.5.7 The contractor will demonstrate to the Review Team how the management control systems are structured and used in actual operation. This entails interviews with relevant personnel. Pertinent extracts of the System Description and operating procedures must also be available to all relevant areas of the contractor's organisation. Detailed operating procedures should delineate: responsibilities of operating personnel; limitations on actions; and, internal authorisation required.

9.5.8 Activities. The review will normally consist of five basic activities. These are:

a. an overview briefing by the contractor to familiarise the Review Team with the management control system identifying any changes which have occurred since the management system was last subjected to a Demonstration Review or subsequent application review;

b. a review of the documentation which establishes and records changes to the contractor's baseline plan for the contract. This includes work authorisations, schedules, budgets, resource plans, and change records (including management reserve and undistributed budget logs). The purpose is to verify that the contractor has established and is maintaining a valid, comprehensive integrated baseline plan for the contract;

c. a review of the reporting of cost and schedule performance against the baseline plan, along with appropriate analyses of problems and projection of future costs. Also an audit of the procedures used to prepare Cost Schedule performance data from the lowest level of formal reporting to the Report to the customer;

d. interviews with contractor managers to verify that the contractor's systems are fully implemented and are being used in the management of the contract and that the management systems are operated by competent personnel; and

e. an exit briefing by the Review Team covering the team's findings. During this briefing any open discrepancies should be discussed along with the agreed upon corrective action plan which establishes responsibility and a time-frame for corrective action.

9.5.9 Duration. Duration of a review depends on the scope, type, number of team members and the contractor's competence in demonstration. Typically, a Formal Demonstration will require a team of about 15 members for about 10 to 12 working days and a SAR is completed in four or five days by a team of about eight members.

9.5.10 Compliance Assessment. The Review Team will follow the evaluation and demonstration checklist (ANNEX D) to achieve an orderly, comprehensive and conclusive review. It may employ sampling techniques when it is not practical to review entire systems. Based upon the best judgment and advice available, the Team Chief will identify the cut-off point in any test when he considers that sufficient evidence has been obtained on which to base conclusive findings.

9.5.11 Compliance and Corrective Action. Instances of failure to meet Criteria will be formally documented in Discrepancy Reports (DRs). The contractor must take corrective actions as necessary to achieve compliance with Criteria. Any areas to be re examined will be clearly identified to the contractor by the Review Director. A schedule for developing solutions and for a subsequent review to confirm compliance will be agreed upon by the contractor and Review Director. Where the corrective actions cannot be completed by a contractually required date, the Review Director will refer the contractual aspects to the Project Authority.

9.5.12 Management of Subcontractors. The prime contractor is normally responsible for the review and acceptance of each subcontractor's management control system that requires application of CSCSC unless Defence has accepted that responsibility. The Review Director may hold open the review of the prime contractor should he fail to adequately discharge his obligations in this respect. Alternatively he may elect to close the review even though all relevant subcontractor's management control systems have not accepted because of such factors as 'subcontract not defined', or 'insufficient work in hand to date to permit evaluation'. In these cases, the Review Team must determine, as a minimum, that the procedures for review of subcontractor's systems and the plan for subcontractor compliance are adequate. The Project Authority may require later confirmation that the prime contractor has properly reviewed compliance as a Phase II Surveillance item, or a follow-up review by the Review Team may be planned.

9.5.13 Subcontractor Review. The reviewing authority must notify the Project Authority, the prime contractor's Review Director and the prime contractor, as appropriate four weeks in advance of any demonstration of a subcontractor's management control system. Representatives from both Defence and the prime contractor shall be entitled to attend or participate.

9.5.14 Discrepancy Reports. Deficiencies found in the review process are categorised by level as follows:

Level 1. Significant Criteria non-compliant deficiencies which directly affect performance measurement; or deficiencies resulting from failure to implement an approved Systems Description.

Level 2. Less significant deficiencies which may affect performance measurement, but are not significant enough to require correction before the review can be closed; or minor deficiencies which have little or no impact on performance measurement.

9.5.15 Review Status. Reviews are either 'Open' (still in process) or 'Closed' (satisfactorily completed). The review is normally closed by the Review Director when the Demonstration is completed and all Level 1 DRs are closed. In either event, however, the contractor must initiate a corrective action plan to resolve problems in a timely manner. Under normal circumstances, Level 1 deficiencies will require Review Team participation to verify satisfactory corrective action while Level 2 deficiencies may be closed by the Project Authority as a surveillance action.

9.5.16 Report of Review. Within 30 days of closing the review, a complete draft of the review report will be passed to the contractor and the Project Authority for information. A final review report will be forwarded after completion and approval of all corrective actions. The format of the report is in ANNEX F.

9.5.17 Report Distribution. Each report will normally contain a statement indicating that it contains proprietary data or commercially-sensitive information owned by the contractor and distribution of copies will be limited. Contents will not be disseminated outside Defence without the agreement of the contractor.

9.5.18 Acceptance Procedures. Acceptance of the contractor's management control systems is decided by the Deputy Secretary (Acquisition and Logistics) on the basis of the Demonstration Review report. He will formally advise the contractor regarding acceptance of the system. The first acceptance of a system constitutes 'validation'. Following any acceptance a Memorandum of Understanding (MOU) will normally be executed between Defence and the contractor concerning continuing compliance (see ANNEX G).

9.6 COMPLIANCE AFTER ACCEPTANCE

9.6.1 MOU. After acceptance of a contractor's management control systems, the contractor's System Description should be updated as necessary to describe accurately the system as accepted. A MOU may then be executed. Pertinent features of the MOU (ANNEX G) are described below:

a. The MOU is basically a statement of intent on the part of the contractor that it will maintain its systems in a compliant state and, on the part of Defence that it will, subject to satisfactory surveillance, continue to regard the systems as compliant and not require a full Demonstration Review of that type for subsequent contracts.

b. The MOU contains references to a description of accepted systems and sub-systems; it identifies facilities and locations; and provides for Commonwealth access to pertinent contractor records and data for surveillance purposes. Provision is also made to permit changes to accepted systems.

c. An MOU may be executed after the contractor's management control systems are applied to a single contract requiring application of the Criteria or it may be developed without an existing or pending contractual requirement when requested by the contractor or Defence, provided that a Demonstration Review has been successfully accomplished.

d. Applicable Demonstration Reviews may involve any contract in the facility where performance measurement systems are applied, provided that the contracts selected will ensure that a representative appraisal of the contractor's system is made.

e. When an MOU is entered into between the Department of Defence and the contractor, it will be executed in the context of a review report and formal acceptance.

f. An MOU will normally be limited to a single contractor facility and may be limited as to application to development or production contracts.

g. A contractor may respond to RFTs requiring CSCSC compliance by citing the MOU in the proposal.

9.6.2 Surveillance. Surveillance to ensure that contractors properly maintain their systems after acceptance (Surveillance Phase II) is normally a responsibility of Project Authorities and is described in CHAPTER 13. Indications that a contractor's system fails to comply with any of the Criteria can be cause for scheduling another review and may result in cancellation of the MOU. Specific discrepancies discovered as a result of a subsequent review or normal surveillance procedures should be corrected immediately. Contractor proposed changes to accepted management control systems should be submitted to the Review Director through a relevant Project Authority.

9.6.3 System Development. Acceptance of a contractor's management control systems as meeting the Criteria is not intended to inhibit continuing innovations and improvement of its systems. However, contractors are obligated by MOU and or contract to maintain their systems in a state which satisfies the Criteria.