13.1 PURPOSE

13.1.1 Surveillance. Surveillance is the process whereby a contractor's management control system is subject to recurring examinations to ensure that it continues to meet the Criteria and, in respect of a particular contract, generates valid and timely data. Surveillance may be internal (by the contractor) or external (by the customer - normally Defence).

13.1.2 Guidance. This CHAPTER provides guidance for personnel responsible for surveillance of contractor's performance measurement systems. It is general guidance which should be adapted to specific situations as they arise. Detailed procedures for surveillance of a particular contractor's management control system should be developed by Project Authorities in consultation with the Review Director. Procedures should be consistent with this Standard.

13.1.3 The guidance furnished herein will provide assistance to:

a. Project Authorities in formulating surveillance plans and determining resource requirements;

b. surveillance personnel in accomplishing their surveillance functions; and

c. contractors in understanding the requirements of the Defence surveillance program and responding to them.

13.1.4 Surveillance Policy. In respect of contractors with contracts requiring Criteria compliance, it is Defence policy to:

a. perform recurring reviews to evaluate the effectiveness of a contractor's policies and procedures to assure that the contractor's management control system continues to meet the Criteria and generates valid data; and

b. base such reviews on recurring evaluation of internal management control practices and selective tests of internal and external reported data during the life of the contract.

13.2 CONCEPTS

13.2.1 Phases. Surveillance begins with the award of the contract, continues through system demonstration and acceptance (Phase I), and extends throughout the duration of the contract (Phase II). (See FIGURE 13-1.) Phase I surveillance is conducted in conjunction with the Review process by the Review Director in consultation with the relevant Project Authority. The surveillance plan for Phase II should be formulated by the Project Authority during Phase I and fully implemented after system acceptance. Phase II surveillance is the responsibility of the Project Authority who should appoint a Surveillance Monitor. To facilitate the coordination of, and to promote consistency in the implementation and surveillance process, policy guidance on surveillance activities should be sought from DPMS who should also be represented on surveillance activity from time to time during the course of a contract.

Evaluation of Proposals (Pre-Award)	Implementation Visit	Baseline Review	Readiness Assessment	Demonstration Review	Acceptance	Surveillance Phase II (Project Authority)
	Surveillance: Phase 1 (Review Director/Project Authority)

13.2.2 Objectives. The objectives of surveillance are:

a. To ensure that the contractor's management control system continues to:

(1) provide valid and timely management information;

(2) comply with the Australian Cost Schedule Control Systems Criteria;

(3) provide timely indications of actual or potential problems; and

(4) provide baseline integrity.

b. To ensure that the contractor's external cost and schedule reports:

(1) contain information that is derived from the same data base as that used by contractor's management;

(2) contain explicit and comprehensive variance analyses including proposed corrective action in regard to cost, schedule, technical, and other problem areas; and,

(3) contain information that depicts actual conditions.

13.2.3 Surveillance Scope. The scope of surveillance includes:

a. understanding the contractor's internal management control system;

b. monitoring the contractor's implementation of the management control system on the applicable contract;

c. participating in implementation visits, readiness assessments, and reviews; and monitoring the contractor's corrective action following each of these activities to bring the contractor's management control system into compliance with the Criteria;

d. monitoring, throughout the life of the contract, the continuity, consistency, reliability, and effectiveness of the system in operation. This function includes:

(1) assuring that the accepted system is in fact being used in the management of the project;

(2) evaluating changes to the accepted system to assure continuing compliance with the criteria or reporting standards;

(3) conducting periodic system reviews, evaluations, and tests to ensure that the quality of the accepted management control system is maintained; and

(4) informing the contractor of any uncorrected deficiencies which affect overall acceptability of the contractor's management control system, and requesting that corrective action be initiated;.

e. assuring that contractor-prepared reports (internal and external) identify current and potential problems;

f. reviewing, evaluating, and processing submitted contractor performance measurement reports; and

g. monitoring the contractor's corrective actions required as a result of surveillance.

13.3 ADMINISTRATIVE ASPECTS

13.3.1 Surveillance Responsibilities. The Project Authority will participate in reviews of the contractor's management control system and subsequently will perform the required Phase II surveillance. DPMS is responsible for providing general guidance regarding surveillance, conducting staff training, and resolving issues of compliance raised during surveillance.

13.3.2 Surveillance Plan. A surveillance plan will be prepared which describes how the Project Authority will carry out surveillance. Development of the plan may be discussed with the contractor, and if the Project Authority and DPMS have no objection, a copy of the plan may be provided to the contractor. This plan should be approved by the PA and should be implemented immediately after the Demonstration or Subsequent Application Review. Development and content of the surveillance plan are covered in paragraphs 13.9.1 et seq.

13.3.3 Accepted System Description. After acceptance of a contractor's management control system, the System Description is updated to reflect the accepted management control system. The contractor is then obligated to maintain the management control system in accordance with the accepted System Description. However, this is not intended to inhibit continuing innovations and improvement of the management control system.

13.3.4 System Changes. The surveillance effort must consider changes and improvements that the contractor may wish to make to his accepted management control system. Such requests for changes should be promptly evaluated for compliance with the criteria or reporting standards. The proposed changes will be submitted by the contractor to the Project Authority. Changes proposed by the contractor also require FASCEP approval. The Project Authority should advise the contractor of the acceptability of such proposed changes within 60 days after receipt from the contractor. A copy of the accepted change to the system will be forwarded to DPMS. If these vary the system as previously accepted and documented in the System Description, the Project Authority will endorse the proposed change and forward it to FASCEP for formal approval to vary the accepted system and System Description.

13.4 DEVIATIONS

13.4.1 Surveillance System Discrepancies. During surveillance, if the contractor's practices are found to differ from the accepted System Description or if unilateral changes to the accepted management control system have been made without Defence approval, the changes will be analysed and Project Authority who will notify DPMS and require correction of deficiencies.

13.4.2 Identification as "System Deficiencies". If deficiencies are discovered in the contractor's compliance with the accepted management control system, they will be identified as system deficiencies to differentiate them from specific contract problems. The contractor will be advised of the system deficiencies by the Surveillance Monitor. The Monitor will follow-up on a timely basis to determine when action taken resolves each discrepancy.

13.4.3 Notification of Deficiencies. Contractor's management systems should be as previously accepted by Defence. When surveillance personnel determine that the contractor's accepted management system is not as accepted, the contractor and DPMS will be promptly notified of the specific areas of deviation. DPMS is to be notified of all system and major program discrepancies and the Project Authority will seek advice regarding items of disagreement if the contractor disputes non-compliance. In those cases where problems cannot be resolved, the discrepancy will be elevated to FASCEP for resolution.

13.4.4 Appeals. The burden of proof is on the contractor to demonstrate that the management control system and its operation, in fact, comply with the system as accepted. If a contractor is notified of a discrepancy and the contractor disagrees, he may appeal through the Project Authority to FASCEP. If the contractor is unable to obtain agreement, he will be notified through the Project Authority to take corrective action within 60 days. The Project Authority will monitor the corrective actions. If inadequate action is taken by the contractor, the acceptance of the management control system will, if conditions warrant, be withdrawn by DEPSEC A&L.

13.5 SUBCONTRACTORS

13.5.1 Surveillance - Subcontracts. When a subcontractor is contractually required by the prime contractor to comply with Criteria, surveillance is a basic responsibility of the prime contractor as part of the total management of the subcontract. The Project Authority's function normally is limited to evaluating the effectiveness of the prime contractor's management of the subcontract. However, there may be occasions when a prime contractor will request Defence assistance to perform or assist in performing surveillance. Such assistance should generally be provided only when:

a. The prime contractor is unable to accomplish the required surveillance because it would jeopardise the subcontractor's competitive position or when proprietary data is involved; or

b. There is a business relationship between the prime contractor and subcontractor not conducive to independence and objectivity, as in the case of a parent-subsidiary or when prime and subcontracting roles of the companies are frequently reversed; or

c. The subcontractor is sole source and the subcontract costs represent a substantial part of the prime contractor costs.

13.5.2 Subcontract Surveillance - Responsibilities. When it is in Defence's best interest to perform surveillance for the prime contractor, such surveillance will be performed by the Project Authority at the subcontractor's facility. A Memorandum of Understanding (MOU) should be executed between Defence and the subcontractor which delineates surveillance activities and responsibilities.

13.6 SURVEILLANCE PERSONNEL

13.6.1 Personnel Qualifications. Individuals involved with surveillance should receive specialised training dealing with management control systems concepts, cost performance measurement requirements, interpretation of the Criteria and surveillance of management control systems at the earliest practical date. Normally such personnel should be qualified to participate in Reviews (see CHAPTER 8) and, where possible should have participated in the Demonstration or Subsequent Application Review for the contract under surveillance. Specific courses dealing with the planning and executing of a plan for surveillance, the analysis of contractor performance measurement data, and the maintenance of systems discipline within the contractor's organisation are available in the USA and are being introduced in Australia. All training should be supplemented by additional instruction and on-the-job training to enlarge upon background experience and classroom training wherever possible.

13.7 RECORDS AND REPORTS

13.7.1 Surveillance Monitor. A single individual is normally assigned the overall responsibility for the coordination and accomplishment of the total surveillance program. This individual is the Surveillance Monitor who should be selected by the Project Authority on the basis of background and knowledge. The Surveillance Monitor should also possess the ability to relate contract and program performance and assure that the data presented by the contractor to Defence are accurate, timely, and consistent with the contractor's internal data. When practicable, the Surveillance Monitor should be a Project Authority representative who participated in the Demonstration Review or SAR.

13.7.2 The Report. The Surveillance Monitor should prepare a periodic (normally monthly) report of surveillance activities and results. Although a thorough evaluation of the contractor's monthly CPR report may not be required, sufficient sampling of significant data items should be evaluated to assure data prepared by the contractor are timely, are accurate, and reflect the actual conditions. Reports should provide clear statements of the scope of review and any deficiencies noted, together with recommendations for their correction. Comments should also be provided regarding the results of discussions with the contractor's representative on deficiencies disclosed. To ensure that all pertinent data have been considered, the findings and recommendations should be discussed with the contractor when appropriate, prior to issuance of the report.

13.8 RESPONSIBILITIES

13.8.1 General.. Surveillance requires participation and full cooperation of the Project Authority, DPMS, and the contractor. If a surveillance program is to be successfully conducted, a spirit of mutual cooperation and proper rapport must exist among all interested parties in their interactions.

13.8.2 Project Authority. The Project Authority is primarily responsible for surveillance The application of the Criteria is not intended to replace any of the techniques, functions, or responsibilities normally undertaken by the Project Authority. However, it does facilitate the use of the more classical methods of contract administration. For example, the monthly CPR, where required by contract, shows the Cost Schedule status of the contract for the previous monthly report period, highlights significant Cost Schedule variances that have occurred and their probable causes. The data in the CPR quantify the magnitude of existing problems and potential problems and indicate Cost Schedule trends which are used for estimating contract completion costs. Reliable data in this format are very useful for effective contract administration as well as project decision-making. The responsibilities of the Project Authority in relation to surveillance of a contractor's system include the following:

a. participating in pre-acceptance surveillance activity (Phase I);

b. performing post-acceptance surveillance activity (Phase II) to ensure continuing operation of the contractor's accepted performance management control system;

c. requesting DPMS assistance in the resolution of contractor's systems problems and requested changes, and keeping DPMS informed relative to actions and matters which could affect system surveillance;

d. assisting resolution of problems cited in surveillance reports by providing required support to the Surveillance Monitor; and

e. apprising DPMS of the adequacy and usefulness of surveillance reports, and where necessary, coordinating required changes to reporting practices.

13.8.3 The Surveillance Monitor. The Surveillance Monitor will assure that the results of surveillance program efforts are documented and maintained as part of a chronological record of the contract. A surveillance file will be established to contain all pertinent data and information regarding the surveillance program. The file should include areas reviewed, findings, actions taken, and results.

13.8.4 The Surveillance Monitor has the following responsibilities:

a. assuring Project Authority coordination with the Review Director in the preparation of surveillance plans to assure that surveillance is performed in a systematic manner;

b. executing a program of surveillance to assess continuity and consistency in the operation of the contractor's accepted management control system;

c. performing recurring evaluations of the effectiveness of the contractor's policies and procedures;

d. performing selective tests of the contractor's cost and schedule data flow and external performance measurement reports to determine validity of reported data;

e. assuring that the cost, schedule, and contract-related financial and program status reports submitted to the Project Authority are timely and accurate, and depict actual conditions;

f. calling upon the assistance of DPMS if required to assist in accomplishing the surveillance plan;

g. assuring accurate and adequate files are maintained relative to surveillance matters;

h. acting as the point of contact in matters relative to cost and schedule control surveillance within the Project Authority;

i. assuring that DPMS is fully advised of status of cost and schedule control surveillance and any major problems pertaining thereto; and

j. preparing and submitting surveillance reports.

13.8.5 Other Surveillance Personnel. There are a number of other Defence personnel who examine various aspects of the contractor's capability to complete the contract successfully. These include other Project Staff examining engineering, production, finance aspects, quality assurance personnel and DPMS personnel undertaking performance analysis. The Surveillance Monitor should maintain regular contact with these personnel to discuss items of interest or concern to enable the planning of future surveillance effort. These discussions will enable the Surveillance Monitor to determine the scope, depth and areas of surveillance activities for the subsequent period. The Surveillance Monitor may also be able to suggest to the other personnel certain areas of the contractor's system that may need more in-depth examination.

13.9 PLANNING AND PERFORMING SURVEILLANCE

13.9.1 Planning for Surveillance. Planning for surveillance should begin as soon as it is anticipated that a contract will be awarded. Active surveillance should commence immediately after contract signature (Phase I, the prime responsibility of DPMS) to ensure that management control system implementation is satisfactory and to highlight any obvious system deficiencies. Continuing surveillance (Phase II, the prime responsibility of the Project Authority) should be directed toward all procedures and functions of the contractor's cost and schedule control system. From immediately after contract award, through all phases of system implementation and demonstration, and until system acceptance, activity should be devoted to gaining a full understanding of the contractor's management control systems, to monitoring the implementation of CSCSC, and to planning and developing the comprehensive surveillance plan for Phase II. The surveillance plan should define the surveillance to be performed.

13.9.2 Development of the Surveillance Plan. Because the Criteria do not prescribe a specific management control system, each Project Authority will be monitoring a unique system consisting of different scheduling, budgeting, cost accumulation, etc. sub systems. The main purpose of the surveillance plan is to provide an organised and comprehensive set of guidelines and techniques for use by cognisant Project Authority personnel in performing surveillance on the management control system. Primary considerations in the design of the surveillance plan are the specific contractor management control system being evaluated, the contractual requirements, the desires of the Project Authority, and the availability of qualified personnel.

13.9.3 Content of the Surveillance Plan. The surveillance plan will normally consist of two basic sections, one devoted to general guidance and management responsibilities, and one devoted to specific procedures and techniques. The first section should describe organisation responsibilities, reference documents, frequency of reports, the review cycle and other general administrative information. The second section should outline and discuss techniques of accomplishing surveillance, tests to be used, areas to be evaluated, and functional skills within the Project Authority to be used.

13.9.4 Requirements of the Surveillance Plan. The intent should be to effectively examine the contractor's complete system at least once during a 12 month period. This will require:

a. evaluating all the important features and disciplines of the contractor's accepted management systems;

b. performing this evaluation in each involved major functional group of the contractor's organisation; and

c. performing this evaluation in the most active areas of the work breakdown structure.

The surveillance plan should contain procedures for conduct of surveillance throughout the life of the contract. However, the plan should not be so rigid as to result in routine mechanical reviews. Instead it should be flexible and require periodic re-evaluations to determine redirection of emphasis necessary to meet changing conditions. It should provide for adjustment in effort and shift of emphasis as the program progresses and as familiarity with and confidence in the contractor's management control system is gained.

13.10 SURVEILLANCE DURING PHASE I

13.10.1 Since contract decisions must be made from the day of contract signature, contract administration including surveillance, must also begin upon contract signature, to assure the Project Authority that the provisions of the contract are being met despite the fact that the contractor's management control system has not yet been reviewed or demonstrated and found acceptable. This is the primary responsibility of the Project Director.

13.11 SURVEILLANCE DURING PHASE II

13.11.1 Objective. Phase II surveillance should ensure that the contractor's management control system continues to meet the contractual objectives. During Phase II, surveillance personnel should concentrate their activities on management control system reviews, and evaluation of contract data and reports.

13.11.2 Surveillance Steps. In evaluating the contractor's management control system during Phase II, surveillance personnel must always remain cognisant of the policy governing the requirement for CSCSC. In order to assure that these requirements continue to be met, surveillance personnel may follow a number of surveillance steps:

a. Evaluate the Management Control System. Review the contractor's practices to assure they are in consonance with the accepted System Description. As part of the demonstration review process, each contractor submits a formal description of the accepted management control system supported by detailed operating procedures. Once accepted, the System Description and related procedures form the basis for the review of the actual operation. These documents should be reviewed and tests performed to determine if the contractor's practices comply with the stated procedures, and if management utilisation of the system and data is appropriate. In the course of surveillance, the Monitor should be continually alert to contractor practices, procedures, and systems that do not meet the relevant requirement.

b. Evaluate System Changes. Evaluate all changes to the accepted system. The Surveillance Monitor must be made aware of all changes to the contractor's management control system. Changes will be evaluated as to compliance with the Criteria, impact on the integrity of the management control system, effect on contractual provisions, and cost of implementation. The proposed changes should be subjected to an immediate and exhaustive evaluation by DPMS, the Surveillance Monitor, and the Project Authority to determine acceptability and to allow for rapid implementation if approved. The purpose is to detect those changes to an accepted management control system which are not in compliance with the Criteria and may therefore impact contract requirements. In addition, surveillance personnel should always be concerned that the System Description accurately describes the accepted system and be vigilant for unauthorised contractor departures from the accepted system. Deviations should be brought to the immediate attention of DPMS, the Project Authority, and the contractor.

c. Verify the Data Base and System Discipline. On a recurring basis, surveillance personnel should perform evaluations as to the validity and traceability of the contractor's cost and schedule data base. By performing certain selective tests of the contractor's cost and schedule data flow and by comparing the results with other appropriate internal and external data reports, surveillance personnel are able to ascertain the accuracy of the contractor's data base, and the discipline of both the contractor's management personnel and the management control system involved. In addition, by tracing the cost and schedule data flow the Monitor is able to determine that all applicable sub-systems related to cost and schedule control are integrated and use the same data source.

d. Verify Reconciliations. Contractor reconciliations of appropriate financial data should be verified periodically to assure that data presented in various external reports and documents are valid, reconcilable, and traceable to other external financial reports and to cost and schedule data bases in the contractor's management control system. Differences isolated in the data must be explained consistently and logically. The mechanics of the contractor's procedure for reconciling data should be reviewed in the early stages of contract surveillance. After attaining assurance that reliable procedures are consistently followed, such verifications should be required less frequently.