Industry Resources
Earned Value Management - Frequently Asked Questions
EVM Home | EV Primer | Policy Documents | Training| FAQs | Contacts
|
Tender Evaluation Reporting And Variance Thresholds Common Contract Issues Earned Value Reviews Payment By Earned Value |
Many of these FAQs were originally generated as DMO project office questions
but, in the interest of maintaining a common understanding between the Commonwealth
and industry, are included on this Industry Resource site.
Tender Evaluation
TOP
Our company has a previously validated system (may be validated against the U.S. Standard, the Australian Defence Standards or other such as the UK Standard). How is this addressed?A previous validation is not a guarantee that you are still applying EVM effectively. An EVM system rated against other EVM standards should also meet the requirements of AS 4817-2006, and the Defence Supplement to AS 4817-2006, since the requirements are essentially the same across all the standards. The provision of a validation certificate, or some other proof that an EVM System has been found to meet EVM requirements, can be used as a risk mitigation factor when deciding upon the EVMS review strategy. It is highly desirable to finalise the review strategy (tailoring) prior to contract signature. For example, while each contract will require an IBR and an EVMS review it may be possible to tailor the scope of the reviews based on corroborated past experience, contractor history, etc. These, and other issues, should be discussed and agreed with prior to contract signature. Further information on tailoring the review process can be found in the EVMS Review Handbook.
What if we do not want to use EVM?
The use of EVM in contracts will be based on contract risk. The greater the
contract’s risk the greater the need for a good performance measurement
system by which you, the contractor, manage performance. The DMO guidance is
for projects to utilise EVM for all acquisition contracts valued at more than
$20m. It is possible to gain an exemption but only where the Project Authority
provides adequate justification and demonstrates that the risk is low.
The risk for contracts valued at less than $20m may also justify the use of EVM. The project office will evaluate the contract risk profile to determine if application of EVM is warranted. The risk items to be considered will include: contract cost, duration, capability importance, schedule, technical issues, payment method, reporting requirements and project resources.
The majority of Australian and U.S. Defense Industry have already adopted the use of EVM. Most companies not utilising EVM are still planning the work, measuring progress, and collecting data on actual costs, which would imply that minimal tweaking would be required to integrate this data and put in place the controls that are necessary to provide a DMO compliant EVM system. It is possible at the lower end of the risk spectrum that additional tailoring of requirements may be possible.
If we submit a poor quality draft Work Breakdown Structure, how will
this affect the EVM System?
The Work Breakdown Structure (WBS) is the cornerstone of a good EVM System as
it is the link to the scope of the contract, the planning of the work, and reporting
of progress and actual costs. A poor WBS will provide poor information on progress
and variances against the plan. A poor draft WBS may also be an indicator that
your company does not understood the scope of work. Finalisation of the WBS
must represent the agreed position of both the project office and the contractor.
The document must be useful to both and representative of the way the you intend
to undertake activity under the contract.
If we do not submit a well-developed and integrated draft Contract
Master Schedule, is this an issue?
The draft CMS is reviewed in several areas of the project office. A poor draft
schedule may indicate a lack of scheduling skill and may also indicate a lack
of understanding of the scope of work. It is essential that the final schedule
be horizontally and vertically integrated, that the critical path is identified
and that all dependencies are linked, (e.g. predecessors and successors). The
Contract Data Item Description for the CMS requires visibility down to the Work
Package level. However, it is important that the project office does not require
onerous linkages at the lowest level. Where there is subcontractor work it is
expected that at least milestones, key events and performance indicators will
be integrated into the prime contractor CMS. It is not necessary for entire
subcontractor schedules to be input to the CMS. It should also be noted that
the statused schedule information will feed into the Commonwealth’s project
schedule (refer to IPSSR and/or PMMv2).
While an EVM System may highlight a schedule variance against the plan, it is in dollar terms not time and therefore the schedule must then be reviewed to ascertain the length of the delay and the impact if any on the successor activities. Special emphasis should be placed on those activities which lie on the critical path.
How important are the contractual milestones in relation to an EVM
System?
Contractual milestones must be included in the Contract Master Schedule and
identified in the Work Breakdown Structure Dictionary. Contractual milestones
are, in part, intended to incentivise you to meet particular requirements by
a specified agreed date. The requirement for EVM does not negate the importance
of milestones but provides additional information on progress for the whole
program. This also enables a project office to identify where scope for a particular
milestone may be getting progressed at the expense of scope for other critical
path items.
What standard requirements are open to negotiation with the project
office?
The project office will seek expert advice from the contracting section and
domain knowledge holders prior to negotiating clauses in the Terms and Conditions
of Contract or the Statement of Work.
If the contract is considered a low risk/short duration/production type of contract but it meets the minimum threshold of $20m, then it is possible to seek an exemption to the requirement for EVM.
Items that cannot be negotiated out but may be tailored to suit the project office and contractor include:
- reporting level (refer ASDEFCON Handbook);
- reporting requirements; and
- variance thresholds.
The reporting requirements can be tailored to suit the needs of the project office. The ASDEFCON Handbook provides details on Earned Value Performance Reports (EVPRs) which are available in five formats. It is possible to remove the requirement for EVPR Format 2 which provides progress data against performing organisation, although this report is particularly important when there are several large subcontractors. Format 4 is another report format which can be removed. However, while Format 4 is often not well developed or properly understood, it is extremely useful to both the project office and the contractor when properly prepared. Removing any of the reporting requirements should be done only when the risk of removing these items is assessed. Formats 1, 3 and 5 are not negotiable unless you can provide other reports that provide better data.
What is the Tender Evaluation process for EVM?
The EVM requirement lies within the Project Management aspects of the Tender
so the Tender Evaluation Working Group (TEWG) assigned to evaluate the EVM part
of a tenderer’s response will logically reside with the Project Management
TEWG. The EVM component will include a detailed review of the draft EVM plan
and a cursory review of the CWBS and CMS. These areas are normally reviewed
in detail by other areas of the project office but, due to their importance
in EVM, it is necessary to review these draft documents to ensure that the basis
for the EVM system will be adequate. There are other areas that will be reviewed
including but not limited to the draft contract for the access clauses, subcontractor
flow down, and other measurement aspects.
The tender evaluation team will review the following:
| General Tender Review – EVM | Comment |
| Compliance matrix for any non-compliance statements regarding EVM | This is a quick check to ensure any further effort is warranted. If tenderer does not intend on implementing EVM this would be a high risk and should be subject to clarification with the tenderer. This would be grounds for setting aside a tender as EVM is required for medium to high risk contracts. |
| Has an EVM Plan been provided? | If there is no EVM Plan then check to see if the items in an EVM Plan are covered elsewhere, for example, in the Project Management Plan. If there is no plan then it would appear that the tenderer has no plans to implement an EVM System (EVMS). This would be a high risk and should be subject to clarification with the tenderer. This would be grounds for setting aside a tender as EVM is required for medium to high risk contracts. |
| Project Management Plan | There should be some mention of the EVM Plan and how it integrates with other plans. |
| CWBS | The CMS should be linked/based on the CWBS. Milestones such as IBR should be visible in the CMS and meet the requirements of the CMS DID. |
| CMS | The personnel to control the EVMS should be identified and should have appropriate project management, EVM, and scheduling skills. |
| Key Personnel | Where proposed by a tenderer advice should be sought from Contracting Policy. |
| Is Payment by Earned value being proposed? |
The EVM Plan at tender is not expected to be a complete document as this is required at or shortly after contract signature. The EVM Plan at tender stage should:
| EVM PLAN | Comment |
| Describe the tenderer’s use of EVM | It is necessary to know the maturity of the tenderer’s EVM System, as this will be one of a number of indicators when planning the EVMS Review process. Refer to the EVMS Review Handbook for further information. |
| Describe how EVM is to be implemented for this contract, including the tenderer’s proposed system assurance strategy. | This will be another consideration when deciding on a review strategy. An assessment of the adequacy of the proposed implementation plan is required here to assist in choosing the appropriate review strategy. |
| Outline the proposed timetable for implementation of the EVM System, including the Implementation Workshop and the Performance Measurement Baseline, culminating in the Integrated Baseline Review | The tenderer is best placed to choose an appropriate timeframe. However, since work commences at contract signature, a detailed plan should be developed and available for review as soon as practicable. See the reviews section of the FAQs for further information. |
| Discuss the intent for subcontractor flow down and management, including the proposed subcontractor reporting regime. | EVM should be flowed down to any subcontractor where subcontract price will exceed $20m and contract duration exceeds 12 months. The decision to flow down EVM should be risk based. Therefore, if a subcontract is <$20m but is a critical or high-risk element of work, flow down of EVM should be considered. Likewise for a contract over $20m but mainly production or hardware and low risk, EVM may not be required. The flow down should be discussed and agreed by the project office. |
| Provide a proposal for reporting including proposed reporting levels and variance thresholds | Many tenderers, especially those already using EVM, have developed their own internal reporting regime. In some instances a mix of standard EVM reports plus contractor provided reports (e.g. various specific software reports) will be of most value to the project office. It would always be advisable to have an equivalent of EVPR Formats 1, 3 and 5. Further information on reporting level and variance thresholds can be found in that section of the FAQs. |
| Provide the tenderer’s history with EVM | No history implies no previous use of EVM – full EVMS review required. If they have a well-established EVM system – low risk. Tailor reviews to suit the project requirements and the contractor’s EVMS strength and weakness. See the EVMS Review Handbook for details. |
| Provide a proposal for internal quality assurance of the implemented EVM System | The contractor shall develop and propose an internal EVMS quality assurance strategy which the contractor will implement to ensure that the EVMS remains compliant. This does not negate the need to conduct external EVM System Assurance Reviews, but is a determining factor in deciding the timing and methodology to be used by the project office when conducting System Assurance Reviews. |
Reporting And Variance Thresholds
TOP
How will the reporting be different if we provide online access?Paper reporting may be reduced by providing online visibility to earned value data. This is particularly useful where there is a project office resident team and may reduce the cost of reporting if online access is available. Online access provides improved visibility due to the timeliness, however sensitivity must be exercised by the project office when using online data. Online data must not be used inappropriately by the project office and an awareness of the contractor’s management processes is important, e.g. variances addressed monthly/fortnightly/weekly, how often is the schedule updated and when does it occur in the EV reporting cycle. Where a good relationship exists between the resident team and the contractor paper reporting may be streamlined to provide a highlight or exception report. An exception or highlight report will not be sufficient reporting for a payment by earned value claim.
Is it acceptable to provide reports at level 2 of the WBS?
The reports are to provide the Project Authority with visibility of the plan
and progress against identified WBS elements of work. A report at too high a
level will provide very little visibility and will not provide adequate information
about progress and variances against the plan. High level reports disguise variances,
e.g. a schedule variance indicating poor performance in one lower level element
may be washed out by good performance in another element. At the same time a
report at too low a level will provide a great deal of unnecessary paperwork.
It is important to choose a meaningful level for reporting. This level does
not have to be the same level across the WBS or for particular phases. For example,
if there is a design phase and a production phase the need for great detail
in the production part of the WBS should be less than during the design phase.
More detailed reporting may be required for a high-risk element regardless of
the level of the WBS in which the element resides.
Final decisions about reporting levels are difficult to make prior to receipt of the detailed CWBS but the contract will allow for the reporting levels and variance thresholds to be changed. The contract also provides for access to further data at lower levels upon request. The adequacy of the reporting levels is reviewed at the Integrated Baseline Review.
Can the Earned Value Performance Reports be delivered electronically?
The ASDEFCON template CDRL states that the EVPR should be delivered in both
hard and soft copy. The soft copy should be delivered in a manner that allows
for easy input into Analysis tools such as wInsight. The soft copy can be provided
in an MS Excel spreadsheet, xml format, or an X12 format (from COBRA). This
requirement is detailed in the Earned Value Management Plan (DID-PM-MGT-EVMP)
and will ensure that the data provided can be utilised by the project office.
The project office will be cautious not to accept data in electronic format
with the data inserted as pictures instead of tables or spreadsheets. Once the
data has been converted to a picture it cannot be utilised for electronic transfer
into analysis tools. Further, reports provided in a PDF format lose their formatting
structure when converted back to MS Word or Excel. It is also possible to have
the data available online where you have arrangements enabling online access
to the project office personnel.
What should be the timing for the delivery of Earned Value Performance
Reports?
The frequency and the timing of reports are usually a negotiating point. The
project office would like to get the reports as soon as possible after month
end and the you may want as much time as possible after month end to incorporate
subcontractor data and to follow up variance reporting. It should be possible
in all cases for the reports to be delivered within ten working days or less.
Some tenderers and project offices also arrange that the EVM data (EVPR Formats 1 and 2) are delivered within days of the end of month with the analysis and other report data, e.g. variance reporting, following the next week. Where a resident project team is in place online access to the earned value data will enable a regular review of progress and may reduce your costs for reporting. A prime factor driving report timing is the difficulty of getting timely information from your accounting system. Most contractors are able to provide EV data well in advance of Actual Costs (AC). This may be an option for an interim report for you and the project office to obtain timely schedule performance data.
One reason for wanting to delay reports is the difficulty in arranging for any subcontractor’s data to be included in the reports in a timely manner. Subcontractor reports may be staggered as above, with the earned value data being received in advance of the variance reporting. It is possible to have a mix of subcontractors where some are able to provide data early enough to allow analysis and input to your baseline (and reports) and others that may lag one month. While it is not ideal to be getting the subcontractor information several weeks late this may be an option. You should attempt to align subcontractor delivery of reports with the end of month cycle to enable a timely delivery to the project office.
The other timing factor that can be negotiated is more frequent delivery of earned value data. Historically the reports have been provided monthly but there has been a recent trend with the easy delivery of data by electronic means for at least EVPR Format 1 to be provided weekly, or fortnightly, for critical, high risk elements of a contract or during critical phases of the contract.
This is a fixed price contract, why does the Commonwealth need access
to actual costs?
An Earned Value Management System is unique in that it integrates cost, schedule
and technical aspects of a contract. Reporting against a schedule may give schedule
variance, but there is a big difference when looking at EVM data that also provides
the cost factor. For example, a contract that has negative schedule and cost
variance has different issues from one where there is negative schedule and
positive cost variances. Actual costs, when combined with Planned Value and
Earned Value, also enable trend analysis and evaluation of estimates of cost
at completion at all levels of the contract. Actual costs are an important part
of the big picture revealed by an integrated system and are not to be written
out of the contract. Actual costs are an important indicator of risk, e.g. if
you cannot afford to complete the contract due to cost overruns (liquidity of
the company) it is highly likely that product quality will suffer in an attempt
to reduce overruns in actual costs.
Variance thresholds - what is recommended?
The variance thresholds are vital to good variance reporting. It is important
to establish reasonable variance thresholds that will cause problem analyses
and narrative reports to be prepared. Careful selection of these thresholds
is needed to prevent unnecessary work associated with preparing an excessive
number of written analyses. The analysis of every cost and schedule variance
is unnecessary and unproductive. It is advisable to use the project risks to
assist in looking at the thresholds and aligning the thresholds to provide the
most value. For example, if a contract includes design, test and evaluation,
and production of many items, and the design and test and evaluation are seen
as the risky areas, then it would make sense to have lower thresholds for these
phases (triggering more variance reporting). The thresholds can also be set
lower for particular WBS elements that are risky, e.g. software development.
Setting of thresholds requires some thought by you and the project office and
once set should be reviewed as necessary during the various phases of the contract.
We believe our software measurement data is all that is needed, so
why is an EVPR necessary?
The type of reporting to be provided should be meaningful and meet all the company
project management and project office needs. It is unlikely that software measurement
data alone will meet all these needs even for a software intensive project.
The monthly EVPR will provide progress detail using objective measurement techniques
which should be based upon the detailed software measures identified. EVM and
software measurements are not exclusive but complementary. The use of software
measures will add integrity to the earned value techniques which will show overall
progress on the contract (not merely the software aspects). This applies equally
to other measurement types including Technical Performance Measures (TPMs).
Common Contract Issues
TOP
We want to base our system on current day dollars. Is this acceptable?The preference is for base date dollars as specified in the conditions of contract for the financial basis, although it is not mandated for EVM. Reporting in base date also facilitates input into analysis software used by the project office for reporting to senior management. If you insist on providing current day dollars then you must also provide visibility of the process of escalation against the various EVM elements such as planned value, earned value etc., just as you would provide visibility of the process of de-escalation to base date dollars. Reporting current date dollars will also mean continual, regular changes to the baseline.
Note: Payments on contracts are made in base date dollars so for those contracts utilising payment by earned value it is highly recommended that the EVM system be in base date dollars also. This will allow for easier translation into payment formulae and for keeping project financial records.
We want to provide the reports in price basis. Is this acceptable?
By removing profit from the Contract Price the balance then makes up the Project
Budget (defined EVM term which incorporates the Performance Measurement Baseline
and the Management Reserve). Reporting in price basis would mean no visibility
of management reserve (used for unforeseen work and risk) and inconsistent application
of profit across elements may disguise true performance which is an unacceptable
risk for the project office.
We, and our subcontractors, do not wish to provide access to cost data
during reviews. Is this acceptable?
It is a requirement that the reporting be in cost, not price. To enable an accurate
review to be conducted this means that access must be allowed. The access clauses
in the contract will not be removed.
Our proposed subcontractors will be implementing an EVMS but are not
prepared to report ‘cost’ information to our company. Is this acceptable?
This represents a common situation in that commercial sensitivities limit companies’
desire to provide this level of information to other companies where they may,
in the future, be competitors. Project offices are aware of this type of sensitivity
and are willing to consider alternative arrangements. One arrangement often
used is where the subcontractor reports priced-based EVPRs to the prime while
providing cost-based information to the Commonwealth. This allows the Commonwealth
the visibility necessary for management while the prime contractor is still
able to maintain the required degree of reporting to the Commonwealth (subcontract
‘price’ to the prime contractor equals ‘cost’ to the
Commonwealth). This arrangement is particularly useful where significant effort
is being undertaken by subcontractors and the Commonwealth wishes to maintain
detailed visibility of this effort.
We do not plan to flow down the earned value provisions to subcontractors.
Is this acceptable?
Given the major role played by subcontractors in most major projects, the requirement
for subcontractor flow down should be carefully considered. Its purpose is to
enable visibility of subcontractor planning and progress and should be seen
as a major risk mitigation factor, especially for risky subcontractors. If a
subcontract is >$20m, mainly procurement in nature and, is considered low
risk then an exemption could be granted. Likewise, for a subcontract <$20m
but considered high risk, the flow down of EVM requirements would be appropriate.
We are an overseas-based contractor where EV is not well established
or plan on subcontracting with one. How does this influence your EVM requirements
approach?
EV is established or at least understood within most countries where the DMO
has contracted in recent years. EV is well established in the U.S., the UK,
Canada and Sweden while some companies have experience in France, Germany, Israel
and Italy. Where companies operate in an environment where EV is not well understood,
greater emphasis will need to be placed on ensuring the requirements are clear
and understood. Further, tailoring of both the requirement and the timetable
for system implementation will need to take into account the company’s
current operating methodologies, experience and the contract scope of work.
Earned Value Reviews
TOP
Is an Implementation Workshop necessary?The idea of the workshop is to answer any questions and clarify any issues that you and the project office may have in relation to the EVM requirements. Issues and agenda items may also arise from the development and review of the initial EVM Plan (implementation plan). Many issues will be resolved prior to contract signature, but during implementation of an EVM system other issues often arise. The workshop is also a preliminary step on the way to the Integrated Baseline Review and EVMS Review. The value of the workshop will vary depending on the maturity of your EVMS, and the format of the workshop will vary to suit the contract, your company, and the project office. If there is seen to be little need for the workshop then it may be advisable to include EVM System implementation and preparation for reviews as an agenda item in the Start of Work meeting. Refer to the EVMS Review Handbook for further information.
When should reviews occur?
Normally the review dates for IBRs and EVMS Reviews are negotiated and become
contractual milestones while System Assurance Reviews are conducted based on
an ongoing risk assessment. It is therefore very important that the IBR and
EVMS Review milestone due dates be realistically established. The timing of
the reviews will to a large extent depend on the process maturity of your company
and any critical subcontractors. It is possible to have an IBR within two months
of contract signature where there is no subcontractor flow down and you have
a well established EVMS. If there are subcontractor baselines to be incorporated
then the timing for the IBR will be delayed until the subcontractors have undergone
successful IBRs. It is important to conduct the IBR as early as possible since
it provides a valuable insight into the entire plan and forms the basis for
reporting progress. You need to justify any delays to the IBR schedule. Refer
to the IBR Handbook for further information.
Earned Value Management System (EVMS) Reviews would normally be conducted a few months after an IBR. Where there are subcontractors requiring EVMS reviews this will be a factor in the timing of the review. Companies who have never used EVM before may need a number of months to fully implement a system, train staff, and begin using the EVMS. Refer to the EVMS Review Handbook for further information,
Who will be on the review team and how will it be conducted?
There is guidance on the selection of team members and their roles as well as
the format of the review in the IBR Handbook and in the EVMS Review Handbook.
The key difference in the team composition between the IBR and the EVMS review
is the change in focus between a technical review (the IBR) and a system review
(the EVMS review). The review team is usually led by the Project Manager, will
have at least one EVM expert (more required for a system review), and the team
members will consist of various project office staff. It is possible to conduct
the review jointly with your staff as part of the review team. Review training
should be tailored to your needs and held immediately prior to the IBR or EVMS
Review wherever possible.
What is a System Assurance Review and why is it necessary?
The System Assurance Review is designed to confirm that your EVM System continues
to meet the contractual requirements. The need for a review will be based on
risk and further information on the decision making process can be found in
the EVMS Review Handbook. Provision is made in the contract for this review
to be conducted as required.
Payment By Earned Value
TOP
Is there concern over the amount of Level of Effort (LOE) being used as the Earned Value Technique for our contract which includes payment by earned value?It is a concern, but may still be appropriate. Where the value of LOE work packages represents more than 5% of the contract price (less any profit or fee) that is not subcontracted, the measurement of project performance may be affected. The project office will, and you should, scrutinise the selection of LOE as the Earned Value Technique (EVT) for those work packages to ensure it is being appropriately applied. The selection of EVTs such as LOE or 100-0 based on your payment objectives compromises the integrity of the EVMS and, thus, the validity of the contract payment approach.
What documentation needs to be provided for the review of an Earned
Value Claim for Payment?
Clause 7.3.6 of the ASDEFCON (Strategic Materiel) SOW outlines the documentation
and personnel that you must, if requested, supply in support of an EV claim
for payment. These include:
- the list of all Work Packages that comprise the reported progress and the associated Cost Performance Report and Schedule; (these are ESSENTIAL documents for claim review)
- relevant Earned Value Status Sheets;
- any relevant contractor and subcontractor invoices, receipts, progress/conformance documentation and relevant Progress and Supplies Acceptance Certificates; and
- managers associated with earned value progress.
Last updated: 16 June 2008
