Human Resources, Management, and Organizational Development Consulting Services

Methodology 

Truly effective performance management and program evaluation efforts center on four general steps depicted in the following figure and described below. 

Basic Evaluation Methodology of Advent Consulting Associates

1.  Planning

Our application of research skills begins with an understanding of the question(s) the client seeks to answer.  We make special efforts to understand the client’s informational needs and the long-range goal of the evaluation findings.  For example, some evaluations might be for purposes of monitoring and updating progress of on-going programs, while other studies are conducted to guide important decisions about funding, structure, and operations.  The ultimate purpose of each study helps to determine the scope of the effort (e.g., thoroughness and level of detail needed) as well as its general design and cost.  When necessary, we are prepared to conduct a thorough search of published research as a foundation for focusing investigation into well-defined issues.

2.  Evaluation design

Our professionals apply research design principles to suit the particular needs, resources, and goals of the evaluation.  We help clients and stakeholders to clarify expectations of performance and outcomes, identify measureable criteria and key indicators of success and effectiveness, and a variety of resource requirements for performance improvement. 

Two aspects of research are especially important in evaluation:  a) the identification of clear and explicit criteria; and b) appropriate methods for measuring those criteria.  It is largely with the criteria selected that we can evaluate performance (of a person, program, or policy), and we need an effective and reliable measuring process for any criterion measures to be valid.

Our experience has shown that many programs and policies are introduced and implemented without a lot of detailed planning for evaluation.  In these situations, criteria might have to be constructed so that they accurately reflect the original purpose and nature of the program or policy, such as by studying initial legislation and agency charters or by interviewing subject matter experts.  We highly recommend formalizing a consensus by the client and other stakeholders about the validity and adequacy of evaluation criteria as early in the process as possible in order to ensure acceptance of the evaluation results.

Measurement methodologies must be appropriate for the types of data that are available and selected for criteria.  On the whole, we prefer objective measures over subjective ones for complex evaluations, although we know that both types of data have important value.  Sometimes, careful analyses of scales of measurement, sampling procedures, and normative references (e.g., comparison groups, historical records) are needed to assist in interpretation of complex processes where numerous outcomes occur over different times.  In most evaluations of complex programs and activities, the measurement of a multiple criteria is required in order to obtain a full picture of a program’s operations.

In special circumstances, we can tailor research approaches and analytical techniques from a variety of research disciplines, including basis statistical process control, quality improvement techniques, and organizational assessment.  We also offer an array of reporting and feedback processes to help clients assimilate and apply the lessons learned from evaluation findings.

3.  Gather and analyze evidence

We employ a variety of techniques for the collection of relevant data, tailored to suit the nature of the investigation.  These include procedures designed for collecting both quantitative data and qualitative data such as the following:

• Interviews with stakeholders and subject matter experts. These usually are semi-structured interviews with pre-determined topics.  We usually provide ample opportunity for open-ended comments to ensure coverage of the full range of relevant issues.
• Focus group facilitation.  These groups provide information that often goes beyond the data collected in individual interviews.  We often use a semi-structured approach to organize and facilitate the discussion of group members.  Groups can range in size from two to a dozen or more, although for most purposes we recommend no more than 8 or 10 so that everyone is free to contribute to discussion.
• Observation of real-life behavior and environment. Our professionals are skilled in behavioral observation of specific activities, recording of behavioral events, and documenting the environment in which target behaviors occur.
• Questionnaires and surveys. We use psychometric techniques to develop custom-designed survey questionnaires for quantitative analysis and assessment.  We compose items using proven methodologies to ensure clarity and unity of measurement.
• Archival measures. Where operational data are available that indicate performance outcomes and results, we analyze these data using statistical techniques that are appropriate for the nature and characteristics of the data.  We can also apply statistical techniques to assess measurement reliability and process quality reflected in the data.
• Analysis.  We apply advanced analytical skills in the analysis of both quantitative (scaled) and qualitative (open-ended) data, and provide guidance in the interpretation of results.  Our techniques help to clarify group perceptions and attitudes, changes in baseline measures, pre-and post-intervention changes, program evaluations, policy impacts, and the stability of operating measures over time.

Advent Consulting Associates applies both quantitative and qualitative methods to analyze data from research, and identify scales of measurement that are appropriate for the specific sources and types of data.  We often perform descriptive statistics of the sample(s) and relevant stratification groups, and statistical tests of research hypotheses.  Our consultants are very familiar with parametric, non-parametric, multivariate, and univariate statistics and their interpretation.

4.  Communication and reporting of results

Our consultants are skilled in the communication and writing of research results, from a description of the research background through the interpretation of findings for decision-making by policy experts, elected officials, and agency managers.  We can assist in the preparation of materials for public outreach and presentations, such as slides and graphs, tailored to the characteristics of each specific audience.  We are familiar with a variety of formal writing styles including APA, economics, and Turabian styles.

Our approach to performance management is similar to our program evaluation research, at least conceptually.  Both processes focus on acquiring and analyzing data, comparing results with expectations, and being fair and impartial.  An important difference is that performance management is usually focused on individual performance and it is much more concrete than is the evaluation of programs, operations, policies, and organizations.  Because of interpersonal difficulties, stress, and uncertainty, many managers find that performance management is an exceedingly difficult responsibility. 

The basic elements of performance management for an individual employee require communication and fairness of treatment.  The basic steps of the process include the following:

• Communicate to each employee clear expectations for job performance, including work tasks, outcomes, quality and quantity, and timeframes.
• Ensure that any normal and necessary resources are provided for successful performance, e.g., training, equipment, time.
• Monitor job performance and provide feedback, direction, and assistance on an ongoing basis.  In most cases this can be performed informally.  If performance problems have been indicated or are anticipated, however, monitoring and feedback activities should include at least some form of documentation.
• Perform an evaluation of job performance when sufficient time has elapsed for reasonable proficiency to be attained.  If job performance is below established standards, corrective action should be initiated and documented after first informing higher management of the situation and consulting with designated human resources professionals.  In general, corrective action should be taken that is consistent with precedents and policies of the organization; any inconsistencies in corrective action should be carefully documented.
• Monitor progress toward improvement in job performance.  If acceptable performance is attained then corrective action can cease once it is documented.  If performance continues to be unacceptable, the process of corrective action needs to progress until the situation is finally resolved.

At the individual level, remaining objective is more difficult for a supervisor or manager than when focusing on the effectiveness of a program.  It is both more personal and more stressful to deliver job-relevant feedback that is perceived as threatening to a fellow employee.  In addition to the potential threats to an employee’s employment (e.g., security, income, benefits coverage), risks for supervisors also can arise from the process.  On occasion, the application of performance management steps with individual employees has resulted in legal and financial liability for the managers involved. 

We have found that the complex emotional issues entailed in performance management require careful planning and considerable preparation on the part of managers in order to carry out those responsibilities effectively. 

Our Approach 

With technical and consultancy experts who have worked as business and public service leaders themselves, Advent Consulting understands first-hand the challenges our clients face:

• Doing more with fewer resources
• Heightened service expectations from the public
• The need to raise business processes to Internet speed. 

As a partner in performance management and program evaluation, Advent does not bring cookie-cutter solutions. Each organization is unique and deserves a customized, client-centered strategy for assessment, analysis, planning, and action. 

We offer extensive experience in the application of research methodologies and statistical analysis techniques for performance management and program evaluation studies. Basic features of our approach are the close engagement of our clients, or stakeholders, coupled with flexibility in our methods to ensure our assessments meet the unique needs and situation of the organizations we serve.

Our methodologies emphasize the following as key characteristics for performance management and evaluation efforts:

• Objectivity: We seek to collect both quantitative and qualitative data that can provide balanced and accurate information about the current performance the client. Our analyses derive reasonable conclusions that are based on clearly demonstrated facts.

• Thoroughness: Our data collection must include all relevant sources of information and points of view. When a variety of opinions might exist about performance outcomes or effectiveness, we will need to cover the full range of opinions and explore them with appropriate care.

• Documentation: We formalize all of our findings in the form of written notes and summaries for reference at a later time. This information allows for the replication of data and findings for management decisions, future analytical studies, and organizational learning.

We offer a track record of success in these types of initiatives.