[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Evaluation Tool

Dear Pat,

With respect to measuring and reporting company performance, you might want
to check out the sustainability reporting guidelines of the Global Reporting

With respect to overall program performance, you might wish to consider
contacting Mr. Charlie Zammit of the Univ. of Southern Queensland in
Australia, who I understand is doing some very interesting work in this
area.  I have attached an email that Charlie sent me and some colleagues
recently to give you a better feel for the work he is doing....

Please do keep me posted on project progress, as we will soon be exploring
program evaluation options for a different project.

Good luck!

Deborah E. Savage, Ph.D.
Senior Scientist
Tellus Institute
11 Arlington St.
Boston, MA  02116-3411
tel: 617-266-5400 xt 281
fax: 617-266-8303
email: dsavage@tellus.org
web: www.tellus.org


-----Original Message-----
From: Charlie Zammit [mailto:zammit@usq.edu.au]
Sent: Wednesday, February 21, 2001 8:37 PM
To: Deborah Savage; Howard Pearce; Kristin Pierre; Darlene Boileau;
Tarcisio Alvarez-Rivero
Cc: 'Richard (Dick) Osborn'
Subject: RE: Evaluation Tool


I wanted to follow up on Dick Osborne's e-mail of a few days ago regarding
my work on policy and program evaluation. Here is a quick overview. Last
year I received a 6 month contract from a federal government R&D agency to
develop a conceptual framework for evaluating the effectiveness of natural
resources policies and programs. Their interest was based on the observation
that institutional structures appear to be a significant constraint to
delivering effective natural resources management (NRM) outcomes, while in
parallel there is a substantial push for government to be more accountable,
so systems have been established to monitor and report on program and agency
performance (mostly through financial audits)

My background for this kind of work is complex in the sense that I am a
academic ecologist but have spent the last 10 years (before returning to
academia in 1999) in our federal government working as a natural resources
policy analyst/adviser in several departments, but mostly in the Department
of The Prime Minister and Cabinet, and in the Foreign Affairs and Trade Dept
(details so my background are at the website below). To that extent I've
been closely involved in much of the momentum around developing ESD policy
and the transition to sustainable resource use. Policy and program
evaluation is something I've struggled with for many years. Australia, as a
federation, has three levels of government: Commonwealth, State and local,
with Constitutional authority for NRM the responsibility of State
governments. But much policy direction (and funding) comes from the
Commonwealth, and quite a bit of actual implementation takes place through
local government. Evaluation through this maze is a challenge.

The framework I developed with 2 colleagues (one a public policy theorist
and the other an evaluation professional)is built around the hierarchy of
policy commitments evident in national strategies, State government
strategies and thereafter specific programs (so called action agendas) for
implementation at regional and/or local and catchment scale. The framework
clearly separates strategic goals and objectives from operational outputs
and outcomes, and it makes explicit the connectedness between outputs and
outcomes at each level of the hierarchy. One consequence is that outputs at
one level in the hierarchy become inputs for the next, and the evaluation
framework needs to 'map' these linkages carefully.

The framework draws extensively on evaluation theory and methodology, in
particular 'program logic', which as far as I can tell has not been much
developed in natural resources management but which is used in evaluating
health, education and transport policies for example. The key lesson from
this theory and methodology is the need to map the hierarchy of outputs etc
against objectives, and then ensure that a robust system of questioning is
initiated early in the evaluation process (rather than rushing immediately
to indicator devlopment work, which may or may not be appropriate). The
framework provides detailed guidance on how to do this.

I'd be happy to discuss any questions or other information you might want
from this work. I expect to be testing the framework locally this year, but
would also be interested in any oportuniites for wider pilot studies of the
framework under different systems of governance/institutional arrangements.


Professor Charlie Zammit
Land Use Study Centre
University of Southern Queensland
Toowoomba QLD 4350

-----Original Message-----
From: Patricia Gallagher [mailto:patg@lanl.gov]
Sent: Monday, July 09, 2001 1:39 PM
To: p2tech@great-lakes.net; nppr@great-lakes.net
Subject: Measures of env/program successes

Dear Folks,

First, please forgive cross postings...

The State of New Mexico is creating a two-pronged performance measurement 
and reporting system.
One part of the system would track the improvements for companies 
participating in the Green Zia Program; this would include traditional env. 
measures, financial measures and human measures (employee satisfaction. 
stakeholder satisfaction,etc).  This part of the system would also track 
the company's performance over time, looking for evidence of continuous 
improvement.  Each organization would be tracked individually and the 
overall performance of companies rolled up into performance measures for 
the overall state program.

The second track is would measure the program's overall 
effectiveness:  customer satisfaction, retention of companies, continuous 
improvement of program, marketing effectiveness, volunteers or examiners' 
satisfaction, quality of feedback reports to companies,etc.  These measures 
are more programmatic to help assess the success of the program and 
identify areas to focus improvement efforts.

We want to benchmark other state programs to see how they are measuring 
these types of activities.  Can you please share information on measurement 
systems of these types and experiences (both good and bad) that you have 
had in doing this type of measurement?

Please send your responses to me at patg@lanl.gov, mjmitch@sandia.gov and 

Thanks in advance!

Pat Gallagher
Los Alamos National Laboratory