Recent Publications

Impact, Influence, Leverage and Learning: Evaluation that Makes the Case to Funders

By Tom Kelly, Originally Published on Tamarack CCI on October 16, 2015Access the Impact, Influence, Leverage, and Learning (I2L2) Framework Last year was my 50th birthday with half of those years spent in evaluation (and 15 of those in philanthropy). It did prompt some reflection about life and past work but more or less just added to the To Do list or refocus on old items on the list still undone.Like finishing blog posts and actually posting them.   With apologies and thanks to Chris Lysy, Sheila Robinson, and Ann Emery who have provided much encouragement, advice, tips, and technical assistance on evaluation blogging. The student still has to do the homework.Years ago while working at the Annie E. Casey Foundation we struggled with how to organize, summarize and communicate very diverse grant making strategies and results (from direct service work to community change, from technical assistance and capacity building to policy and advocacy). We had plenty of numbers and examples without a common framework for communicating them. This triggered both a return to basics around an intentional outcomes focus (with Results Based Accountability) as well as common definitions for the “types” of outcomes and results we were aiming for. There was a lot of experience with naming and describing outcomes for child and family well-being (e.g., increased employment, improved school attendance) but the struggle was often with summarizing more developmental outcomes like organizational capacity, changes in attitudes and beliefs, and early policy and advocacy investments. Collective memories may be fuzzy but I credit Miriam Shark at Casey with advancing a set of 3 result categories:Impact – results that were intended to achieve direct change and impact on peopleInfluence – results that described intended change in organizations, beliefs and behaviors, contexts, and policies and practicesLeverage – changes in resources and funding, in this case, especially where the foundation investments influenced others to change how they invest in the same or similar strategiesWe later discussed including a second “L” for Learning outcomes–especially where there was intentional strategies to acquire knowledge needed to inform other work. This list may seem simple (and even obvious now in retrospect) but it provoked some key thinking and behaviors.First, it helped program officers and grantees organize and report results in all 4 categories which was especially helpful for activities and investments that could not measure community impact directly or within a short time period. The influence and leverage results could describe the early evidence that change was happening on a path to impact outcomes. In addition, it allowed for not only results from different strategies within a portfolio or across the foundation to be consolidated, it also helped people to look at all four types of impact, influence, leverage and learning outcomes for individual grants or activities.Second, it provided a prompt to everyone to define their intended results (and intentional strategies) for all four categories at the beginning of the planning and work. Again, this is certainly obvious within most results and outcome based planning but often the focus is only on the long term impact and less upfront attention is given to the earlier and needed influence and behavior change outcomes needed to achieve changes in people and places. What often happens is that impact results are defined up front and measured but if not fully achieved both foundation and grantee fell back on narrative or bullet-form examples of “other changes” that occurred–often influence and leverage–defined and documented in retrospect.Starting with this initial set of ideas I asked ORS-Impact to prepare a tool for Making Connections community change sites and grantees to understand how to define and measure influence and leverage. This initial guide helped get the concepts and early definitions to a limited audience of grantees and it provided examples of indicators and ways to measure both influence and leverage. Later in 2006, we focused on the policy and advocacy aspects of influence which took the form of other manuals and guides, and also contributed to the growing policy advocacy evaluation work.I continued to use the initial framework with the inclusion of learning outcomes in different work with multiple organizations and ORS-Impact also went back to it in work with other clients. Deceptively simple but helpful as an organizing framework when the work or array of investments and strategies have different levels of focus and change and operate in different timeframes but are meant to relate and be complementary. Certainly deeper and more comprehensive theory of change exercises help to define these same elements in different ways but these can be challenging to summarize and communicate to audiences not immersed in the work (like board members or the general public).So we decided to go back to the original ideas and publications and spend time documenting good case examples of how the framework has been used and what organizations have gained from it. Jane Reisman, Anne Gienapp, Sarah Stachowiak, Marshall Brumer, Paula Rowland, and the ORS-Impact team worked with current and past clients and colleagues to assemble these examples. We also shared the examples at the American Evaluation Association conference and other meetings which helped to develop the version you can read here as I2L2.We continue to receive positive feedback especially around the I2L2 framework’s ability to help organize thinking and definitions of expected change and results. Again, this doesn’t replace theory of change and other in-depth planning but when community change strategies and their intended outcomes are complex and highly interrelated (sometimes without distinct sequencing), I2L2 helps groups to organize, define, document and communicate the results they are aiming for and achieve.So where are we now? We have spent a lot of time and effort on defining terms and examples for influence and leverage. (Others have also contributed their work on these categories–see Jim Casey Youth Opportunity Initiative‘s Assessing Leverage guide. We would like to focus on how to help people and organizations focus on defining intentional and planned learning results and the strategies to get there. Here we want to define learning to not be only the lessons acquired from (usually) failing to achieve impact or successfully reach targets but more importantly the intentional agenda for acquiring needed knowledge. Defined at the beginning and evaluated along the way. We hope to be discussing this more with others including at this year’s AEA meeting.Do you have examples of work defining learning results? Learning outcomes? How have you evaluated learning?We’d love to hear from you.

Moving On: The Effective Management of Partnerships, Transitions, Transformations, and Exits

A Toolkit
This tool book by the International Business Leaders Forum is an excellent guide for multi-sectoral relationships. Recognizing that partnerships of diversity create differences in timeframes, language, culture, and goals, they have developed a tool kit meant to guide partners all the way through the life cycle of the relationship: from the consideration phase - through the initial conversation - to scoping out the work- sustaining the outcomes - dealing with transitions - and strategizing the end of the partnership. In this guide you will find: Checklists Tools for assessment Handy tips Case studies Templates and more!  Access Moving On: The Effective Management of Partnerships, Transitions, Transformations, and Exits   

Ontario's Social Assistance Poverty Gap

By Kaylie Tiessen
Kaylie Tiessen and the Canadian Centre for Policy Alternatives have published a report on Ontario's Social Assistance Poverty Gap Introduction Just over seven years after the Government of Ontario launched a major poverty reduction strategy, it has broadened the scope to include not just families with children but, also, adults and people experiencing homelessness. This paper drills down on one key but complex policy file that is essential to the province meeting its commitments to reduce poverty and to improve income security for both children and adults: social assistance. It measures the poverty gap for singles and families who qualify for either Ontario Works or the Ontario Disability Support Program. It concludes that the poverty gap—the distance between total benefit income and the poverty line—for people who qualify for social assistance has worsened over time, especially so for single people receiving Ontario. Read the report

Temporary Foreign Worker Advisory - Brochure

From the Government of Alberta, Temporary Foreign Worker Advisory Office
Find the brochure released by the Governemnt of Alberta's Temporary Foreign Worker Advisory Office, as a resource employees and employers to understand the rights of and responsibilities Temporary Foreign Workers, and where to look for support or advice.The roles and responsibilities of the TFWA Office, include:Help clients understand their rights and responsibilities. Help resolve unfair, unsafe or unhealthy working onditions by working with federal, provincial and municipal governments, as well as not-for-profit and community based organizations. Respond to complaints, inquiries and requests for information. Work with not-for-profit organizations to help TFWs adapt to living and/or working in Alberta. Provide assistance in accessing, completing and submitting TFW-related formsAccess the brochure here

The Direct Burden of Socio-Economic Health Inequalities in Canada

An Analysis of Health Care Costs by Income Level
People who enjoy higher social and economic positions relative to others based on their income, education or occupation tend to be healthier. As such, they generally need and use fewer health care services, resulting in lower health care costs. This report examines health care cost differences between socio-economic status groups in order to estimate what these differences cost the Canadian health care system. This estimate is called the direct economic burden of socio-economic inequalities in health. This report offers the first national-level estimate of the contribution of health inequalities to health care costs. Highlighting the costs of poor health informs Canadians about potential economic gains from improving health and reducing health inequalities by addressing the social, economic and environmental conditions that strongly influence health. Key Findings Health care costs generally decline as income rises for both men and women. Health care costs could be potentially reduced by $6.2 billion if all Canadians had the same health care utilization and cost patterns as those in the highest income group. Over 14% of total annual expenditures are currently spent on on acute care in-patient hospitalizations, prescription medications and physician consultations. The lowest income group accounts for 60% ($3.7 billion) of the health care costs of socio-economic health inequalities. Read the full report here: The Direct Burden of Socio-Economic Health Inequalities in Canada Discover the technical report here: Health Costs by Income Level 

Better is Always Possible: A Federal Plan to Tackle Poverty and Inequality

Alternative Federal Budget | February 2016
This technical paper originally appeared on the Canadian Centre for Policy Alternatives website by Seth Klein and Armine Yalnizyan. The federal government has recently mandated Minister Yves-Duclos to work on a federal poverty reduction strategy. What it looks like has yet to be seen.In February 2016 the Canadian Centre for Policy Alternatives has released their Alternative Federal Budget, including recommendations based on research throughout the year. What needs to happen?Reduce the overall poverty rateEqualize outcomes for vulnerable populations and disadvantaged peopleReduce income insecurityIncrease emergency housing in the short-term and housing stock in the long-termHalve food insecurityHow can we turn these recommendations into reality?Identify measurable targets and timelinesTake a human rights approach and include the voices who have lived experience of povertyForm a strong relationship with the provinces and territories that transfers funds to those showing achivements of their poverty reduction goalsSet national standards for welfare ratesEstablish a Canada Child Cenefit for low income familiesIncrease the GIS to keep seniors out of povertyLegislate decent wages Tackle homelessness and the housing stockImplement universal child careInvest in green social enterpriseRead the full report here

Handbook on Citizen Engagement: Beyond Consultation

By: Amanda Sheedy, Canadian Policy Research Networks Inc.
This handbook builds on years of work at the Canadian Policy Research Networks bringing together cutting edge thinkers and practitioners in the field of citizen engagement. While it is not possible to capture all of CPRN's and others' work in one handbook, the hope is that this tool will provide a good overview of the breadth of the field – both the concepts and the methods – and supply ample resources (particularly online resources) with which to deepen knowledge on specific subjects.The handbook is intended to whet the appetite for citizen engagement for those new to citizen engagement, and for those with experience to deepen the analysis behind citizen engagement projects and provide a synthesis of the field and a concise reference tool. The long term vision is to contribute to the closing of the gap between governments and citizens, to allow public servants and politicians to reconnect with citizens’ needs, priorities and values.This handbook is not a prescriptive how-to manual on citizen engagement. There is no one-sizefits-all in citizen engagement. Each context, policy or program development process requires a unique approach and adapted tools to address its specific needs. Engaging citizens in a meaningful way first requires an understanding of the philosophy and vision of citizen engagement. It calls for planning and preparation and sometimes institutional capacity building. It can demand a shift in organizational or departmental cultural conceptions of what citizens can bring to a policy process. This handbook is a starting point to think about these issues and a reference guide for those who wish to deepen their understanding and practice of citizen engagement.Citizen engagement is premised on the belief that people should have and want to have a say in the decisions that affect their lives. While some may claim that voting and consultation processes achieve this, it is clear that citizens are increasingly frustrated with these democratic mechanisms. They feel that their voices are not being heard and that decisions made by elites do not necessarily reflect their values. Citizen engagement provides a vision for a way forward – a way of reinvigorating current democratic practices and institutions, bringing meaning to people’s participation and fostering a two way dialogue between citizens and governments. The hope is that this will not be seen as a mechanism for placating peoples' desires, but actually bring about a more just society where governments' choices more closely reflect the needs of its population.Read the Handbook here

Provincial/Territorial Monitor

November 2015
Manitoba was the most prolific jurisdiction in terms of releases in November, including the publication of its 2014-15 annual report on ALL Aboard, the province’s poverty reduction and social inclusion strategy.

Federal Policy Monitor

November 2015
November brought many changes, including the publication of the Prime Minister's mandate letters.  His letter to the Minister of Families, Children and Social Development pledged the development of a Canadian Poverty Reduction Strategy that will set targets to reduce poverty and publicly report progress.  The federal strategy will align with and support existing provincial and municipal poverty reduction strategies.

Welfare in Canada, 2014

By Anne Tweddle, Ken Battle, and Sherri Torjman
This report focuses on the incomes of four different households living on social assistance, commonly known as “welfare.” It is a continuation of the welfare incomes series published regularly by the former National Council of Welfare. Total welfare incomes consist of the sum of two main components: social assistance provincial/territorial and federal child benefits as well as relevant provincial/territorial and federal tax credits. Social assistance is the income program of last resort.  It is intended for persons who have exhausted all other means of financial support. Every province and territory has its own social assistance program, so no two are exactly the same. Each program has different administrative rules, eligibility criteria, benefit levels and provisions regarding special assistance. However, the basic structure of social assistance is much the same across the country, even though the specifics may vary. The most common way of assessing the adequacy of any income program is to compare it to a recognized standard and then determine how far it diverts from that indicator. There is no single or commonly accepted baseline, but rather several measures that typically are used for comparative purposes. They fall into one of two groups: poverty measures and income measures. Poverty measures are considered to be the baseline level below which households are deemed to live in poverty. Two poverty measures are employed in this report: low income cut-offs (LICOs) and the Market Basket Measure (MBM). In 2014, welfare incomes for single employable households ranged from 38.2 percent of the after-tax poverty line in Manitoba to a ‘high’ of 64.7 percent in Newfoundland and Labrador. Most of the other jurisdictions cluster around the lower rate. Welfare incomes for single persons with disabilities, while low, were slightly higher, ranging from 49.6 percent of the poverty line in Alberta to 69.9 percent in Ontario. Alberta provides a separate program (AISH, or Assured Income for the Severely Handicapped) for persons with disabilities, which pays higher rates than the standard welfare program. In 2014, incomes of single persons on AISH came to 96.5 percent of the after-tax LICO, far higher than the 49.6 percent for persons with disabilities on standard welfare. The Saskatchewan Assured Income for Disability (SAID) program also pays higher rates than the standard welfare program. For 2014, the income of single persons on SAID was 86.3 percent of the after-tax LICO, compared to 66.8 percent for those receiving Saskatchewan Assistance Plan benefits. For single-parent households with one child age 2, welfare incomes represented 63.1 percent of the poverty line in Manitoba and a surprising 102.4 percent of the after-tax LICO in Newfoundland and Labrador. For two-parent families with two children ages 10 and 15, welfare incomes as a percentage of the poverty line ranged from 57.5 percent in British Columbia to 85.6 percent in Prince Edward Island. The report also compares total welfare incomes in 2014 with the Market Basket Measure. As in the case of after-tax poverty lines, welfare incomes fall well below the designated baseline for all household types and in all jurisdictions, with the exception of persons on Alberta’s AISH program. Income measures comprise the second group of comparators. This set of measures assesses the adequacy of welfare relative to the level of income of other households in the population. There are several different indicators that can be used for comparative purposes. Two have been selected for this analysis: after-tax average incomes and median incomes. For 2014, these data are drawn from the new Canadian Income Survey (CIS). Because the CIS uses a different methodology than the Survey of Labour and Income Dynamics (SLID), income data for 2014 cannot be compared to data for earlier years. After-tax average incomes represent the amounts that households actually can use in their daily lives – their so-called ‘disposable income’ after they have paid federal and provincial/territorial income taxes. After-tax amounts represent a good basis for comparison to welfare, which is not subject to income taxation and is therefore effectively a de facto disposable income. Welfare incomes for the four illustrative households typically ranged between 20 and 40 percent of after-tax average incomes. Only two approach 50 percent and both are for single parents. In Newfoundland and Labrador, welfare incomes represent 49.5 percent of average incomes and in New Brunswick they stand at 47.4 percent of average incomes. The figures tell a powerful story about the adequacy of welfare incomes relative to the after-tax average incomes of Canadians. While the conclusions are basically the same when the welfare incomes are compared to after-tax median incomes – differences are typically only a few percentage points – the adequacy picture comes out only slightly better because of the different comparator base.