Demand-Side Management Strategy

Oracle Energy and Water: Testing the Efficacy of Home Energy Reports

Oracle Energy and Water logo

Share article

Key takeaways

Equity and access icon

Home energy reports continue to save energy effectively, with households that receive them saving an average of 88 kWh annually for electric customers and 4.4 Therms for gas customers. The longer customers receive these reports, the greater their energy savings tend to be.

Icon image of site selection

There is a lack of consistency in reporting conventions across home energy report evaluations. More consistency would enable deeper analyses across more factors and could substantially improve our understanding of home energy reports and their role in residential energy efficiency.

Icon showing on time

The percentage of savings achieved varies significantly across home energy report vendors, and as cohort maturity increases (the duration customers have received the reports), the average annual electric savings also tends to increase, demonstrating that longer engagement is associated with higher savings.

The challenge

Home energy reports (HERs) are a key part of utility behavioral energy efficiency programs, but many program administrators and industry experts are questioning their effectiveness in capturing customer attention and motivating energy savings. Additionally, the efficacy of HERs may vary based on factors such as fuel type, vendor, cohort maturity, and distribution mode. Oracle Energy and Water was interested in finding out and engaged E Source to take a systematic look at recent HER evaluations.

The solution

E Source conducted a meta-analysis, which systematically and statistically combines results from multiple studies to generate a weighted average for a particular metric. This provides an estimate of the metric across multiple studies, accounting for differences in sample size, study approach, treatment effect size, and sensitivity to inclusion criteria. This is a useful approach for assessing multiple HER evaluations because it provides a rigorous, statistically precise methodology for determining the energy savings generated by HERs across evaluations and how various factors impact those savings. 

To complete this meta-analysis, E Source:

  • Identified relevant studies: We gathered HER evaluations from public sources like HER vendors, utility websites, state evaluation sites, and internet searches. Most of these studies focused on Opower HER programs because of differences in market share, program duration, and state rules about publishing evaluation reports.
  • Screened studies: We used consistent criteria to select documents for the study. We excluded those that did not describe the control group, did not specify the number of treatment group participants, evaluated programs before 2018, or did not provide unadjusted savings data at the HER cohort level. These exclusions ensured we focused on recent HERs and had the essential data for our meta-analysis.
  • Screened cohorts: We screened the cohorts within the included evaluations and excluded those that had received HERs for less than ten months or more than five years. This ensured greater consistency of experience with HERs across the cohorts in our study.
  • Collected data: From each evaluation, we gathered data on characteristics of the utility program, cohort maturity, program treatment experience, the number of participants in the treatment and control groups, and the unadjusted savings.
  • Analyzed data: We calculated weights for each cohort’s savings estimate based on the precision of their estimate. Each cohort’s weight is inversely proportional to the precision of their savings estimate. We used these weights to calculate a weighted mean for electric and gas saving metrics.

The results

The analysis demonstrates that HERs continue to effectively influence customer energy-saving behaviors, with measurable impacts on electricity and gas consumption.

  • HERs are still effective at influencing customer behavior to save energy. For electric users, HERs save 88 kWh on average annually per household, which is approximately 1.16% of their total electricity use. For gas users, HERs save on average 4.4 Therms annually per household, roughly 0.87% of their average gas consumption. Savings vary slightly between vendors, but these differences are not statistically significant.
  • As cohort maturity increases, so do electric savings. As customers receive HERs for more years, their average annual electric savings tend to increase. We found a moderate positive correlation between the number of years receiving reports and electric savings. There is also a weaker positive correlation between the number of years receiving reports and gas savings.
  • Percent savings goal achieved varies across HER vendors. We found differences across vendors in the percentage of electric savings goal that the program achieved. Similarly, there is variation in gas savings goal achievement across vendors.
  • Inconsistencies in HER evaluations hinder analysis. There is wide variation in the way evaluations report savings for HERs. These inconsistencies limited the data we could use to a subset of available reports and cohorts.