Thursday, September 30, 2010

'Smart' electricity meter based on Efergy Elite monitor, part 1

Update (16 April 2011): I've got a follow-up post (part 2) which I describe a more efficient signal decoding technique using PIC interrupts and timers and allows for simple multitasking.

This series of posts describes how I hacked a low cost (€37 from Maplin) Efergy Elite electricity monitor to obtain real time power use data and to record and chart this data with the aim of understanding how electricty is used in the home and how to reduce consumption.

Describing the entire solution in one go is going to result in an excessively long post, so I'm going to break this into several parts over the next few weeks. In this post I'll describe the process involved in getting access to the raw data from the sensor unit.

The term 'smart' in the title is a misnomer. All I want to do here is record real time (at least several samples per minute) power use into a computer system. A true 'smart' meter will do telemetering, facilitate demand based tariffs, feed-in tariffs etc.

The problem with low cost energy monitors is you only get the current instantaneous usage and perhaps hourly, daily and weekly averages.

What I want is something that looks like this:


In this chart I can see my coffee machine (left on accidentally), the oven (not much I can do about that) and the fridge. From that chart I can figure the base load due to broadband router, WiFi access point, DVR etc and work out the average use of my fridge (by estimating the duty cycle and subtracting the power level while on from the base load).

So, how did I arrive at this chart?

I had an Efergy Elite electricity monitor which I received as a gift a few years ago. The product comprises a current transformer sensor which clips on to the utility meter outside. This transmits a power reading every 6 seconds over a 433MHz radio to the receiver unit indoors. The receiver has a LCD which displays instantaneous power use, and hourly, daily weekly averages (and other less interesting stuff like CO2 emissions). Photos of the sensor, receiver and internal electronics are here. There is no computer interface to tap into this data, so one needed to be hacked in.

There is a block diagram of my solution:


Looking at the receiver unit electronics, there is a test pad located near the radio part of the PCB. (The A72C01AUF chip to the bottom left of the test pad does all the radio heavy lifting both in the sensor and  receiver).


Looking at the signal at this point with an oscilloscope one can see there is burst of logic level (0/5V) signals every 6 seconds. This would seem to be the base band output from the radio.  An oscilloscope isn't the most useful tool for decoding digital IO if there are more than a few bits involved. I put a recently purchased Saleae 'Logic' (logic analyzer) on the task and this is what the signal looked like:



This signal didn't seem to make much sense. So I also simultaneously recorded the sensor's digital input line to the transmitter (pin 18 on the sensor's MCU – which by the way is a GSSP 22682X01 – a difficult chip to locate a datasheet for).


Thanks to the logic analyzer's big picture it seems there is an initial block of what looks like random noise preceding the actual data (I'm guessing this is just noise from the radio transmitter powering up).

Now looking more closely at the actual data:

The data is encoded using a digital equivalent of Frequency Shift Keying (FSK). A logic 0 is represented by three long pulses of 2ms duration in total (a low frequency) and a logic 1 by 4 shorter pulses totaling 2ms also (a high frequency).

Decoding this could be achieved with analog circuitry but this would involve a handful of (relatively) expensive passive components. Instead I chose a pure digital technique. I used a simple pattern matching algorithm to look for 3 successive long pulses or 4 successive short pulses and ignore anything else (code appended below).

From bits to bytes

So now I have a string of logic '0's and '1's. The next problem is to determine where the byte boundaries are.

Fortunately each data packet starts with the following 4 bytes: 0xAB 0xAB 0xAB 0x2D. This helps with synchronization. When looking for the start of data I shove the bits into a shift register until I get a byte value of 0xAB. I take all subsequent bits as groups of 8 to form the bytes of data. After the first 0xAB I ignore all subsequent byte values of 0xAB until I get a 0x2D. I am now at the start of the packet payload.

A typical packets look like this:
AB, AB, AB, 2D, 00, 0B, 5A, 40, 98, 00, 02, 00, 41
After some experimentation this is what I believe the packet consists of:

0Sync, always 0xAB
1Sync, always 0xAB
2Sync, always 0xAB
3Sync end,  always 0x2D
4? always 0x00
5Device address high byte (always 0x0D for my device)
6Device address low byte  (always 0x5A for my device)
7sbssaaaa where s = bits of sampling period, b = battery low indicator (0 = low) and aaaa is the upper 4 bits of sensor A. 0-00---- = 6s sampling, 1-01---- = 12s sampling, 0-10---- = 18s sampling. It's odd that ‘b’ is sandwiched in among s bits.
8Least significant 8 bits of sensor input A (nearest MCU edge of board)
9? probably upper 4 bits of sensor B and C.
10Least significant 8 bits of sensor input B (middle)
11Least significant 8 bits of sensor input C
12Checksum (arithmetical sum of bytes 4 - 11).


So in summary, each packet comprises a unique sensor address, the sampling rate, a battery low flag and 3 x 12 bit ADC values for each of the sensor's three inputs  (normal domestic power is single phase, so only one input is used. A three phase system would use all three inputs) and finally a checksum of the contents of the packet for data integrity.

I soldered a small (12F675) PIC MCU to a piece of veroboard (pictured right) and wrote a program to decode the baseband signal from the radio and convert that to actual data.

The PIC code used to decode the radio baseband signal is listed below (this excerpt is not the full PIC application – contact me if you need the lot):

The next problem is to figure how to translate these ADC values to Amps (and by multiplying by 230V to Watts) and how to feed this data to a database for recording and visualization. These will be covered in subsequent posts.

Related Links
Updates to post:


#include 
#include "efergy_config.h"
#include "efergy_elite_decode.h"


byte getPulseWidth();
byte getBit ();
byte getByte ();
void resync();

byte t3,t4,tm;


/**
 * Decode baseband radio signal from Efergy Elite receiver.
 * 
 * Packet consists of a radio preamble of what seems
 * like random noise for about 200ms followed by a quiet period
 * of 100ms followed by data for 200ms. Encoding is a FSK type
 * scheme.
 * Logic 0 is represented by 3 cycles of "low" frequency square 
 * wave (period 8t per cycle). 
 * Logic 1 is represented by 4 cycles of "high" frequncy square
 * wave (period 6t per cycle).
 * Logic 0 and Logic 1 have the same period of 24t which is
 * very approx 2ms [confirm]. Ie t ~= 83 microseconds.
 *
 *    +---+   +---+   +---+
 *    |   |   |   |   |        = Logic 0
 * +--|   +---+   +---+
 *   +--+  +--+  +--+  +--+
 *   |  |  |  |  |  |  |       = Logic 1
 * +-+  +--+  +--+  +--+
 *
 * Decoding this not but looking at frequency, but looking
 * for 3 long pulses in succession or 4 short pulses. 
 * 
 * Data has 3 bytes of synchronization: 0xAB, 0xAB, 0x2D.
 * Look initially for 0xAB to get byte boundary synchronization. 
 * Then look for 0x2D to flag start of actual data.
 *
 * All inter-pulse calculation must be complete 
 * within 64x5 microseconds ie ~ 300 microseconds or 300 instructions @ 4MHz)
 */

byte decodeEfergy(unsigned char *buf) {
 byte i,bitc=0, bytec=0, b=0;
 byte t,pt;
 //byte bufIndex=0;
 unsigned int rt=0;

 t3=0x1a; // short pulse duration of 3t
 t4=0x23; // long pulse duration of 4t
 tm = 0x1e; // mean of short and long pulse, ie 3.5t

resync:

 // Look for sync bit sequence
 // TODO: update: just look for AB followed by 2D.

 // Get byte boundary sync
 while (b != 0xAB) {
  b = (b<<1) | getBit();
 }
 for (i = 0; i < 8; i++) {
  b = (b<<1) | getBit();
 }
 if (b != 0xAB) {
  goto resync;
 }

 for (i = 0; i < 8; i++) {
  b = (b<<1) | getBit();
 }
 if (b != 0x2D) {
  goto resync;
 }


 b=0;

 // 9 bytes left to retrieve

 do {
  b = (b<<1) | getBit();

  bitc++;
  if (bitc == 8) {
 
   buf[bytec++] = b;
   b=bitc=0;
  }
 } while (bytec < 9);

 // Calculate checksum
 byte cs = 0;
 for (i = 0; i < 8; i++) {
  cs += buf[i];
 }

 // Display checksum fail if they don't match
 if (cs != buf[8]) {
  return 4;
 }

 return 0;
}

byte getByte () {
 byte i,b;
 for (i = 0; i < 8; i++) {
  b = (b<<1) | getBit();
 }
 return b;
}

byte getBit () {
 char t;
 byte nshort=0,nlong=0;

 while (1) {
 
  t = getPulseWidth();

  if ( t < tm) {
   nshort++;
  } else {
   nlong++;
  }

  // Check for anomalous condition
  if ( (nshort>0) && (nlong>0) ) {
   nshort=nlong=0;   
   if ( t < tm) {
    nshort=1;
   } else {
    nlong=1;
   }
  }

  if (nshort == 4) {
   return 1;
  }
  if (nlong == 3) {
   return 0;
  }
 }
}

byte getPulseWidth() {
 byte t=0;
 while (RADIOBB_PIN) {
  CLRWDT();
 }
 while ( ! RADIOBB_PIN) {
  t++;
 }
 return t;
}

void resync () {

 byte pt,t;

 restart_resync:

 // Can have either 3t (short) or 4t (long) pulse. Get pulses of
 // two different lengths. 3t is the short pulse, 4t is the long one.
 pt = getPulseWidth();
 do {
  t = getPulseWidth()+2;
 } while (t - pt < 4);
 t -= 2;


 if (pt < t) {
  t3 = pt;
  t4 = t;
 } else {
  t3 = t;
  t4 = pt;
 }

 if (t4 < 0x1A) {
  //TXREG='s';
  goto restart_resync;
 }
 if (t4 > 0x30) {
  //TXREG='l';
  goto restart_resync;
 }

 // Didn't have to do this at the sensor, but receiver has a lot more incoming crud
 if ( ((t4>>2)*3 ^ t3) & 0b11111100 ) {   // (t4/4)*3  ~= t3 (ignoring 2 LSB of precision)
  //TXREG='?';
  goto restart_resync;
 }
 

 // tm is mean of t3 and t4
 tm = (t3+t4)>>1;

 byte b=0;

 // Look for sync bit sequence
 while (b != 0xAB) {
  b = (b<<1) | getBit();
 }
 
}

27 comments:

rwijbenga said...

Hi,

Any updates on this GREAT project ?

I would like to build your interface, but I just have basic electronics skills so writing the rest of the PIC program is a bit to steep for me... :)

And of course the database that needs to run on the PC...

Keep up the good work!

Cheers !

Remco

jdesbonnet said...

Yes, more updates on the way. Getting distracted by day-job stuff. Hopefully in a few weeks. Joe.

Alex said...

Hello,

I've been using the E2 meter, not the Elite as you are.

I've already unsuccessfully tried to hack the protocol with no result.

Instead of tapping on the soldering point like you did (thanks for the tip!), I've connected a small lead to the serial chip, on the AMICCOM chip
(here's the datasheet
http://www.amiccom.com.tw/upload/2010102115450423933.pdf ), testing my SMD soldering limits..

Unfortunately they've changed the protocol since the (old) Elite and the (newer) E2.

It's not 3pairs or 4pairs of short/long, it's short OR long for 1/0 now.

I have basically the same setup as you have, a Logic probe from Saleae, so that pretty much eliminates the measurement error.

The e-link software (.net) is obfuscated, so no luck there either.

The last way of acquiring the data that crossed my mind was OCR on a webcam facing the monitor...

Luckily I found your website and you gave me a new hope on cracking it.

For the time being I've gathered some raw data (various screen values vs received signals). I've written them all down and yes, there is a pattern (the same noise at the beginning, the same 1st part of DATA that stays the same in between messages, etc), but the rest of it has changed.

Will keep you posted if I manage to do it.

Greetings from France

jdesbonnet said...

Alex,

Thanks for the feedback. Actually I have an Efergy E2 also, but I didn't try to hack that one (yet :). I would be very interested to know how you get on. I did notice that the E2 packets seem shorter in duration (and the battery life of the sensor unit is much better than the Elite). By the way: I did make some head way in decoding the USB protocol between the E2 and the PC. Unfortunately it is only hourly aggregates there. But if you think it is useful I can post it on my blog.

Joe.

Kjell_norw said...

Hi

Look very similar to Clas Ohlson item 36-4000?

Regards from Norway

Alex said...

Hi Joe,

no, absolutely no luck with the efergy e2 so far.

What I've been able to assume is that:
- the protocol relies on short/long(er) burst for 1/0 (or vice-versa).
- garbage is sent before a 7-times repetition of short burst begins. This is the start signal of a meaningful packet and the real information I believe it's the next following bytes.
- the message ends without any meaningful bit. It seems like the emitter is cut off and generates some noise before being put to sleep.

What we could do is share some Logic probe files and see if we can find a pattern.

What really scares me off is that even if we crack it, my microcontroller programming skills (assembler mostly for robotics - pwm, uart, servo, a/d, timers, etc) are far too basic in my opinion to get the (pic) interpreter right.

My suggestion: I'm volunteering for a C# interface of the project (web, winforms, mobile if you like) and would gladly send any logic probe files / readings in exchange for a little bit of help on the protocol interpreter.

If there are some volunteers on the hardware side, I think we can get in touch via e-mail.

Joe, if interested, drop me a line.

Cheers

jdesbonnet said...

Alex, if time permits, I would be happy to take a look at the logic capture files. Email address is user
jdesbonnet at gmail dot com.

Jose said...

Hi Joe,
Great work!
I've been trying to do the same hack and I've found some differences in the data format... How did you realize about the three sensors?
Please look at http://electrohome.pbworks.com/w/page/34379858/Efergy-Elite-Wireless-Meter-Hack

Thanks
Jose

jdesbonnet said...

Jose,

Looks like you've got some good stuff there. I didn't get around to figuring the ADC -> Amps function yet. As you point out in your article it doesn't seem to be linear.

BTW: I do have some info on the MCU: The chip is marked with manufacturer "Gssp" which I found very difficult to find any reference to. After a lot of googling I came to the conclusion it's a PIC clone based on a "GMC14 core". I found a datasheet/manual for this. I can forward that to you if you wish.

Joe.

jdesbonnet said...

To answer the question about the sensors: it was a while ago and I can't remember to be honest.

Since my meter is inconveniently located outside the house in a cabinet I used my desktop lamp for testing. It's like this one here: http://goo.gl/YIjWw

The bulb is powered by AC 12V but voltage doesn't matter. I clipped the sensor around one of the support/power beams.

That way I was able to try the three different sensor sockets with current on/off without leaving my seat.

To get an ADC vs current graph I was thinking of rigging something using a car battery and some high power resistors to go from 0 to 13A with low voltages. I think it needs to be AC which complicates things. Maybe a square wave generated using a FET might be a good approximation.

Jose said...

Joe,
Thanks a lot
I'll aprecciate if you can send me the ucontroller data sheet. That's for the transmitter, right?
My email is jolcese [at] gmail

Thanks!
Jose

jdesbonnet said...

Yes, that's the SOIC marked with manufacturer "Gssp" on the transmitter. I have a hunch the receiver/display unit has the same or similar IC under the blob on the PCB.

Can you sent me an email and I'll reply with an attachment: jdesbonnet at gmail dot com.

jdesbonnet said...

I believe (but have not verified) that this is the datasheet of the MCU used in the transmitter unit of the Efergy Elite: http://www.abovsemi.cn/data/manual/MC71PB204_UsersManual_ver1.0.pdf

Alex said...

Hello,

May I suggest they are using a lookup table? I would clearly suggest that because finding the function that follows the shape of some plotted values is nearly impossible (it's zig-zagged based on the graph).

I've collected some values, all @ 230 volt. Feel free to join them to your list

11251125,1135,1609,1927,2543,2583,2624,3138,3314,3464,3492,3564,3622,3651,3665,3866,3953,3967,3981,4039,4053,4110,4340,4369,4455,4470,4484,4498,4527,4556,4585,4599,4613,4829,4845,4890,4906,4936,4982,5013,5044,5258,5274,5350,5412,5442,5665,5750,5770,5791,5854,5895,5916,5937,6272,6335,6376,6460,6481,6627,6690,6773,6794,6899,6956,6995,7033,7052,7263,7282,7302,7378,7455,7685,7819,7934,8164,8260,9530

jdesbonnet said...

I think a lookup table is unlikely. It provides no technical advantage while requiring considerably more memory space in the MCU.

When I did some work on the E2 over a year ago I did spend time looking at the USB protocol used to dump data to the supplied Windows software. The current/power values were 24 bit floating point. I think it was the least significant 8 bits were the exponent. It is possible that is being transmitted in the wireless protocol too.

jdesbonnet said...

Just checked my notes re the Efergy E2 USB protocol. The power/current values are 24 bit floating point in this format:

byte 0: exponent
byte 1: low byte of significand
byte 2: high byte of significand (of which the most significant bit is the sign bit)

Jose said...

@alex
Do you have the corresponfing power/current for those measurements?

Has anyone measured the output of the current sensor? I'm thinking on creating a variable current/voltage source to feed into the transmitter directly and mimic the sensor for different currents...

joakimahlen said...

Hello there - just recently catching up on this thread since i own an Efergy Elite and want to contribute in trying to hack it.

What would you think of using a Tellstick device to emulate a sensor? That way you could send lots of test data and see how the device converts the input values to what is printed on the screen. Maybe that way we could reverse engineer the ADC-to-current algorithm.

jdesbonnet said...

I'll take a look at my data again and see if there is an obvious ADC -> current function.

It would be nice to be able to generate a controlled current that will ramp from 0A to 30A (or whatever the maximum is) in increments of 0.1A. However I cannot think of any easy way do this at utility voltages.

What might be possible is this: The sensor is a current transformer... so the voltage is not important. With a high current source, eg a car battery and some *big* FETs and a smoothing capacitor use PWM to create a 50 or 60Hz current signal up to whatever the battery can deliver or the FETs can handle.

joakimahlen said...

Check out what a tellstick does:

http://www.telldus.se/products/tellstick

It might be possible to write your own software to the tellstick to mimic an efergy transmitter.

What do you think?

jean said...
This comment has been removed by the author.
jean said...

@jdesbonnet

I have the E2 connected the pins as per your diagram, using an arduino but having no luck reading data from E2 with digitalRead(E2Pin,INPUT).

I have GND to GND and dataout(pin18) to D8 on arduino. Can send you code but dont want to spam here.

Help

Jean

jdesbonnet said...

Jean,

The Efergy E2 uses a different protocol to the Efergy Elite.

The good news is that there is a separate project making great progress for the E2 protocol. I believe they are using a PIC, but it may be possible to port to an Arduino.

I'm not sure where the official home page is (I'll find out for you), but this is one of the links to the source:
http://electrohome.pbworks.com/w/browse/#view=ViewFolder&param=READER_FILES

Joe.

SteveG said...

Hi
You seem to know your way around this piece of kit. So maybe you can help me. I was thinking of using the Efergy's alarm function (when used as a PV monitor) to produce a control voltage to switch an appliance on.
The Efergy alarm pulses (LEDs and beeper) but stops after about 10 secs. Is there a way to get a permanent alarm on/off voltage from the rx or would it be best the hack the sensor?

jdesbonnet said...

SteveG, My hunch is if you need a signal high/low when power use is above or below some threshold, the only reliable way of doing that is to intercept the power readings as described in this article and implement that in software (on an MCU eg Arduino, PIC, etc). I've forgotten how this unit actually works, but if the alarm is only on for few a few seconds, perhaps you could use it to trigger a latch. But then how do you reset that latch when power levels return to normal? Maybe just have it timeout after a few minutes. I think a 555 timer will help you there. But I don't think that's what you want.

SteveG said...

Thanks for that. Funny - a 555 timer crossed my mind also!

SteveG said...

I was hoping for an analogue solution as my processor/software knowledge would fit on a postage stamp. Agree the digital solution would be much neater are more controllable (1kw steps are the accessible alarm settings!) but hey ho.
(Btw the alarm sounds for 30sec then sleeps for 5min before sounding again if the power is still above the threshold!)