Blog entries tagged with "power"

RAVEn to MQTT over wifi

Saturday, July 10th, 2021 at 1:29 pm

I have written before about how I use a Zigbee dongle (the now discontinued RAVEn from Rainforest Automation) to get power consumption information from my smart meter. A bit over two years ago I also wrote about how it appeared that a failing power supply was interfering with the Zigbee connection and also confusing my UPS.

I think that my logic still holds for the UPS issues, but now I think I was wrong about it interfering with Zigbee…

Things improved when I rewrote my script to use a proper serial port library and then improved even more when I started to use that library correctly. However recently I noticed that there were still some gaps appearing in the power usage data, so I had another look.

My first action was to further cleanup my script so it both made more sense and to also report on more messages. I was only caring about InstantaneousDemand which is sent every 8 seconds, but it also often sends CurrentSummationDelivered and ConnectionStatus which I was ignoring.

Once I started logging ConnectionStatus I thought I might be seeing a pattern. At the times (for an hour or so every few days) that I stopped getting InstantaneousDemand messages, all that it was getting were ConnectionStatus messages, and the LinkStrength values were around 60 or 70, not the 100 it normally way.

So was something still interfering?

I couldn’t spot a regular pattern in the dropouts so I had no idea what possible interference there might be. The information about the RAVEn and other smart meter in-home displays mentions that placement of the device is important as Zigbee is low power. I had the RAVEn plugged into the back of my linux box which placed it about 10 metres from the smart meter with walls, furniture and the metal case in between. So what if I simply moved the RAVEn closer?

I had a long USB extension cable so it didn’t take long to relocate the RAVEn to be about 4 metres closer and not directly behind a metal box. After a week in this position I found that there was still the occasional reconnection, but nothing like the hours of no InstantaneousDemand messages that I was seeing before.

I could have just left the USB extension cable in place, but I didn’t like how it looked and didn’t want to add it to the mess of cables under the desk.

The RAVEn is a USB device that provides a USB serial port, on that port are XML messages. My script takes those XML messages and writes them to MQTT as JSON. Is there something else that I could plug the RAVEn into to do this, something that I can place close to the smart meter?

I’m a fan of using ESP8266 devices in the D1 mini form factor for my current home automation. Could I use one of these to build a RAVEn to MQTT bridge over wifi? Maybe, but with some caveats and possibly some effort and delay waiting for parts.

Other Arduino based options seemed just has hard so I turned to the old Raspberry Pi I had sitting around. I had previously used this to test out Kodi so I instead loaded up Raspberry Pi OS Lite and quickly found that I could transfer my existing perl script over, tweak it a little bit, and have it working.

The next hurdle with this as a solution was that it used wired ethernet and I don’t have wired ethernet in the room near the smart meter, I would need wifi if I wanted to have the RAVEn as close as possible.

What would it take to add wifi to this Pi? A USB wifi adapter can be used, but which one and how much would that cost? Would a cheap $8 adapter work, or would I have to get a $50 adapter? As well as not working, it might take a while for a cheap adapter to be delivered.

What about a newer Pi that included wifi? The Pi 3 B+ that I am using for Kodi has wifi even though I am using it on a wired connection, but that also costs towards the $60 mark, and seems to be out of stock.

Hold on, what about a Pi 3 model A+? This is smaller and cheaper than a B+, it also has wifi and the single USB port is fine for my single RAVEn. An online order and a few days later it arrived and was quick to get functional:

Rasperry Pi 3 model A+ in a Core Electronics Slim Case and a Rainforest RAVEn plugged in to the USB port

I needed a case for protection and liked the exposed look of the slim case from Core Electronics.

This has been running like this for about 36 hours now, taped the wall which puts it about 1 metre from the actual smart meter. Time will tell if this solves the reliabilty issues, but it is looking promising.

Of course this does seem like a bit of an overkill to use a whole Pi for this simple task. I might consider a Pi Zero, but I would need the W version (for wifi) and when you include the USB OTG cable the price is getting closer to that of the A+, and these are sold out everywhere I checked…

The final thing I want to mention today is that after I cleaned up my script I also wrote up some documentation and pushed it up to GitHub:

Just don’t yet look at my other repos as all you will see are two that were created seven years ago, only one of which has any code pushed to it. One of the items buried on my todo list is to clean up and push some of the other things I have written for my home automation setup…

No Comments, Tagged with: , ,

The wrong way to use a serial port

Monday, January 20th, 2020 at 10:40 pm

Up until yesterday the last change I made to my script that pulls usage information from my smart meter was to make it use a serial port library. At the time I grabbed the example and apart from adding in a hack to remove the null characters I was getting it worked fine.

Inspired by some things at LCA2020 last week I decided that I would change how I was getting the usage data.

The adapter I am using appears as a serial port and while you can send commands, if you just read from the serial port it will output an InstantaneousDemand XML payload every few seconds. The script I was using would connect to the serial port and read until it got one of these payloads (erroring if it didn’t see one in 30 seconds), writing the data to MQTT for Home Assistant to pick up.

This didn’t seem to be the best way, it sometimes wasn’t able to open or read from the serial port (one hung process could give hours of errors) and it was also skipping over information.

My replacement version would instead run as a daemon, connecting to the serial port once and then publishing MQTT messages whenever an InstantaneousDemand payload appears on the serial port. I still had it die if no payload had been seen for 30 seconds, but it would be restarted as I set it up as a service.

This was working quite nicely for a few hours, but then I noticed the load value was high, checking with top I found that this process was taking 100% of CPU. This wasn’t right…

Upon checking the documentation for Device::SerialPort I found that while I had used examples from the SYNOPSIS section, there was a full EXAMPLE that was much better.

  • In my loop I was trying to read() a single byte each time, instead I should be reading more bytes. The example used 255, so I went with that.
  • I wasn’t checking the count of bytes returned from read(), this explained the null data I had been seeing. I was reading when no data was available, if the count was zero there was no data.
  • There is a timeout value that controls how long the read will wait before returning zero data. If not set then the loop in my code would be calling read() as fast as it could, setting this to one second meant it was handled within the serial port library.

After making these small changes (aka using the library correctly) the CPU usage of this process dropped to nothing. While it wasn’t actually impacting the performance, it is good to do things properly.

Something else I an trying with this process is setting an MQTT Last Will and Testament message. This is a message that you set up when connecting to your MQTT server, but it will only get sent if there is an ungraceful disconnect. Coupled with a startup message I can get a notification when the process is starting and a notification if the process crashes. Time will tell how useful this will be, but it is interesting to learn about.

Tagged with: ,

Late night cable replacement

Saturday, January 18th, 2020 at 11:52 pm

Power and internet comes into my home above my bedroom window. There have been a few times I have been laying in bed watching the cables whip around in the wind, wondering what damage that could be causing and how long the cables should last. This has in part been answered for me as the power cable (likely from when this place was built around fifty years ago) has now been replaced.

Tonight’s events possibly started around 8PM when I started to make some dinner, with one thing on the stove I started another in the microwave, at which point the kitchen lights dimmed (more than ever before) and in my study the UPS started to beep. For a while I had been tracking under voltage in my power because often the UPS would be boosting from battery, this time the voltage had dropped enough that the UPS was still able to boost, but also wanted to start shutting things down. Luckily I was done with the stove and microwave, once they were off the voltage was back to ok levels and the UPS was happy.

After I ate my dinner I started to check my logs and research who to contact about this, then my phone rang. It was my neighbour from the other unit asking if my power was on, because hers was not. This was unexpected as our units are semi-detached, we share the same power cable and the meters are in the same box. But then my power also went out.

Heading outside to talk we saw that the streetlights were on and that other houses in the street still had power. In the meter box I saw that both of our meters were on, so had our power been turned off remotely?

United Energy is my electricity distributor, nothing was listed on their outage page and as I was looking for a faults number to call I received a call. It was someone from United Energy to tell me that my power had been switched off as they had detected a fault. They asked some questions around if I had noticed any issues (I said I had) and told me that a work crew had already been requested and should be there some time that evening.

About an hour later the crew arrived, two people, one in a van and the other in a bucket truck. There were not the friendliest of people (they did have a job to do after all) but from what I could gather they identified the cable as old so would be the likely issue. An hour later they had replaced the cable and run some tests, my power was now back on and they were gone.

I mentioned that I was checking my logs, so what logs do I have?

After I resolved the issue of my UPS shutting down every few weeks I noticed that something was happening to cause the fans in the UPS to come on. The UPS is connected via USB to my linux box but all I was logging was the power output every five minutes. Adding the UPS component into Home Assistant gave me a nice view (and history) of a number of values:

I set up an automation to notify me (via Pushover) whenever the status changed. Over time I was able to see that the UPS was detecting a drop in input voltage, so was using the batteries to boost back up to a normal level. This would typically happen if I used two large applicances at the same time (eg washing machine and stove), but also at other times as well. It was on my todo list to find out what the acceptable voltage range was and probably get it looked at.

So what was the voltage doing tonight? This:

A few things can be seen:

  • Around 230V before 5PM when I wasn’t home
  • Initial drop to around 210V as I put my hot water on boost, then sitting around 220V
  • Big drop to 200V and then as low as 180V around 8PM
  • No data when the power was off
  • Back to 233-239V once I turned everything back on

I will continue to monitor the UPS status with the expectation that I should not (or at least rarely) get notified about undervoltage again.

The final thing I want to say now is that this has been an interesting coincidence. The reason I needed to boost my hot water was because it had been turned off all week while I was at LCA2020 where the overall theme was security and privacy. The specific coincidence I am thinking of is that there was one talk about the smart meters (I hadn’t realised that outside Victoria the rollout was still ongoing and contentious) and I was part of a couple of discussions about their advantages and disadvantages. In this case the remote detection of a fault and quick rectification was a good thing.

We know when you are sleeping: The Rise of Energy Smart Meters – Rachel Bunder (LCA 2020)

Tagged with: ,

An interference coincidence?

Monday, April 8th, 2019 at 7:30 pm

Many years ago I started monitoring the power consumption of my house via a ZigBee dongle that communicates with my smart meter. The quick and dirty solution I did at the time stayed in place for a long time, until it started to play up…

The first symptom I noticed was that my script would hang, in itself not that bad, except that I was calling it directly from MRTG… which would cause the entire MRTG polling loop to hang, so I wouldn’t get any monitoring. In response I split the script, one to read the usage to write to a log, and another (called by MRTG) to read the latest entry in the log.

The reading script would still get stuck, so I added a generous timeout and stopped thinking about it. Until I thought it would be a good idea to also drop the usage data onto MQTT for Home Assistant to grab. Nothing wrong with this, except that I upped the frequency from once every five minutes to every minute. Sometimes it would fail for hours, sending an email with the error every minute…

I initially noticed the increase in timeouts around the same time that I upgraded the hardware in my linux box and did a fresh install of Ubuntu. Due to default permission changes I found that I could no longer simply cat the serial port, I needed to connect using Device::SerialPort, so I was thinking that there was something wrong with the usb-serial chipset or the driver.

That was until a few weeks ago, when I upgraded the hardware in my Windows desktop.

For a long time I have been aware of a slight ticking sound coming from that computer, you know the sound of a fan slightly hitting something or being slightly off balance?

The new build went great and although I didn’t really need the upgrade it let me do an upgrade that was required, to my parent’s computer. As I was handing down parts I thought that I could replace the power supply fan instead of buying a whole new power supply, so I powered it up and heard the same noise…

… coming through my speakers.

Oh, this is not a bad fan, this is a failing power supply.

So what does serial port problems in one computer have to do with a bad power supply in another one? Probably nothing, but since replacing that power supply I haven’t had a single timeout error, it is as stable as when I first picked up the adapter. I guess it is possible that the power supply was generating interference at around 2.4GHz…

Another thing I didn’t think was related was that the UPS that these two computers run from had started to shut down. Not in any graceful manner, but with a sudden power off followed by a loud continuous beep. I suppose it is also possible that a bad power supply could also trip the UPS…

Tagged with: , , ,

Monitoring power

Tuesday, November 26th, 2013 at 9:18 pm

While some people are heavily against smart meters, I am not concerned. In fact when I received the initial notification about the installation my first thought was about the possibility of playing with new technology.

When they finally installed the meter it coincided with a co-worker getting solar panels installed which led to a number of conversations about ways to monitor power generation and consumption. Based upon previous research I had done I knew that other people in Melbourne had had success with Rainforest Automation devices which use the ZigBee Smart Energy wireless protocol to connect to the smart meter.

My co-worker opted for the EAGLE as his solar panels also connected over ethernet, while I opted for the (cheaper) RAVEn which is a USB dongle.

Once the devices arrived I had to wait a few more days until my smart meter was completely setup, but then it was a simple matter of entering the address and install code from the RAVEn into the online portal for the smart meter. After a quick test with the windows app, I plugged it into my Linux box to see it come up as a serial port.

It was nice to simply run cat /dev/ttyUSB1 and see it output an XML fragment reporting the instantaneous demand every few seconds.

The documentation for the XML API showed me how to send commands and how to convert the hex values to decimal and how to apply the multiplier and divisor. It didn’t take long to wrap this in a script to be called by MRTG:

Events visible in this sample include:

  • Hot water turning on at 11PM AEST and topping up a couple of times
  • Fan and light while getting ready for work
  • Fridge cycling on during the day
  • Use of oven to cook dinner
  • Lights and other usage when I am at home

Due to complications with having off-peak electric hot water (aka a ‘controlled load’) I do not get summation values, instead that is just a graph of the instantaneous load. So while it doesn’t give true usage, it is a good enough indicator.

Currently this graph is available to anyone who knows the URL, and while it can let someone determine when I am or am not at home, the easier way to do that would be to go into my backyard and look through a window.

Tagged with: ,