Administered by:
Eleven things I've learned about wrangling half-hourly energy data.
http://blog.ldodds.com/2025/02/24/falsehoods-this-programmer-believed-about-half-hourly-energy-data/
@ldodds are there any issues with decimal precision? ISTR that one of my solar panels reported to 10dp which was, undoubtedly, just some floating point nonsense.
@Edent hmm, not that I've noticed. But I will take a closer look. We're usually getting data via a solar monitoring system.
My biggest issue with solar is when the various meters don't align because they've suffer independent connectivity issues.
@ldodds Awesome, will read in full later.
I didn’t know you had a blog so that’s another for my list
@ldodds and no matter how much data you think you have there still seem to be gaps that make calculating value vs cost harder than you'd expect
@ldodds What kind of device can I use to read data from a domestic meter? Me and folks from @LaCorrienteCoop have been wondering about this for a while.
One of my pet projects is a PV surplus manager and I'd love to get data from the meter, am not afraid of soldering stuff.
@IvanSanchez @LaCorrienteCoop am only familiar with UK, you'd normally have to get data from your supplier. Or, if you've got a smart meter through a service like https://carbon.coop/portfolio/smart-meter-data-service/ or by buying an in-home device that supports bridging to wifi.
You can also get some "clamp on" meters which you either wire into your circuits or clamp onto cables which can do monitoring via Zigbee/Wifi. No personal experience with those but they do work for some use cases.
@IvanSanchez @LaCorrienteCoop wouldn't recommend attaching anything to your actual meter though. And if wiring in an extra device, then get an electrician to help. Although I think the clamp on devices just wrap around a cable.
@ldodds @LaCorrienteCoop Yeah, I'm already familiar with CT clamps, and I can interface with stuff like a PZEM-004T or Shelly plugs or HomeAssistant.
But the meter communicates with the supplier by some kind of PLC protocol (supposedly IEC 62056-21?) and I've been looking for the means to listen into that PLC channel from inside the house.
Here (Spain) meters work in 15-minute intervals and the PLC bits of the system *shouldn't* be too different from the UK.
@ldodds just reading this and feeling slightly relieved that I don’t have to think too much about this at the moment!
@jamesjefferies @ldodds I take this and raise you the railway
@ldodds @Edent you mentioned you didn’t know of any way to programmatically get the gas calorific values. There is (or at least was) a way: I used it in my smart meter script I published here: https://github.com/richardloxley/smart_meter_to_emoncms (Sadly that script no longer works as n3rgy turned off their public API - I’m not sure if the National Grid calorific value API is still working).
@richardloxley @Edent fantastic, thank you will take a look.
@ldodds you bring back fun memories from when I worked on smart meter analytics. Daylight saving was especially fun, when two different meter manufacturers did it completely differently. With one brand returning 46 reads and 50 reads on the change overs.
@ldodds time change means you have 1 day with 50 readings, another with 46.
Other assumptions I made:
HH data always uses the same time zone. Sometimes it's in Europe/London and changes on time change, sometimes it's in UTC and never changes.
HH data is universal. Different countries use different standard intervals so if you go international, it becomes interesting.
Getting access to HH data is easy.
HH data feeds send updates every day.
HH data feeds use a secure connection.
@ldodds
Assumptions, continued:
Identifying that an HH data feed has stopped sending data is easy.
Identifying spurious or missing data points is easy.
<insert your own phrase here> is easy.
@brunogirin nice list. I should have noted the time zone issue, will add this.
Interested in the different standard intervals internationally, any pointers on that?
@ldodds I'll see if I can find some pointers but I've seen 10 and 15 minutes intervals.