Making everything a connected device

Abstract

Today we are definitively getting to a point where it becomes challenging to draw the line between a household product being bought and built to serve its consumer, and that same product being designed and equipped for serving other entities beyond the knowledge of its official owner.

Many forms of technological progress converged to get us to a point where it becomes ambiguous and unclear what is the full scope of the sophisticated technological products that are all around us and so often extremely affordable, in spite of its immense capabilities. One of the most obvious examples is of course the smartphone. Its computing power often overlaps that of a personal computer, while still fitting in a pocket and having a plethora of sensors and hardware features, some of which the user may never know these are present and for what purpose these serve.

At home, is more and more the normal scenario to have the most prosaic devices, such as light bulbs and switches, all equipped with a chip that renders these connected to the Internet. Once this is done, all bets are off. For the consumer is very convenient, because at the reach of an app, one can control and monitor his property or perform tasks remotely.

Today is a bit of a gamble. Between the comfort and convenience provided by technology in the realm of the home environment, there is also the security risk, privacy and the user being manipulated and targeted by manufacturer campaigns, or its information sold to third parties for similar purposes.

Still, however powerful and discrete these technologies might be, the greatest joy lies in leveraging its power for the own use, and that is, with no strings attached, and the user really running the show.

That is where I like to sit at. It is true that for IoT tech to become so abundant, cheap, and with routes for open use, the fact is that intentionally or not, manufacturers (regardless of their agendas) gave way for the communities to build on their hardware, but ultimately not being dependent of their clouds and platforms. Projects like Tasmota or ESPHome are one such example, where off-the-shelf products like bulbs and switches are often modified/reflashed to run one of these firmwares.

Concept

Here I am presenting a bit of a different approach though. Starting with a very conventional insect zapper with nothing that can be pinged from the Internet (no electronics whatsoever -  a couple of light bulbs, a ballast, two starter switches, and a Cockcroft–Walton multiplier for generating the HV to zap the insects), I wanted to integrate it better in home automation without requiring external devices such as Wifi sockets or such. 

Implementation

And so I went. The first iteration was to put an ESP32 relay board on it. I used the "ESP32_Relay X1" board which I had used in previous projects with good results. This board has the advantage of having an integrated 230 V insulated SMPS, making it very convenient for integrating in mains powered devices:

It was a relatively uneventful assembly. Drilled some holes in the case and attached the board with the help of some standoffs.

Worked just as it was supposed to. Installed Tasmota32-nspanel (yes, this variant, because with the regular build, the relay GPIO will temporarily go high on boot-up because by default the firmware uses this pin and a few other pins for communication with the PSRAM chips - a poor routing choice from the board manufacturer).

But then for me it felt boring to just be able to turn the zapper on and off. What about if I could also detect and count the number of times the zapper would kill insects? That could be fun.

And so I decided to go to the 2nd stage. The ESP32, as some of you might know, has plenty of GPIO ports, and is overall a much more eclectic chip than its predecessor, the ESP8266. One can connect digital signal inputs, as well as DC analog signals to some of these pins, and use the data and events for their projects.

My challenge was to be able to sense the insect getting zapped. My first assumption was that when an insect gets zapped, there is a burst of current flowing across the HV power supply of the zapper. But measuring this HV current could be tricky do to safely and without a lot of precautions. Also I did not want to spend too much time with this project and to risk hardware and especially my health with iterations. So I went for an option that seemed a bit safer and that could also provide results. The idea was to measure the mains AC current on the input side of the HV power supply. 

Unsurprisingly in most products of this kind, this device uses a Cockcroft–Walton multiplier for generating the HV. There is basically a set of diodes and capacitors (9 capacitors and 9 diodes to be more exact) where each pair of diodes and capacitors forms a stage. In this case we have 4.5 stages, which means that the output voltage can reach close to  2 * 4.5 times the input voltage Vpp. In this case it means approximately 2070 Volts.

This curcuit also has three resistors. One or two I presume are for limiting the input current (without it there would be one massive jolt of current for recharging the capacitors), while the other acts as a bleed resistor, to make sure the capacitors are discharged before someone might think of putting the fingers where is not supposed to.

So my idea was to (non-invasively) tap into the phase or neutral lines of this AC input in order to measure the current. My expectation was that during and after a zap there should be a flow of current, therefore making it possible to detect the insect getting zapped. For that effect I grabbed one of these CT boards sold cheaply on Aliexpress:

I did a modification on it however: first these have a capacitor that removes the DC offset from the LM358 opamp output. I didn't want that, because I had to supply a positive signal to the ESP32 ADC input. Secondly, these boards output a voltage that varies with the current signal. In this case the AC signal could be difficult to deal with. So I removed the capacitor, replaced it with a diode, and added a 4.7 uF capacitor in parallel with the output. This way I had a stable signal with no ripple, and more or less proportional to the current. Easier to measure by the ADC.

Connected it to GPIO34, which is the pin behind ADC1 of the ESP32, one of the pins that can be used by Tasmota:

At first, after this modification, I noticed stability issues, i.e. often as I would turn on the zapper, the ESP32 would restart. Because before this modification this behaviour didn't occur, I suspected that it could be related to noise being induced to the 3.3 Volt line due to the mains being rapidly switched. As I was powering the CT board from that rail, and as the wires had to cover some distance, I suspected this could be the cause. That is a particularly sensitive rail, as it also powers the ESP32 microcontroller.

In order to fix it, I added a ferrite bead to the 3.3 Volt rail, and twisted the wires going to the CT board around each other. And, yes instability was gone.

Once the hardware work was completed, it was time to play with the software! My first idea was to use Tasmota rules to detect when the ADC input would measure a value above a certain threshold. I tried this approach but it proved unsuccessful, as the short current pulses would rarely get detected. It is not clear from any of  the project documentation, but apparently rules have evaluation periods that are too long for this scenario.

Without that option, the next best thing I could think of, was to define a driver in Berry (yes, with ESP32, Tasmota opens a whole new dimension of flexibility, by providing a scripting language in the firmware itself, that among many other things, allows the user to code drivers for devices not supported natively by Tasmota), and use its every_50ms callback to put the pulse detection cycle inside.

As such, in a driver we can define code that will get called every 50 ms (that is the maximum you can go). Taking advantage of that, I wrote a small driver that would read the ADC1 pin, and check if the value would be above a certain threshold or not. The threshold I have empirically verified as being a probable discharge and less likely to be noise related. Everytime the threshold is crossed, the driver will publish a message to a specific MQTT topic.

import mqtt

import json

import string

event_threshold = 2110

class ZapperController

def every_50ms()

var sensors = json.load(tasmota.read_sensors())

var zapper_current = sensors['ANALOG']['A1']

# Update the sensor state:

if zapper_current > event_threshold

mqtt.publish("tele/ovalesublime-insect-zapper/EVENT", "{\"event_type\": \"zap\", \"detection\":" + str(zapper_current) + "}")

end

end

end

zapper_controller = ZapperController()

tasmota.add_driver(zapper_controller)

To have a consumer for that event, I configured Home Assistant to subscribe and handle it as a MQTT event:

mqtt:

event:

- name: insect_zapped

state_topic: 'tele/ovalesublime-insect-zapper/EVENT'

event_types:

- "zap"

availability_topic: "tele/ovalesublime-insect-zapper/LWT"

payload_available: 'Online'

payload_not_available: 'Offline'

qos: 1

Testing

For testing (because you don't have flies around when you need them), I picked up a plastic stick, attached a strip of paper to it, and wet the tip. Then by approaching that paper tip from the electrode bars I was able to provoke a discharge the same way an insect would.

And sure enough I was able to produce events:

Conclusions

If you ask if this solution for detecting events is ideal, I would probably say no. First, relying on the 50 ms polling of the Berry driver is not 100% reliable, especially because we are dealing with short lived events. Second, we are capturing a relatively small current that is difficult to discriminate with a current transformer (CT) that is rated for 5 Amps. It is sufficient, but not a strong unambiguous signal. Third, for events we should resort to whatever provisions exist in the microcontroller and the firmware to handle these events as interrupts. This ensures that the processing is fully asynchronous and no events are missed (the processing will forcefully be triggered as a pin state change is detected). My challenge with applying this  approach is that I didn't have enough time to add the hardware for conditioning the signal coming out of the CT, in order to produce a high signal in case of being above the threshold, and a low signal (0V) if below. This could have been implemented by adding a comparator and an adjustment pot. Unfortunately the ESP32 doesn't seem to have a comparator module internally, which would have been very useful for this application.

Monitoring and controlling the swimming pool water level

Ever since I became owner of a swimming pool, a whole new set of chores and responsibilites were unlocked. Some are the repeating manual tasks of keeping the pool and the water clean for human use, while others are more technical and infrastructure related tasks that go along with keeping this very special reservoir of water that so much importance acquires in the hot days of summer.

While in a typical swimming pool some basic features are automated (for example controlling th pH, running the filtration or generating the correct amount of chlorine to disinfect the water), there are many others which by default are left to the intervention of a human.

One of the aspects that got me particularly concerned was the maintenance of the correct water level not being an automated process. Especially during the summer, a substantial amount of water is lost every day (at least 5 mm per day - depending on the environmental temperature and humidity). The water level dropping too much is problematic, because as the level approaches the bottom of the skimmer opening, it will cause the pump to ingest air, cavitate and unprime. If this persists for too long the pump will get damaged as its seals and bearings will seize in the absence of water.

With that in mind, I tried to come up with a solution with the following aspects in mind:

• simple design without many failure modes;
• be as much as possible based on off-the-shelf and easy to replace components;
• possible to monitor and remotely control if necessary;
• integrate with home automation;
• physically discrete and without any aesthetical impact to the swimming pool and its surroundings;
• not very expensive to implement and maintain.

While there are different types of solutions that one way or another cover the water level control aspect, these are in general either expensive, limited in terms of integration with other home automation platforms, or impossible to control remotely.

As such I decided to call for my ingenuity, and put together a custom solution that would tick all the boxes.

So basically the solution needed two components: a sensor installed in the swimming pool to detect the low level, and an automatic valve to inject tap water.

The sensor

Because I had good experience with Zigbee devices for their long battery life and ease of integration with home automation platforms, I decided to go for a very particular approach in respect to the sensor: retrofit a Aqara water leak sensor the serve the opposite purpose - detect lack of water instead of presence:

While ideally I would be looking for a proportional indication of the water level, so that for example I could also detect an excessively high water level (which will occur when it rains heavily), I could not readily find a solution that would not imply much more complexity and eventually custom parts.

The simplest approach would be to place this sensor inside the swimming pool, and have it setup so that when it would not detect water, the home automation would kick in and activate a valve. This would be the most basic approach . However it would have some implications: first, even though the Aqara sensor is waterproof, it is likely not designed to be permanently in contact with the water, and especially not water containing a certain concentration of chlorine and other chemicals. Also placing the sensor practically at the water level would seriously degrade the Zigbee signal strength. Lastly, as the water oscillates most of the time, debouncing logic would have to be added to ignore the false positives.  This would also carry the implication that battery life would probably plummet, as the device would have to transmit much more often than normal.

This led me to think about a slightly more sophisticated approach. First, as I did not trust the sensor to be robust enough to tolerate being in contact with the water, I went for designing a custom enclosure where the Aqara sensor could be mounted inside and stay safe and dry all the time.

Because I became accustomed with modelling 3D objects for functional purposes since I bought a 3D printer, it proved to be a no brainer approach to create this enclouse myself and print it.

And so it was. The idea was to use a float switch connected to the Aqara sensor electrodes, so that when the water level would be low, the switch would close and trigger the sensor. For convenience and for keeping it visually discrete, the most logical choice was to put the device inside the skimmer cavity, supported under its cover:

Soon I learned that leaving the float sensor directly exposed to the water surface would lead to a lot of detection events, even while the water level was still substantially high.

This first version was still not based on 3D printed parts but rather on off the shelf parts cobbled together. Soon I found that the plastic enclosure was not a good solution as it would crack and let water in.

That is when I decided to make a 3D printed enclosure from scratch.

This enclosure would solve the excess events issue by measuring the water level not directly from the pool surface, but from the level of water inside a cup that would be part of the bottom of this enclosure:

This cup forms a kind of low pass filter for water waves: as the water is only be allowed in via small orifices, it takes some time for the level to rise and lower, effectively providing a damping behaviour to the variations in the water level as observed in the surface of the swimming pool.

So far, the resin material of the enclosure did not show any signs of wear due to the water or the chemicals present in it, and proved effective at keeping the sensor protected from water ingress. At the top cover right at the end of the threading, I have added a rubber seal, which is essential for keeping the enclosure water tight.

The actuator

With the sensor aspect resolved, it was time to cover the other aspect of the solution: how to automatically inject water into the swimming pool when needed?

Slightly overwhelmed with the idea of messing around with the swimming pool hardware, at first I considered using a modified sprinkler to send a jet of water to the swimming pool directly. But this idea seemed a bit cumbersome and would inevitably make the solution less discrete than desirable.

That is when I realized that it would not be such a big challenge to tap into the PVC pipes initially desiged for the installation of a heat pump:

With these I could effectively inject water into the swimming pool in a very discrete manner. Because the public water supply has a much superior pressure compared to the pool filtering pump (4 + bars compared to less than 1 bar for the latter), there should be positive flow even when the pump is running.

As so it was. I had to study a bit of plumbing in order to determine what fittings I had to buy and how to glue PVC-U piping parts together, but at the end it was a success:

Basically I had to add a 2" pipe  to 3/4" inch reduction fitting, followed by a 3/4" elbow, followed by a retention valve (I wanted to make sure it would not be possible to have water from the swimming pool flowing towards the water supply - however unlikely that would be because of the pressure differences). Lastly the most important bit, the electronic valve that would open the water for injection.

For that I have chosen the GiEX QT06 Zigbee solenoid valve. This valve is normally used for garden irrigation, and it has the interesting feature of having a flow sensor that measures the injected water. Very useful for this application as well.

Before this valve I am also using a motorized ball valve which acts as a safety valve which I can close when not injecting water into the pool or having the irrigation running from the public water supply:

The integration of this valve is a bit more custom and uses a ESP8266 Wifi controller running Tasmota.

Because I am using Home Assistant as the "brains" of the house, and multiple Tasmota and other devices that communicate via MQTT messages, for integrating with these Zigbee devices I resorted to using a Sonoff Zigbee bridge flashed with Tasmota (https://zigbee.blakadder.com/Sonoff_ZBBridge.html) and Zigbee2Tasmota (on the Zigbee coordinator chip).

Then with all the hardware prepared and the Zigbee devices paired to the bridge (very simple procedure, some information can be found here - https://tasmota.github.io/docs/Zigbee/#introduction), this is where all the fun begins. 

Configuring Home Assistant

On HA we need to define the entities that correspond to the Zigbee devices we are adding. In this case the level sensor and the valve.

For the valve I defined the following MQTT switch in the HA configuration.yaml file:

mqtt:

  switch:

    - name: swimming_pool_water_injection

      state_topic: 'tele/zbbridge/SENSOR'

      value_template: >

        {% for key, value in value_json.ZbReceived.items() %}

          {% if 'Name' in value and value.Name == 'valve' %}

            {% if 'WaterState' in value %}

              {{ value.WaterState }}

            {% endif %}

          {% endif %}

        {% endfor %}

      command_topic: 'cmnd/zbbridge/zbsend'

      qos: 1

      payload_on: '{"device":"valve","Write":{"WaterState":1}}'

      payload_off: '{"device":"valve","Write":{"WaterState":0}}'

      state_on: '1'

      state_off: '0'

      availability:

        - topic: "stat/zbbridge/RESULT"

          payload_available: true

          payload_not_available: false

          value_template: "{{ (value_json['ZbStatus3'] | selectattr('Name', 'eq', 'valve') | map(attribute='LastSeen') | first | int) < 14000 }}"

      retain: false

This implies that we first gave the valve a friendly name on the Zigbee bridge, via the ZBName command. For example in the Tasmota console:

zbname 0x2916,valve

Regarding the sensor, I have defined the following MQTT binary_sensor:

mqtt:

  binary_sensor:

    - name: swimming_pool_level_low

      state_topic: "tele/zbbridge/SENSOR"

      value_template: >

        {% for key, value in value_json.ZbReceived.items() %}

          {% if 'Name' in value and value.Name == 'level-sensor' %}

            {% if 'Water' in value %}

              {{ value.Water }}

            {% elif 'ZoneStatusChange' in value  %}

              {{ value.ZoneStatusChange }}

            {% endif %}

          {% endif %}

        {% endfor %}

      device_class: problem

      payload_on: '1'

      payload_off: '0'

      availability:

        - topic: "stat/zbbridge/RESULT"

          payload_available: true

          payload_not_available: false

          value_template: "{{ (value_json['ZbStatus3'] | selectattr('Name', 'eq', 'level-sensor') | map(attribute='LastSeen') | first | int) < 3300 }}"

This exposes the Zigbee level sensor as a binary sensor in Home Assistant, allowing it to be used in monitoring and automations.

For the automation I considered important to add some hysteresis (not trying to compensate the water level on every received event), and alert and give up if it tries to compensate too often:

- id: swimming_pool_water_level_low

  alias: swimming_pool_water_level_low

  trigger:

    - platform: numeric_state

      entity_id: sensor.total_swimming_pool_level_low

      above: 3

  condition:

    condition: and

    conditions:

      - condition: template

        value_template: "{{ (as_timestamp(now()) - as_timestamp(state_attr('automation.swimming_pool_water_level_low', 'last_triggered')) | float) / 3600 >= 24 }}"

      # Data is only valid if it is being updated through the Zigbee gateway. If it is down, we cannot trust the sensors:

      - condition: template

        value_template: "{{ states('sensor.zbbridge_wifi_connect_count') != 'unavailable' }}"

      - condition: numeric_state

        entity_id: sensor.level_sensor_last_seen

        below: 3100

  action:

    - service: script.multi_notification

      data:

        title: "Swimming pool level low!"

        message: "Swimming pool level low! Compensating level."

    # Set the zigbee valve to track the water quantity:

    - service: mqtt.publish

      data:

        topic: "cmnd/zbbridge/zbsend"

        payload: '{"device":"valve","write":{"WaterMode":1}}'

    - delay:

        seconds: 4

    # Set the desired water quantity:

    - service: mqtt.publish

      data:

        topic: "cmnd/zbbridge/zbsend"

        payload: >-

          {% set target_volume =  states('input_number.swimming_pool_water_injection_amount') %}

          {{ {"device":"valve","write":{"IrrigationTarget":target_volume}} | tojson }}

    - delay:

        seconds: 4

    - service: switch.turn_on

      target:

        entity_id: switch.swimming_pool_water_injection

    - service: switch.turn_on

      target:

        entity_id: switch.garden_valve

This automation will only starting injecting water if the level is low for more than 3 hours. Also it will refuse to compensate if the last time it injected water was less than 24 hours ago. Because this Zigbee valve has a flow sensor, we are able to tell it exactly how much water (in 1 liter increments) we want to add to the swimming pool.

Once all of this is setup I can monitor everything from the distance of a web browser and a comfortable chair:

Conclusion

With this solution I was able to monitor the water level, have Home Assistant inject water automatically and have one less thing to worry about especially in the Summer where this solution is most necessary as I am physically away from the site to be able to inject water manually.

In the future I plan to also support the opposite, which is to be able to remove excess water when the level is above the skimmer opening. This is also an important addition because in the most rainy periods of the year water will regularly rise above the skimmer port and render it ineffective at colecting debris. Also there is the risk of water infiltrating behind the liner, causing problems to the adhesion between the latter and the pool walls, ultimately leading to its detachment.

Turning a doorbell switch into a home security camera...and still retain its original function

Today there is an off-the-shelf solution for practically every average-Joe-grade idea one may have. There is always the chance that when one thinks of something that could have a practical purpose and cover a specific need, that there is already a product in the market that will do that function.

Even though home automation may still be in its infancy, today we see a market practically flooded of solutions that one way or another aim to respond in terms of comfort, or to help improve the home experience, or even to enhance the safety and security of its occupants.

Tasmota-based multi-channel air quality sensor station

The sense of smell is an amazing function in vertebrate animals. Even the human nose, which is not particularly notable in comparison with other animals, is fascinating in its sensitivity to trace amounts of a large variety of substances. It has a level of selectivity and capability of distinguishing between different kinds of smells, which is very difficult to match by even the most sophisticated artificial sensors.

Having as the main motivation the enrichment of the IoT gadgetry sitting around in my house (and eventually fill up an entire 192.168.1.1/24 network address range), I found that (after the motion and entrance detection device), an interesting device to invest time on would be something capable of sensing multiple air quality parameters at the same time.

ZMAi-90 (or SMTONOFF WDS688) DIN rail meter/switch - more details on GPIOs and configuration

As an update to the previous post where I have shared the details on how to "Tasmotize" this device, I am adding more detail on what is the physical assignement of GPIO pins from the ESP8266, to other components in this device.

Given the pins from the ESP8266 microcontroller that are exposed in its breakout board (which in turn is SMD soldered to the main PCB):

Intrusion / motion / door opening detector using a NodeMCU, some sensors, and Tasmota

The Espressif Systems chip manufacturer kind of created a revolution by opening the door to the creation of dirt cheap chips for building IoT devices. Its low cost led to introduction in the market, of many types of home automation devices, such as switches, light dimmers, smart bulbs, HVAC systems and what not.

On par with that, the open source community quickly became an interested party as well, and this led to the introduction of tools for quickly allowing developers to write interesting applications for practically anything based on these chips. It is the case of the Arduino core for the ESP8266 chip -  https://github.com/esp8266/Arduino. This allowed for Arduino IDE accustomed developers, to write their own code, and either replace the original firmware on commercial devices featuring the ESP8266, or use open-source board designs such as NodeMCU and build their own custom devices.

Reverse engineering the ZMAi-90 DIN rail meter/switch and integrating with Hass.io using Tasmota - Part 2

I finished the first post with  a tone of optimism, in spite of not being quite there yet. But this time I'm bringing the complete story, with something which hopefully can be a useful takeaway for most users.

Initial analysis of the MCU communication

After figuring out what kind of communication was going on between the ESP8266 and the Vangotech V9821 chip (the specialized MCU which does all the metering functions - and a bit more which I will go in detail afterwards), I got a bit puzzled with the output and its consistency. I first connected a known AC current source through the shunt mounted in the relay's output rail, and in the middle of a stream of garbage, some values seemed consistent with the current I was putting and  being shown in the device's display.

I still cracked my mind at trying to figure out a pattern (I felt as if I was trying to incarnate John Nash while looking for patterns in seemingly chaotic data), and trying to prove assumptions such as the last byte being a checksum. But nothing fruitful came out of that first iteration.

Attempting to reverse engineer a home automation oriented smart-meter - Part 1

In my quest to make my house smarter, but still looking forward to keep having control over it, I have been doing some additions which I plan to further document here, in the short term.

In the meantime I thought it would be more relevant to share my findings in regard to a device a bit more "exhotic" than the Sonoff boxes we are all used to. This device is a sort of a miniature smartmeter that fits in a DIN rail next to the circuit breakers.

Just like the Sonoff modules, it also pairs with your WLAN, and connects to a cloud service. Instead of the eWeLink cloud to which Sonoff devices connect to, in this case it connects to another relatively popular cloud service called Tuya.

Building a kick-ass home automation by reflashing the Sonoff devices with Tasmota and getting it all working with hass.io

For some time I have been gradually bringing more devices to my house, which are either designed or having features allowing these to be integrated to a home automation system.

In spite of all the concerns that can arise from bringing smart/connected devices to the place where you expect personal privacy to exist, the convenience of having these ends up speaking louder overall..

It all started with having a set of unrelated devices in the house, each featuring connectivity and some cloud-based features provided by the vendor. This is the case for the Xiaomi Rockrobo vacuum cleaner, the Sonoff switches, the multimedia devices such as the TV set (an LG smartTV), and also the Google Chromecast and Assistant devices.