Using AI to keep bears, humans and livestock safe

So, there I was, backpack full of equipment on my stomach, climbing a cage ladder up into an abandoned 4G tower on top of a hill in rural Romania.

To find out which sequence of events eventually led to me taking my career to new heights (pun intended), we have to go back a few months to the start of an unusual project.

In this blog post, I'll take you through the process of how we developed a bear-repelling system for use in the Carpathian mountains in Romania, including all the interesting challenges that came with it.

Why?

A while ago, Hack The Planet, Q42's tech for good venture, was contacted by Carpathia, an NGO that strives to preserve and restore as much nature as possible in Romania. The thousands of hectares of land and forests that they manage, houses a large population of wild bears. Bears are one of the few species of animals where their cuddly appearance is severely inversely related to how cuddly they actually are. Fortunately though, bears don't really like being around humans. So, what’s the problem?

As it turns out, when bears wander into villages or farms, they can get used to humans and figure out that a farm is basically a McDonald's for bears. When this happens, it becomes almost impossible to keep them away. Those bears have become so-called “problem bears”. Since these problem bears are incredibly difficult to capture, the last resort is often to kill them. That's why it's better for both humans and bears if they don't get too familiar with each other.

Carpathia had read about the Elephant Repeller work and anti-poaching project that Hack The Planet has done in Africa to mitigate human-elephant conflict. They wanted to know if we could also help them mitigate human-bear conflict.

This is where it all started. Thijs and I got to work, starting development on a system that would have to keep bears away from the picturesque Romanian village of Rucăr.

Step 1: Teaching a computer what bears look like

We knew we'd be using camera images for our bear detection mechanism. The first thing on our to-do list was therefore teaching a computer what bears look like, i.e. training a computer vision model.

Fortunately for us, Carpathia was able to supply us with roughly one terabyte of photos and videos from their wildlife monitoring cameras. We extracted stills from the videos and then ran all the images and video stills through Mega Detector, a pre-trained model that is great at drawing bounding boxes around animals. When we combined the Mega Detector bounding boxes with the image labeling that Carpathia had done already, we had a dataset to train our very own bear-detection model!

We initially tried our luck with Google's Vertex AI platform, but we eventually ended up getting the best result training a custom model with YOLOv5.

Step 2: Stealing camera images

Now that we had a way to detect bears on images, we needed a way to capture images in the field and feed them to the computer vision model running on our cloud infrastructure.

The area in Romania where we'd be installing the detection cameras and repellers has 4G coverage. This meant that we could just use off the shelf, 4G-enabled, camera traps. A camera trap is a camera with a motion sensor. Whenever it senses motion, it takes a picture.

A key challenge here was speed. We wanted to get the images from the cameras into our system as quickly as possible. After all, a bear-detection is useless if the bear is already long gone by the time the detection is completed. We also wanted to make sure the system could work with various brands of 4G cameras, so that we would not get locked into any proprietary systems. Most 4G camera traps allow you to enter your own SMTP credentials that are then used to email the camera photos to a specified email address. We decided to abuse use this feature for our purposes. We set up our own little server that pretends to be an SMTP server and then pointed the 4G cameras to it. Whenever the camera takes a photo, it will then use our pretend SMTP server to 'email' the photo. But what happens instead is that our server just sends the message contents straight to our detection service. No inboxes to setup, manage, or check. And no email delays. Just a stateless SMTP to HTTP relay (with a bit of authentication sprinkled in).

Step 3: Building a detection cloud platform

Setting up the service that would scan incoming images for bears and execute various actions based on those detections was actually the most straightforward part of this project, so I'll keep it short. Since there were a bazillion different things to do on the project, we wanted a framework for our backend that would be easy to use and would come with a lot of built-in tools and utilities. Another requirement was that the operating costs of the system would be very low.

Those factors combined made us choose Django. Django is a very pragmatic framework which allowed us to develop the backend quickly and even add features whilst we were out in the field. We were able to leverage a lot of Django's built-in tools and features, freeing up time that we could then spend on other things. To keep the running costs low, we deployed everything to Cloud Run, which allowed specific parts of the system to scale down to zero, keeping the running costs low.

Step 4: Learning everything there is to know about LoRaWAN and lead-acid batteries

So, we now had a way to get real-time images from the field and we had a cloud service that could detect bears on images. All that was left was a way to act upon these detections and actually scare away the bears.

For a project in Africa, Hack The Planet had already developed prototypes for so-called 'repellers'. A repeller is a device with a loudspeaker and bright LED light. When it gets activated, it starts emitting loud noises/music and starts strobing its light. For this project, our hardware partner Above, one of our sister companies at ARC, contributed their time and expertise to turn the prototype repeller design into a more field-ready device. The two main challenges that we had to face for the repellers were related to power and communications. We needed to find a way to keep the repellers running for months whilst being available 24/7 for low-latency activations.

LoRaWAN

In order to be able to talk to the repellers, and activate them, we opted to equip them with LoRa radios and have them communicate over LoRaWAN. 4G was out of the question, as keeping a 4G modem running 24/7 would draw far too much power. LoRa on the other hand only consumes a tiny bit of power. The payload for a LoRaWAN package is rather small, but more than enough for our application. There was only one slight issue with this approach. We had never done anything with LoRaWAN networks. Furthermore, there was no existing LoRaWAN network in the area where the repellers would have to be deployed and the scheduled rollout was only a few months away. So, with no time to spare, I dove head first into an endless LoRaWAN rabbit hole. The next weeks were spent researching LoRaWAN classes, selecting the right hardware to set up our own LoRa network and figuring out how to mount and power said equipment up in an abandoned 4G tower in Romania.

Power

Above, our hardware design partner, had managed to get the stand-by power consumption of the repellers down to a mere 0.06 watts. When activated, the repeller obviously consumes significantly more power. Each repeller has a set of lithium batteries in it which will keep it going for a little while. Our objective however was to keep each repeller going for months without human intervention. Since the repellers would be mounted in the forest in Romania, solar power was not a viable option. The solar panels would either be in the shade of the trees, or covered in snow. So, we decided to equip each repeller with a big lead-acid battery as they're fairly affordable and resist freezing temperatures quite well.

Easy, right... Right?

Cue rabbit hole number two. Turns out that most lead-acid batteries, like the ones you'd find in cars, really don't like being deeply discharged. Also, not all lead-acid batteries are suitable for being carried around through the forests due to them containing liquid acid that can spill out. Furthermore, the batteries and connections had to be weather and bear resistant.

Since there aren't really any ready-made, bear-proof battery solutions for repellers - who would have thought… - , we had to make something ourselves.

To add to the complexity, we had to install these batteries in a location over 1700 km away from our office. Since lead acid batteries are really bulky and heavy, we couldn't purchase them all in the Netherlands and ship them out to Romania. That would be a grizzly undertaking.

As a result, we had to design our own, very custom, solution using materials we could source locally in Romania. The design also had to allow us to assemble it rather quickly when we arrived in Romania. What ensued was weeks of contacting Romanian suppliers, selecting suitable parts and building a prototype in the Netherlands. Eventually I ended up ordering deep-cycle, gel-based, lead acid batteries and strong Peli cases in Romania. After that I spent a good week in our makerspace making all the wiring looms for the battery cases.

Step 5: Stress(test)

It was about three weeks before we were scheduled to leave for Romania. We had already shipped all the cameras and some of the repeller components there. The batteries and cases had also been delivered on-site and the backend detection service worked the way we wanted it to. Yet, we were far from relaxed. One critical component wasn't ready yet, the repeller 'brains'. The brain is the little box that contains all the control circuitry that makes the repeller do its thing. Without a brain, a repeller is just a plank with a speaker and a lamp, so it was pretty essential that we'd get them in time for our rollout.

🇳🇱 👋 Hey Dutchies!
Even tussendoor... we zoeken nieuwe Q'ers!

As software engineers, we generally don't have to worry about components being in stock, manufacturing partners and logistics. But this was hardware. We had been working with some first revisions of the repeller brains, but these still contained some significant flaws, as they were essentially pre-production units.

Component shortages etc. had already pushed back the delivery date of the repeller brains multiple times, and we were getting closer and closer to our planned rollout date. Eventually, we got an update from Above and the hardware manufacturer that all the bits and pieces were in place.  So, we made the call to book our flights and start preparing for departure. Everything from then on would have to go right for us to get the hardware in time. The repeller brains eventually made it to our office bear-ly in time, only a mere seven days before our flight. So we immediately got to testing them. This eventually led to Thijs, now our resident repeller expert, going to the office at eleven o'clock at night, one day before we left, in order to implement a firmware stability improvement.

If there's one thing I can give as a main takeaway from a project like this, it is TEST ABSOLUTELY EVERYTHING. With projects like these, there will always be bits where you have to improvise when you're on-site. But, you can significantly reduce the number of things that need improvising by testing everything, down to the smallest details. In the weeks leading up to the scheduled trip to Romania, we tested absolutely everything. Not just our software, but also the functionality of the hardware and all the manufacturing/assembling we'd need to do once we were in Romania.

We even went as far as to take our entire LoRaWAN setup and a couple of repellers to the beach to do a LoRa range test, where we managed to get a 5 to 7 km range.

Step 6: Rollout time

On June 15 2023, it was time to start the rollout. Early in the morning, Tim (who joined us for the rollout), Thijs and myself boarded a plane to Romania, our suitcases packed with repeller brains and tools. On our way from Bucharest airport to the village of Rucăr, we got to enjoy the views of the Carpathian mountains, our work area for the following week.

Once we arrived in Rucăr, our first order of business was getting the LoRa network up and running. Our client, Carpathia, had arranged for us to be allowed to mount the LoRa gateway in an abandoned 4G tower on top of a hill near the village. This brings us to the start of this blog-post. Tim, Thijs and myself climbed up the tower to install our networking equipment.

Then, it was time to assemble 14 repellers and 15 battery cases. Over the course of two days we sat on the living room couch of the house we were staying in, assembling everything we had prepared.

Once all the repellers were ready to go, we headed out with the rangers. For the remainder of the week, we'd be spending our days hiking through the woods whilst carrying a batch of repellers, battery cases and a ladder with us. All in all, it was a good workout, carrying all that equipment up and down the hills surrounding Rucăr. 😅

The way we set up each repeller/camera combination was as follows. The rangers would take us to a known bear-path. Somewhere along the path, we'd set up a camera. Then, about 30 seconds walking down the path, we'd install a repeller and battery case. This way, when a bear approaches the village, they first encounter a camera, and then a repeller will go off in-between the bear and the village, aiming to scare the bear back into the forests.

Step 7: Profit (?)

So, now what? The cameras and repellers are in place, and the system is still doing its thing. On average, it processes about 250 images per day and, at the time of writing, we get approximately one to two bear detections per week. The project is currently in its pilot phase, meaning that the next step is gathering data on how bears actually respond to our repellers and seeing how we can adjust the system to be as effective as possible in deterring bears.

During the development process we knew what the problem was we were working towards solving. Being on-site and seeing the work the Carpathia rangers do in Rucăr really deepened my understanding of the situation at hand. During the week that we were there, on three nights, bears forced their way into farms and attacked livestock, posing a threat to both humans as well as to the bears themselves.

Being able to support the Carpathia rangers in creating and maintaining a unique nature reserve gave a true sense of purpose and value to my work. This project contained a vast and diverse range of challenges that come with trying something that has never been done before. This made the work severely headache inducing at times but also incredibly fun.

Are you one of those engineers that love to sink their teeth into machine learning and use technology for the greater good? Then do check out our job vacancies (in Dutch) at werkenbij.q42.nl!