Internship Report by Elliot Shine

                                                        

                                                   Trek (on the rocks) and Jonty (in the water) manning the camera.

My summer internship at AusOcean has been one of the busiest, yet most rewarding periods of my life. The experience offered a unique perspective on our oceans and the challenges they face. Working on a small team allowed me to be deeply involved in every stage of the engineering process, providing invaluable learning opportunities and hands-on experience. My work focused on three main projects: integrating a hydrophone into our underwater camera system, developing a stereoscopic camera, and creating an automatic fish detection system.

Integrated Hydrophone

My first internship project involved integrating a hydrophone (underwater microphone) with our existing underwater camera system. Although the hardware integration was straightforward (except for one instance where the Raspberry Pi pins were not all soldered so the microphone didn’t work), the software aspect proved challenging. Our current software architecture was designed for either video-only or audio-only inputs, which made combining the two within our existing pipeline very difficult.

To create a minimum viable product, I worked around this limitation by utilising an existing program called FFmpeg. I developed a program that captured audio from the hydrophone and video from the camera, feeding both into FFmpeg. FFmpeg then combined the two and live-streamed the result to YouTube. This worked, but we ultimately decided to put the project on hold until we could fully integrate it into our own software pipeline.

Stereoscopic Camera

The second project that I worked on was an underwater stereoscopic camera with my colleague Jonty. A stereoscopic camera allows a viewer to see in 3D, which creates a much more immersive experience. Stereoscopic cameras can also be used for scientific purposes such as measuring the size of fish. Scientists use a system called BRUVS (Baited Remote Underwater Video Systems) to capture fish population data. Currently these deployments last only a few days at most and the video analysis is done manually. This involves watching the entire recording (which are hours long), manually selecting the nose and tail of each fish and identifying the species. Our camera will allow a more permanent deployment, allowing for continuous monitoring and potentially bait free operation. Our end goal is for all analysis to be done automatically, allowing scientists to focus on more important tasks and allowing more data to be collected.

Our first prototype was simply zip tying two of our cameras together with some rubber between them to stop them from slipping, which worked surprisingly well. We were able to successfully record synchronised video and view it in a VR headset, which proved that the project was feasible.

Zip ties solve any problem.

On our second prototype, we screwed the cameras together using some bolts and clamps, which was much more rigid. We also made a box that could run the cameras in the field and a program that allowed us to preview the cameras and easily record footage. 

The second prototype of our stereoscopic camera.

The portable stereoscopic box.

A screenshot of the stereoscopic camera capture interface.

We went on a field trip to Second Valley to record some underwater footage to see how well the camera works. We could not have asked for a better day to do it, the water was crystal clear and the skies were blue.

When we first tested the footage, it was very uncomfortable to watch and I was worried that this project would be dead in the water. It turns out that we had accidentally swapped the cameras around 🤦. This was thankfully an easy fix and when we swapped them back, the footage was much more comfortable to watch.

A screenshot of some of the footage we got.

Our next step for the stereoscopic camera is to work on measuring objects using computer vision.

Automatic Fish Detection

The last project that I worked on was automatic detection of fish on livestreams. The groundwork had already been laid for me as we had previously created a platform called openfish to annotate streams but didn’t reach the stage of automated detection. My first task was to integrate openfish into AusOceanTV, our online streaming service. This took a while but it ended up with a result that I am quite pleased with. This will allow members of the public to contribute species identifications and missing fish, creating a dataset that can be used for scientific research and to improve the model in the future. Once there are enough species identifications, we may be able to create a model that automatically identifies species as well.

A screenshot of the annotation viewer with some automatically detected fish.

To create the automatic detection program I started by investigating some existing image recognition models and came across YOLO. I tried using some of the pretrained models that come with it and they had limited success.

I don’t think those are birds.

I decided to train my own model using the Australian Institute of Marine Science’s ozfish dataset and see how well it would transfer to our footage. It worked surprisingly well, capturing most fish.

The model worked surprisingly well.

The next step for the fish detector is to integrate it into our video processing pipeline so that fish are detected in real time.

My internship at AusOcean has been an incredibly rewarding and enjoyable experience. I am grateful for this opportunity and will always cherish my time here. The entire AusOcean team was exceptionally supportive and helpful, and I hope to continue collaborating with them in the future.