little-scale: mediapipe

Showing posts with label mediapipe. Show all posts

Saturday, November 25, 2023

Adds basic OSC output for the Mediapipe object detection solution

By default, OSC will be sent to 127.0.0.1 localhost on port 3000 and with address /mediapipe/objects. This can be changed in utils.py

Each OSC message contains: Category name, object detection instance within the frame, normalised x position, normalised y position and certainty

An example Ableton Max for Live device is provided which allows mapping an object class with x and y position to Ableton parameters

Taking the legacy Mediapipe holistic solution (face + pose + left hand + right hand) and sending all 543 landmarks via OSC to Max or other places.

View this post on Instagram

A post shared by Sebastian Tomczak (@littlescale)

Set up a Raspberry Pi 4 with a clean install of Raspberry Pi OS: https://www.raspberrypi.com/software/
Install Mediapipe as per the instructions here: https://github.com/googlesamples/mediapipe/tree/main/examples/pose_landmarker/raspberry_pi
Install Python OSC on the Raspberry Pi: https://pypi.org/project/python-osc/
Add detect_osc_pi.py (https://github.com/little-scale/Tele-o-Module/blob/main/software/dectect_pi_osc.py) to mediapipe/examples/hand_landmarker/rasperry_pi
If using my Tele-o module (https://little-scale.blogspot.com/2023/01/tele-o-wifi-bluetooth-to-cv-gate.html) replace ip in detect_osc_pi.py with the address of the ESP32 in the module.
If using a different OSC endpoint, replace ip in detect_osc_pi.py with the appropriate address.
It may be beneficial to change the port variable in detect_osc_pi.py - the default is 3000
Pressing together or releasing the thumb and index finger sends a 1 or 0 on OSC path /gate/0
The distance between the thumb and index finger sends a float 0 - 1 on OSC path /cv/0
The X and Y co-ordinates of the index finger are sent as floats of 0 -1 on OSC paths /cv/1 and /cv/2
These OSC paths can be changed with the variables near the top of detect_osc_pi.py