The MediaPipe Holistic Landmarker task lets you combine components of the face, hand, and pose landmarkers to detect human body landmarks in images or video. This task outputs holistic landmarks in normalized image coordinates and 3D world coordinates.
These instructions show you how to use the Holistic Landmarker for iOS apps. For more information about the capabilities, models, and configuration options of this task, see the Overview.
Code example
The MediaPipe Tasks example code is a simple implementation of a Holistic Landmarker app for iOS. The example uses the camera on a physical iOS device to detect holistic landmarks in a continuous video stream. The app can also detect landmarks in images from the device gallery.
You can use the app as a starting point for your own iOS app, or refer to it when modifying an existing app. The Holistic Landmarker example code is hosted on GitHub.
Download the code
The following instructions show you how to create a local copy of the example code using the git command line tool.
To download the example code:
- Clone the git repository using the following command:
git clone https://github.com/google-ai-edge/mediapipe-samples
- Optionally, configure your git instance to use sparse checkout, so you have
only the files for the Holistic Landmarker example app:
cd mediapipe-samples git sparse-checkout init --cone git sparse-checkout set examples/holistic_landmarker/ios
After creating a local version of the example code, you can open the project in Xcode and run the app. For instructions, see the Setup Guide for iOS.
Setup
This section describes key steps for setting up your development environment and code projects specifically to use Holistic Landmarker. For general information on setting up your development environment for using MediaPipe tasks, including platform version requirements, see the Setup guide for iOS.
Dependencies
Holistic Landmarker uses the MediaPipeTasksVision library, which must be installed
using CocoaPods. The task is compatible with both Swift and Objective-C apps.
Add the following dependency to your project's Podfile:
target 'MyHolisticLandmarkerApp' do
use_frameworks!
pod 'MediaPipeTasksVision'
end
Then run pod install to update the dependencies.
Model
The MediaPipe Holistic Landmarker task requires a trained model bundle that is compatible with this task. For more information on available trained models for Holistic Landmarker, see the task overview Models section.
Select and download a model, and add it to your Xcode project as a resource, ensuring that "Copy items if needed" is selected.
Use the MPPBaseOptions object to specify the path of the model bundle to use.
Create the task
You can create the Holistic Landmarker task by initializing it with one of its
initializers. The MPPHolisticLandmarker(options:) initializer accepts values
for the configuration options.
The following code demonstrates how to build and configure this task in Swift:
import MediaPipeTasksVision
let modelPath = Bundle.main.path(
forResource: "holistic_landmarker",
ofType: "task"
)!
let options = HolisticLandmarkerOptions()
options.runningMode = .image
options.minFaceDetectionConfidence = 0.5
options.minPoseDetectionConfidence = 0.5
options.minHandLandmarksConfidence = 0.5
options.baseOptions.modelAssetPath = modelPath
let holisticLandmarker = try HolisticLandmarker(options: options)
Configuration options
This task has the following configuration options for iOS applications:
| Option Name | Description | Value Range | Default Value |
|---|---|---|---|
runningMode |
Sets the running mode for the task. There are three
modes: IMAGE: The mode for single image inputs. VIDEO: The mode for decoded frames of a video. LIVE_STREAM: The mode for a livestream of input data, such as from a camera. In this mode, holisticLandmarkerLiveStreamDelegate must be
set to an instance of a class that implements the
HolisticLandmarkerLiveStreamDelegate to receive the results of
performing holistic landmark detection asynchronously.
|
{RunningMode.image, RunningMode.video, RunningMode.liveStream} |
RunningMode.image |
minFaceDetectionConfidence |
The minimum confidence score for the face detection to be considered successful. | Float [0.0, 1.0] |
0.5 |
minFaceSuppressionThreshold |
The minimum non-maximum-suppression threshold for face detection to be considered overlapped. | Float [0.0, 1.0] |
0.3 |
minFacePresenceConfidence |
The minimum confidence score of face presence score in the face landmarks detection. | Float [0.0, 1.0] |
0.5 |
minPoseDetectionConfidence |
The minimum confidence score for the pose detection to be considered successful. | Float [0.0, 1.0] |
0.5 |
minPoseSuppressionThreshold |
The minimum non-maximum-suppression threshold for pose detection to be considered overlapped. | Float [0.0, 1.0] |
0.3 |
minPosePresenceConfidence |
The minimum confidence score of pose presence score in the pose landmarks detection. | Float [0.0, 1.0] |
0.5 |
minHandLandmarksConfidence |
The minimum confidence score of hand presence score in the hand landmarks detection. | Float [0.0, 1.0] |
0.5 |
outputFaceBlendshapes |
Whether to output face blendshapes classification. Face blendshapes are used for rendering the 3D face model. | Boolean |
false |
outputPoseSegmentationMasks |
Whether to output segmentation masks for the human pose. | Boolean |
false |
Prepare data
You need to convert your input image or frame into an MPPImage object before
passing it to the Holistic Landmarker. MPPImage supports various iOS image formats
and rotation properties.
Run the task
To run the task, use the detect method (for single images) or
detect(videoFrame:timestampInMilliseconds:) (for video):
// Run holistic landmarks detection on the input image
let result = try holisticLandmarker.detect(image: mppImage)