How to convert ImageFrames and GpuBuffers
GpuBufferToImageFrameCalculator convert back and forth between packets of
ImageFrame refers to image data in
CPU memory in any of a number of bitmap image formats.
GpuBuffer refers to
image data in GPU memory. You can find more detail in the Framework Concepts
section GpuBuffer to ImageFrame
can see an example in:
How to visualize perception results
AnnotationOverlayCalculator allows perception results, such as bounding
boxes, arrows, and ovals, to be superimposed on the video frames aligned with
the recognized objects. The results can be displayed in a diagnostic window when
running on a workstation, or in a texture frame when running on device. You can
see an example use of
How to run calculators in parallel
Within a calculator graph, MediaPipe routinely runs separate calculator nodes in parallel. MediaPipe maintains a pool of threads, and runs each calculator as soon as a thread is available and all of its inputs are ready. Each calculator instance is only run for one set of inputs at a time, so most calculators need only to be thread-compatible and not thread-safe.
In order to enable one calculator to process multiple inputs in parallel, there are two possible approaches:
- Define multiple calculator nodes and dispatch input packets to all nodes.
- Make the calculator thread-safe and configure its
The first approach can be followed using the calculators designed to distribute
packets across other calculators, such as
RoundRobinDemuxCalculator can distribute successive packets across
several identically configured
The second approach allows up to
max_in_flight invocations of the
CalculatorBase::Process method on the same calculator node. The output
CalculatorBase::Process are automatically ordered by timestamp
before they are passed along to downstream calculators.
With either aproach, you must be aware that the calculator running in parallel cannot maintain internal state in the same way as a normal sequential calculator.
Output timestamps when using ImmediateInputStreamHandler
ImmediateInputStreamHandler delivers each packet as soon as it arrives
at an input stream. As a result, it can deliver a packet with a higher timestamp
from one input stream before delivering a packet with a lower timestamp from a
different input stream. If these input timestamps are both used for packets sent
to one output stream, that output stream will complain that the timestamps are
not monotonically increasing. In order to remedy this, the calculator must take
care to output a packet only after processing is complete for its timestamp.
This could be accomplished by waiting until input packets have been received
from all inputstreams for that timestamp, or by ignoring a packet that arrives
with a timestamp that has already been processed.
How to change settings at runtime
There are two main approaches to changing the settings of a calculator graph while the application is running:
- Restart the calculator graph with modified
- Send new calculator options through packets on graph input-streams.
The first approach has the advantage of leveraging
processing tools such as "subgraphs". The second approach has the advantage of
allowing active calculators and packets to remain in-flight while settings
change. Mediapipe contributors are currently investigating alternative
approaches to achieve both of these advantages.
How to process realtime input streams
The mediapipe framework can be used to process data streams either online or offline. For offline processing, packets are pushed into the graph as soon as calculators are ready to process those packets. For online processing, one packet for each frame is pushed into the graph as that frame is recorded.
The MediaPipe framework requires only that successive packets be assigned
monotonically increasing timestamps. By convention, realtime calculators and
graphs use the recording time or the presentation time as the timestamp for each
packet, with each timestamp representing microseconds since
Jan/1/1970:00:00:00. This allows packets from various sources to be processed
in a globally consistent order.
Normally for offline processing, every input packet is processed and processing
continues as long as necessary. For online processing, it is often necessary to
drop input packets in order to keep pace with the arrival of input data frames.
When inputs arrive too frequently, the recommended technique for dropping
packets is to use the MediaPipe calculators designed specifically for this
purpose such as
For online processing, it is also necessary to promptly determine when
processing can proceed. MediaPipe supports this by propagating timestamp bounds
between calculators. Timestamp bounds indicate timestamp intervals that will
contain no input packets, and they allow calculators to begin processing for
those timestamps immediately. Calculators designed for realtime processing
should carefully calculate timestamp bounds in order to begin processing as
promptly as possible. For example, the
MakePairCalculator uses the
SetOffset API to propagate timestamp bounds from input streams to output
Can I run MediaPipe on MS Windows?
Currently MediaPipe portability supports Debian Linux, Ubuntu Linux, MacOS, Android, and iOS. The core of MediaPipe framework is a C++ library conforming to the C++11 standard, so it is relatively easy to port to additional platforms.