Important: The Google+ API for Hangouts is no longer supported. Learn more

Google Hangouts Media Effects API

The Google Hangouts Media Effects API (Effects API) lets you alter audio and video sources in Google+ Hangout apps. For example, you can use the effects API to blur participant faces or apply a custom image enhancement to participant video streams. The Effects API supports multiple simultaneous effects through sub-effects.

Example effects

This page gives you an overview of how to use the Effects API from Google Hangout apps.

Applying Effects

Before applying any effect, you must first initialize an object to contain the meta-effect chain (effect chain):

var metaEffect = gapi.hangout.av.effects.createMetaEffect();
var effectChain = [];

Upon execution, the code initializes the metaEffect variable with the effect chain. Now that you have an effect chain, you add a sub-effect by passing the effect ID and parameters as follows:

var bcsProps = {
  'target_brightness': 150, /* 0 .. 255 */
  'brightness': 0.5, /* -1 .. 1 */
  'constrast': 0.5, /* -1 .. 1 */
  'saturation': 1.0 /* -1 .. 1 */
};

var simpleEffect = metaEffect.createSubEffect('auto_bcs', bcsProps);
effectChain.push(simpleEffect);

Before your code applies the effect, the Hangout displays the video with the default configuration:

Original video example

To apply the effects in the effect chain, call initEffects and then pipelineEffects:

metaEffect.initEffects(effectChain);
metaEffect.pipelineEffects(effectChain);

After your code applies all effects in the effect chain, the video renders the effects in real time.

Auto BCS applied

You can add additional effects to the chain to further enhance the Hangout video and audio. For example, add the vignette and BCS filters:

var vignetteProps = {
  color: { 'r': 0, 'g': 0, 'b': 0},
  focus: { 'x': 0.5, 'y': 0.5 },
  size: 0.7
};

var extraEffect = metaEffect.createSubEffect('vignetting', vignetteProps);
effectChain.push(extraEffect);

metaEffect.initEffects(effectChain);
metaEffect.pipelineEffects(effectChain);

Both effects and their order impact the Hangout's output as it renders the video and audio stream.

Vignette effect demo

Listing effects and effect properties

You can programmatically list the available effects and effect properties to dynamically configure your app's user interface.

Create a callback function that detects the presence of effect descriptions.

var detectedEffectDescription = {};

var onNotify = function(notify) {
  if (notify != null && notify.effect_descriptions != null) {
    detectedEffectDescriptions = notify.effect_descriptions;

    console.log(defectedEffectDescriptions);

    // Effect descriptions have been received, remove this callback.
    g_metaEffect.onNotify.remove(onNotify);
  }
};

After you have setup the callback, add it to the list of notification callbacks on an effect object and call getEffectDescriptions to trigger the callback.

g_metaEffect.onNotify.add(onNotify);
g_metaEffect.getEffectDescriptions();

The API returns an array of objects containing the effect descriptions as follows:

[
  {
    description: 'Description of the effect.',
    id: 'effect_id',
    params: {
      param1name: {
        default: 0, // The default value for param1.
        interval: 0.009, // The suggested interval for the parameter value.
        max: 1, // The maximum supported value.
        min: -1 // The minimum supported value.
      },
      param2name: {
        default: 0, // The default value for param2.
        interval: 0.009, // The suggested interval for the parameter.
        max: 1, // The maximum supported value.
        min: -1 // The minimum supported value.
      }
    },
    release_status: 'stable' // Indicates the channel used for the effect.
  },
  {
    description: 'Description of the next effect.',
    id: 'effect_id2',
    params: {
      param1name: {
        default: 0, // The default value for param1.
        interval: 0.009, // The suggested interval for the parameter value.
        max: 1, // The maximum supported value.
        min: -1 // The minimum supported value.
      },
      param2name: {
        default: 0, // The default value for param2.
        interval: 0.009, // The suggested interval for the parameter.
        max: 1, // The maximum supported value.
        min: -1 // The minimum supported value.
      }
    },
    release_status: 'stable' // Indicates the channel used for the effect.
  }
]

Parameter reference

The effect parameters passed to createSubEffect each have distinct parameter objects that are passed along with the effect.

Video Effects

The following table lists available video effects and their parameters.

Effect IDDescriptionParameters
auto_bcs AutoBCS uses face detection to brighten a face to a target brightness. AutoBCS also does facial contrast and saturation adjustments, but not automatically. Setting all parameters to zero leaves the image unchanged. The brightness parameter indicates the actual brightness, which gradually adjusts to the target brightness.
NameDefaultIntervalMaxMin
brightness - 0.009999 1 -1
contrast 0.400000 0.009999 1 -1
saturation 0.100000 0.009999 1 -1
target_brightness 127 1 255 -
backlight The backlight effect adjusts the amount of light behind the foreground subject.
NameDefaultIntervalMaxMin
backlight 0.5 0.009999 1 -1
bilateral The bilateral effect performs an edge-preserving blur to smooth out non-edge regions of the video. Edges and details remain intact while areas of similar colors are blurred together.
NameDefaultIntervalMaxMin
iterations 1 1 10 1
separable true - - -
similarity_range 25 1 255 1
window_size 11 2 641 3
blur Blurs the specified region of incoming video frame as defined by top, left, width and height, all represented with respect to video frame of width 640. The window_size parameter controls the amount of blur wtih larger values resulting in an image that is more blurry.
NameDefaultIntervalMaxMin
height 1 0.009999 1 -
left - 0.009999 1 -
top - 0.009999 1 -
width 1 0.009999 1 -
window_size 15 2 101 1
blur_face Blurs the face in the video frame. If a face is not present, or has not been detected for the specified timeout duration, the full image is blurred. The affinity parameter determines how quickly the blurred region is created on the detected face. The aspect_ratio parameter changes the shape of the blurred region. The scale parameter changes the size of the blurred region. The scale parameter changes the size of the blurred region.
NameDefaultIntervalMaxMin
affinity 1 0.050000 1 -
aspect_ratio 0.699999 0.100000 10 0.100000
scale 6 0.100000 20 0.100000
timeout 1000 250 60000 -
window_size 33 2 641 3
cartoon Cartoon effect produces a cartoon-like rendering of each frame. The properties 'width' and 'height' parameters define the output resolution of this effect to allow you to increase or reduce the performance requirements for the effect. The 'black' and 'white' parameters determine the white and black balance of the image. The property 'window_size' indicates the amount of bilateral filtering (edge preserving blurring) that occurs. See more details about bilateral filtering by querying the bilateral effect.
NameDefaultIntervalMaxMin
black 0.899999 0.003906 2 -
height 180 1 2048 16
iterations 2 2 10 -
similarity_range 20 1 255 1
white 0.800000 0.003906 2 -
width 320 1 2048 16
window_size 25 2 63 1
color_temp Color temperature effect adjusts the color temperature of the video frame. Scale ranges from -0.5 to 0.5, where 0.0 is the original color temperature. Negative scale values decrease the color temperature resulting in a cooler image. Positive values increase the color temperature resulting in a warmer image.
NameDefaultIntervalMaxMin
scale 0.300000 0.009999 0.5 -0.5
copy Copy effect copies the incoming video frame to the resource manager where the frame can be accessed by other effects that have been configured with the same resource key. This effect does not modify the video frames.
NameDefaultIntervalMaxMin
resource_key preprocessed_frame_ssrc42 - - -
crop Crop effect crops the specified region of incoming video frame. The specified region is defined by top_left, width and height, all represented with respect to video frame.
NameDefaultIntervalMaxMin
height 1 - - -
top_left - - - -
width 1 - - -
crop_face Crops to the face in the video frame and resizes the resulting image to the specified width and height. If a face is not present, or has not been detected for the specified timeout duration, the image is not cropped but it is still resized to the new width and height.
NameDefaultIntervalMaxMin
affinity 0.400000 0.050000 1 -
height 90 90 720 90
scale 4 0.100000 20 0.100000
timeout 1000 250 60000 -
width 160 160 1280 160
duotone The duotone effect is the reproduction of an image by overlapping a contrasting color, which is usually black, over another color, which represents a halftone. This effect can be used to create various artistic effects such as images where highlights and lowlights are tinted.
NameDefaultIntervalMaxMin
color1 - - - -
color2 - - - -
face_data Face data effect captures and returns face tracking data back to JavaScript. This effect does not modify the video image. Tilt, pan and roll angle values are returned in 'degrees' ranging from [-180,180]. Zero tilt and pan angles represent frontal pose. Zero roll angle means upright pose. Example face data message:

{
  image_height:
    <int, e.g. 720, image size from camera>,
  image_width:
    <int, e.g. 1280>,
  confidence:
    <double, e.g. 0.8 is 80% likely a face>,
  left_eye_center: {
    x: <float, center at 0.5>,
    y: <float, center at 0.5>
  },
  right_eye_center: {
    x: <float>,
    y: <float>
  },
  left_eyebrow_left: {
    x: <float>,
    y: <float>
  },
  left_eyebrow_right: {
    x: <float>,
    y: <float>
  },
  right_eyebrow_left: {
    x: <float>,
    y: <float>
  },
  right_eyebrow_right: {
    x: <float>,
    y: <float>
  },
  mouth_left: {
    x: <float>,
    y: <float>
  },
  mouth_right: {
    x: <float>,
    y: <float>
  },
  mouth_center: {
    x: <float>,
    y: <float>
  },
  lip_lower: {
    x: <float>,
    y: <float>
  },
  lip_upper: {
    x: <float>,
    y: <float>
  },
  nose_root: {
    x: <float>,
    y: <float>
  },
  nose_tip: {
    x: <float>,
    y: <float>
  },
  pose_tilt: <float>,
  pose_pan: <float>,
  pose_roll: <float>
}
NameDefaultIntervalMaxMin
get_face_data - - - -
face_overlay Overlays an image that has been previously loaded for creating overlays relative to a participant's face. The image overlay location can be static or relative to a pose feature such as a mouth or head tilt.
NameDefaultIntervalMaxMin
affinity 1 0.050000 1 -
h_align 0.5 0.001562 1 -
location - - - -
offset_x - 0.001562 1 -1
offset_y - 0.002777 1 -1
opacity 1 0.003906 1 -
resource - - - -
rotation - 0.003141 6.283185 -
scale 6 0.100000 20 0.100000
scale_reference_frame resource - - -
timeout 1000 250 60000 -
use_pose_rotation - - - -
v_align 0.5 0.001562 1 -
fisheye Fisheye effect simulates a fisheye lens by distorting the image circularly and magnifying the central region of the image. Scale is used to control the amount of magnification.
NameDefaultIntervalMaxMin
scale 0.5 0.009999 1 -
flip Flip effect flips the video frame horizontally (left/right) and/or vertically.
NameDefaultIntervalMaxMin
horizontal - - - -
vertical - - - -
freeze_frame Freeze frame effect freezes the next video frame on enable and unfreezes on disable. All video effects in the pipeline are still processed even if the freeze frame effect is enabled. Effects with an index prior to freeze frame have their output overwritten with the frozen frame. Usually freeze frame goes at the end of your video pipeline.
grain Grain effect introduces noise into the video frame. Scale ranges from 0.0 to 1.0, where 0.0 is the original image and 1.0 is the maximum amount of noise.
NameDefaultIntervalMaxMin
scale 1 0.009999 1 -
grayscale Converts the video to grayscale.
hand_data Hand data effect captures and returns hand tracking data back to JavaScript. This effect does not modify the video image. Tilt, pan and roll angle values are returned in 'degrees' ranging from [-180,180]. Zero tilt and pan angles represent frontal pose. Zero roll angle means upright pose. Example hand data message:
{
  confidence:
    <double, e.g. 0.8 is 80% likely a hand>,
  hand_center: {
    x: <float, center at 0.5>,
    y: <float, center at 0.5>
  },
  hand_width: {
    x: <float, center at 0.5>,
    y: <float, center at 0.5>
  },
  hand_height: {
    x: <float, center at 0.5>,
    y: <float, center at 0.5>
  },
  pose_tilt: <float>,
  pose_pan: <float>,
  pose_roll: <float>
}
  

NameDefaultIntervalMaxMin
frames_per_detect 5 - - -
get_hand_data - - - -
send_on_change - - - -
hand_overlay Overlays an image that has been previously loaded relative to a participant's hands. The image overlay location can be static or relative to a pose feature such as hand rotation.
NameDefaultIntervalMaxMin
affinity 1 0.050000 1 -
h_align 0.5 0.001562 1 -
location - - - -
offset_x - 0.001562 1 -1
offset_y - 0.002777 1 -1
opacity 1 0.003906 1 -
resource - - - -
rotation - 0.003141 6.283185 -
scale 6 0.100000 20 0.100000
scale_reference_frame resource - - -
timeout 1000 250 60000 -
use_pose_rotation - - - -
v_align 0.5 0.001562 1 -
low_bandwidth_sobel The low bandwidth Sobel effect applies resize, spatial blur, sobel edge emphasis, temporal blur, and bitmask quantization to produce a low bitrate stream.
NameDefaultIntervalMaxMin
bitmask FFC0C0C0 - - -
height 90 - - -
learning_rate 0.850000 0.009999 1 -
negative true - - -
width 160 - - -
window_size 11 2 101 1
meta_effect Meta effect contains other effects. Here is an example meta effect:
{  init: [
    // Brightness, contrast,
    // saturation effect
    {
      effect_handle: 'simple_bcs',
      'id': 'simple_bcs',
      'properties': {
        'brightness': 0.6,
        'contrast': 0.6,
        'saturation': 0.2
      }
    },
    // Blur non-edge features
    {
      effect_handle: 'bilateral',
      'id': 'bilateral',
      'properties': {
        'window_size': 5,
        'similarity_range': 25,
        'iterations': 1,
        'separable': true
      }
    },
    // Add dark rounded edges
    {
      effect_handle: 'vignetting',
      id: 'vignetting',
      properties: { size: 0.9 }
    }
  ],
  pipeline:[
    {effect_handle: 'simple_bcs'},
    {effect_handle: 'bilateral'},
    {effect_handle: 'vignetting'}
  ]
}
A meta effect can be a sub effect too!
NameDefaultIntervalMaxMin
clear - - - -
init - - -
pipeline - - -
remove - - -
negative Inverts the colors of the video stream.
normalizebgr Normalizes blue, green, and red values of the video stream.
overlay Overlays an image that has been previously loaded. The image overlay location can be an arbitrary location of the screen or relative to a pose such as a hand or face feature.
NameDefaultIntervalMaxMin
affinity 1 0.050000 1 -
h_align 0.5 0.001562 1 -
location - - - -
offset_x - 0.001562 1 -1
offset_y - 0.002777 1 -1
opacity 1 0.003906 1 -
resource - - - -
rotation - 0.003141 6.283185 -
scale 6 0.100000 20 0.100000
scale_reference_frame resource - - -
timeout 1000 250 60000 -
use_pose_rotation - - - -
v_align 0.5 0.001562 1 -
overlay_background Use background overlay in conjunction with 'replace_background' and other effects in order to apply effects on the foreground and background independently; e.g. blur foreground, blur background, sobel background, and so on. The current Hangout image replaces the background or foreground if either resource field is missing. When a background and/or foreground resource is present, it is resized to the same dimensions as the current Hangout image. For best quality, resource images should have a 16:9 aspect ratio and size equal to or larger than 1280x720.
NameDefaultIntervalMaxMin
background_resource - - - -
foreground_resource - - - -
quantize Quantize effect reduces the number of colors used in the video frame by quantizing original color space into intervals and compressing color values in one interval to a single quantum value, the median of this interval. The level of color compression can be controlled via interval_size which ranges from 1 to 255. The larger the interval_size, the closer the quantized color space is to the original color space.
NameDefaultIntervalMaxMin
interval_size 64 1 255 1
replace_background Background replacement does real-time background-foreground segmentation and allows users to replace the background with an arbitrary resource image. This can be used in coordination with background overlay by using overlays to alter an image and then replacing the background.
NameDefaultIntervalMaxMin
alpha_threshold -1 1 255 -
background_resource - - - -
update_model true - - -
resize Resize effect resizes each video frame to the specified size in pixels.
NameDefaultIntervalMaxMin
height 180 1 4096 1
preserve_aspect_ratio true - - -
width 320 1 4096 1
sepia Gives the video an effect that makes the stream look antique.
simple_bcs Adjusts brightness, contrast, and saturation. Values of zero leave the image unchanged. As opposed to auto_bcs, a target brightness value is not provided to automatically adjust brightness, contrast, and saturation values.
NameDefaultIntervalMaxMin
brightness 0.300000 0.009999 1 -1
contrast 0.400000 0.009999 1 -1
saturation 0.100000 0.009999 1 -1
sobel This applies a Sobel filter to emphasize edges on the image.
static_overlay Overlays an image that has been previously loaded to a specific location on the screen.
NameDefaultIntervalMaxMin
affinity 1 0.050000 1 -
h_align 0.5 0.001562 1 -
location - - - -
offset_x - 0.001562 1 -1
offset_y - 0.002777 1 -1
opacity 1 0.003906 1 -
resource - - - -
rotation - 0.003141 6.283185 -
scale 6 0.100000 20 0.100000
scale_reference_frame resource - - -
timeout 1000 250 60000 -
use_pose_rotation - - - -
v_align 0.5 0.001562 1 -
surface_blur Surface blur performs an edge-preserving blur and trades some artifacts for speed. Edges and details remain intact while areas of similar colors are blurred together.
NameDefaultIntervalMaxMin
edge_max_threshold 80 1 255 -
edge_min_threshold 10 1 255 -
window_size 33 2 641 3
swap Swap effect swaps the incoming video frame with the resource manager's frame specified by the resource key. This effect does not modify the video frames but is used instead for storing resources used in overlays. If the resource manager did not have a frame for the specified resource key, a black frame will be created.
NameDefaultIntervalMaxMin
resource_key preprocessed_frame_ssrc42 - - -
temporal_blur Temporal blur effect blurs the new input frame with the previous blurred result creating the illusion of motion. The blurred result will converge to the most recent image at the specified learning rate (float between zero and one). The higher the learning rate, the quicker the blurred image will converge resulting in less temporal blur. This effect can optionally use a bilateral filter when applying the temporal blur to increase the learning rate per-pixel.
NameDefaultIntervalMaxMin
bilateral - - - -
learning_rate 0.100000 0.009999 1 -
tint Tint effect removes most of the color from an image, but leaves the specified color, creating a tinted image.
NameDefaultIntervalMaxMin
color - - - -
vignetting Reduces the brightness or saturation of the video frame at the periphery compared to the image focus. It is implemented by blending the video frame with a semi-transparent mask image. Size controls the size of the transparent part in the vignette mask. Color and focal point of the mask image can be specified. The 'alpha_only' parameter applies the mask to only alpha channels if set to true. The 'color' parameter specifies the color of the vignetting. The 'focus' parameter specifies the position of the parameter. The 'size' parameter specifies the size of the vignetting.
NameDefaultIntervalMaxMin
alpha_only - - - -
color - - - -
focus - - - -
size 0.5 0.009999 3.402823 -
warmify This performs a parameterless image transform that makes the image warmer similar to the color temperature effect.
whiteblack White black effect adjusts the white and black balance for specified white and black values. The white and black values range from 0 to 2. White of 1 and black of 0 has no effect. The wider the range between white and black values the less contrast will be present, and conversely, the narrower the range the more contrast; e.g. a black point value of 0 and a white point value of 0.8 increases brightness/contrast.
NameDefaultIntervalMaxMin
black 0.200000 0.009999 2 -
white 0.5 0.009999 2 -

Audio Effects

Effect IDDescriptionParameters
pitch_shift Shifts the pitch of the incoming audio stream as defined by shift_amounts - an array of pitch shifts to apply simultaneously, each entry specifying a number of octaves of pitch shift to apply. Pitch is shifted by converting audio samples into the frequency domain and shifting the spectrum by 2 ^ shift_amounts[index] for each array entry. All shifted samples are then mixed back together and converted back to time domain.
NameDefaultIntervalMaxMin
shift_amounts 0 - - -
play_audio Mix audio effect notifies the processor to mix the audio from an already loaded resource. The resource audio is mixed with the mic’s audio if the direction is set to ‘tx’ (transmit) and the combined signal is transmitted to the remote parties. If the direction is to ‘rx’ (receive) then the resource audio is mixed with the incoming audio from the remote parties and played out to the speaker. Clip specifies a portion of the audio to be played. By default, play audio effect plays the whole audio clip. By setting the start and duration of clip field, the effect will only play a subset of the audio clip. The units of start position and duration can either be "ms" (represents milliseconds) or "frames". The default value is "ms". When the microphone is muted, we can control whether to mute the playing sound via the mute_with_mic field. When the playing sound is muted, we still advance the playout positions to synchronize the possible RX and TX playout.
NameDefaultIntervalMaxMin
clip - - - -
get_info - - - -
loop - - - -
mute_with_mic true - - -
reset - - - -
resource - - - -
volume - 0.009999 3.402823 -

Debugging and prototyping Effects

Some effects are not intended to be used in production hangout apps. These effects are included for debugging and prototyping but will not work on published apps. The following table lists debug effects and their parameters.

Effect IDDescriptionParameters
convolution Convolution effect applies 2D convolution over the specified region of incoming video frames. The specified region is defined by top_left, width and height, all represented with respect to video frame. For each pixel, to calculate the convolved value, pixels within a window (defined by kernel_width and kernel_height) of that pixel in the original image are multiplied by corresponding item in the kernel, added up, divided by the divisor and then added to the bias. In general, running 2D convolution is very slow and time complexity depends on the video frame size and kernel size. Therefore, this effect is only for debugging and prototyping purposes.
NameDefaultIntervalMaxMin
bias - - - -
divisor 1 0.009999 3.402823 0.009999
height 1 - - -
kernel 0 - - -
kernel_height 1 2 2147483647 1
kernel_width 1 2 2147483647 1
top_left - - - -
width 1 - - -

Envoyer des commentaires concernant…

Google+ Hangouts API
Google+ Hangouts API