2022/11/11 - Amazon Rekognition - 2 updated api methods
Changes Adding support for ImageProperties feature to detect dominant colors and image brightness, sharpness, and contrast, inclusion and exclusion filters for labels and label categories, new fields to the API response, "aliases" and "categories"
{'Features': ['GENERAL_LABELS | IMAGE_PROPERTIES'],
'Settings': {'GeneralLabels': {'LabelCategoryExclusionFilters': ['string'],
'LabelCategoryInclusionFilters': ['string'],
'LabelExclusionFilters': ['string'],
'LabelInclusionFilters': ['string']},
'ImageProperties': {'MaxDominantColors': 'integer'}}}
Response {'ImageProperties': {'Background': {'DominantColors': [{'Blue': 'integer',
'CSSColor': 'string',
'Green': 'integer',
'HexCode': 'string',
'PixelPercent': 'float',
'Red': 'integer',
'SimplifiedColor': 'string'}],
'Quality': {'Brightness': 'float',
'Contrast': 'float',
'Sharpness': 'float'}},
'DominantColors': [{'Blue': 'integer',
'CSSColor': 'string',
'Green': 'integer',
'HexCode': 'string',
'PixelPercent': 'float',
'Red': 'integer',
'SimplifiedColor': 'string'}],
'Foreground': {'DominantColors': [{'Blue': 'integer',
'CSSColor': 'string',
'Green': 'integer',
'HexCode': 'string',
'PixelPercent': 'float',
'Red': 'integer',
'SimplifiedColor': 'string'}],
'Quality': {'Brightness': 'float',
'Contrast': 'float',
'Sharpness': 'float'}},
'Quality': {'Brightness': 'float',
'Contrast': 'float',
'Sharpness': 'float'}},
'Labels': {'Aliases': [{'Name': 'string'}],
'Categories': [{'Name': 'string'}],
'Instances': {'DominantColors': [{'Blue': 'integer',
'CSSColor': 'string',
'Green': 'integer',
'HexCode': 'string',
'PixelPercent': 'float',
'Red': 'integer',
'SimplifiedColor': 'string'}]}}}
Detects instances of real-world entities within an image (JPEG or PNG) provided as input. This includes objects like flower, tree, and table; events like wedding, graduation, and birthday party; and concepts like landscape, evening, and nature.
For an example, see Analyzing images stored in an Amazon S3 bucket in the Amazon Rekognition Developer Guide.
You pass the input image as base64-encoded image bytes or as a reference to an image in an Amazon S3 bucket. If you use the AWS CLI to call Amazon Rekognition operations, passing image bytes is not supported. The image must be either a PNG or JPEG formatted file.
Optional Parameters
You can specify one or both of the GENERAL_LABELS and IMAGE_PROPERTIES feature types when calling the DetectLabels API. Including GENERAL_LABELS will ensure the response includes the labels detected in the input image, while including IMAGE_PROPERTIES will ensure the response includes information about the image quality and color.
When using GENERAL_LABELS and/or IMAGE_PROPERTIES you can provide filtering criteria to the Settings parameter. You can filter with sets of individual labels or with label categories. You can specify inclusive filters, exclusive filters, or a combination of inclusive and exclusive filters. For more information on filtering see Detecting Labels in an Image .
You can specify MinConfidence to control the confidence threshold for the labels returned. The default is 55%. You can also add the MaxLabels parameter to limit the number of labels returned. The default and upper limit is 1000 labels.
Response Elements
For each object, scene, and concept the API returns one or more labels. The API returns the following types of information regarding labels:
Name - The name of the detected label.
Confidence - The level of confidence in the label assigned to a detected object.
Parents - The ancestor labels for a detected label. DetectLabels returns a hierarchical taxonomy of detected labels. For example, a detected car might be assigned the label car. The label car has two parent labels: Vehicle (its parent) and Transportation (its grandparent). The response includes the all ancestors for a label, where every ancestor is a unique label. In the previous example, Car, Vehicle, and Transportation are returned as unique labels in the response.
Aliases - Possible Aliases for the label.
Categories - The label categories that the detected label belongs to. A given label can belong to more than one category.
BoundingBox — Bounding boxes are described for all instances of detected common object labels, returned in an array of Instance objects. An Instance object contains a BoundingBox object, describing the location of the label on the input image. It also includes the confidence for the accuracy of the detected bounding box.
The API returns the following information regarding the image, as part of the ImageProperties structure:
Quality - Information about the Sharpness, Brightness, and Contrast of the input image, scored between 0 to 100. Image quality is returned for the entire image, as well as the background and the foreground.
Dominant Color - An array of the dominant colors in the image.
Foreground - Information about the Sharpness and Brightness of the input image’s foreground.
Background - Information about the Sharpness and Brightness of the input image’s background.
The list of returned labels will include at least one label for every detected object, along with information about that label. In the following example, suppose the input image has a lighthouse, the sea, and a rock. The response includes all three labels, one for each object, as well as the confidence in the label:
{Name: lighthouse, Confidence: 98.4629}
{Name: rock,Confidence: 79.2097}
{Name: sea,Confidence: 75.061}
The list of labels can include multiple labels for the same object. For example, if the input image shows a flower (for example, a tulip), the operation might return the following three labels.
{Name: flower,Confidence: 99.0562}
{Name: plant,Confidence: 99.0562}
{Name: tulip,Confidence: 99.0562}
In this example, the detection algorithm more precisely identifies the flower as a tulip.
Note
If the object detected is a person, the operation doesn't provide the same facial details that the DetectFaces operation provides.
This is a stateless API operation. That is, the operation does not persist any data.
This operation requires permissions to perform the rekognition:DetectLabels action.
See also: AWS API Documentation
Request Syntax
client.detect_labels(
Image={
'Bytes': b'bytes',
'S3Object': {
'Bucket': 'string',
'Name': 'string',
'Version': 'string'
}
},
MaxLabels=123,
MinConfidence=...,
Features=[
'GENERAL_LABELS'|'IMAGE_PROPERTIES',
],
Settings={
'GeneralLabels': {
'LabelInclusionFilters': [
'string',
],
'LabelExclusionFilters': [
'string',
],
'LabelCategoryInclusionFilters': [
'string',
],
'LabelCategoryExclusionFilters': [
'string',
]
},
'ImageProperties': {
'MaxDominantColors': 123
}
}
)
dict
[REQUIRED]
The input image as base64-encoded bytes or an S3 object. If you use the AWS CLI to call Amazon Rekognition operations, passing image bytes is not supported. Images stored in an S3 Bucket do not need to be base64-encoded.
If you are using an AWS SDK to call Amazon Rekognition, you might not need to base64-encode image bytes passed using the Bytes field. For more information, see Images in the Amazon Rekognition developer guide.
Bytes (bytes) --
Blob of image bytes up to 5 MBs.
S3Object (dict) --
Identifies an S3 object as the image source.
Bucket (string) --
Name of the S3 bucket.
Name (string) --
S3 object key name.
Version (string) --
If the bucket is versioning enabled, you can specify the object version.
integer
Maximum number of labels you want the service to return in the response. The service returns the specified number of highest confidence labels.
float
Specifies the minimum confidence level for the labels to return. Amazon Rekognition doesn't return any labels with confidence lower than this specified value.
If MinConfidence is not specified, the operation returns labels with a confidence values greater than or equal to 55 percent.
list
A list of the types of analysis to perform. Specifying GENERAL_LABELS uses the label detection feature, while specifying IMAGE_PROPERTIES returns information regarding image color and quality. If no option is specified GENERAL_LABELS is used by default.
(string) --
dict
A list of the filters to be applied to returned detected labels and image properties. Specified filters can be inclusive, exclusive, or a combination of both. Filters can be used for individual labels or label categories. The exact label names or label categories must be supplied. For a full list of labels and label categories, see LINK HERE.
GeneralLabels (dict) --
Contains the specified filters for GENERAL_LABELS.
LabelInclusionFilters (list) --
The labels that should be included in the return from DetectLabels.
(string) --
LabelExclusionFilters (list) --
The labels that should be excluded from the return from DetectLabels.
(string) --
LabelCategoryInclusionFilters (list) --
The label categories that should be included in the return from DetectLabels.
(string) --
LabelCategoryExclusionFilters (list) --
The label categories that should be excluded from the return from DetectLabels.
(string) --
ImageProperties (dict) --
Contains the chosen number of maximum dominant colors in an image.
MaxDominantColors (integer) --
The maximum number of dominant colors to return when detecting labels in an image. The default value is 10.
dict
Response Syntax
{
'Labels': [
{
'Name': 'string',
'Confidence': ...,
'Instances': [
{
'BoundingBox': {
'Width': ...,
'Height': ...,
'Left': ...,
'Top': ...
},
'Confidence': ...,
'DominantColors': [
{
'Red': 123,
'Blue': 123,
'Green': 123,
'HexCode': 'string',
'CSSColor': 'string',
'SimplifiedColor': 'string',
'PixelPercent': ...
},
]
},
],
'Parents': [
{
'Name': 'string'
},
],
'Aliases': [
{
'Name': 'string'
},
],
'Categories': [
{
'Name': 'string'
},
]
},
],
'OrientationCorrection': 'ROTATE_0'|'ROTATE_90'|'ROTATE_180'|'ROTATE_270',
'LabelModelVersion': 'string',
'ImageProperties': {
'Quality': {
'Brightness': ...,
'Sharpness': ...,
'Contrast': ...
},
'DominantColors': [
{
'Red': 123,
'Blue': 123,
'Green': 123,
'HexCode': 'string',
'CSSColor': 'string',
'SimplifiedColor': 'string',
'PixelPercent': ...
},
],
'Foreground': {
'Quality': {
'Brightness': ...,
'Sharpness': ...,
'Contrast': ...
},
'DominantColors': [
{
'Red': 123,
'Blue': 123,
'Green': 123,
'HexCode': 'string',
'CSSColor': 'string',
'SimplifiedColor': 'string',
'PixelPercent': ...
},
]
},
'Background': {
'Quality': {
'Brightness': ...,
'Sharpness': ...,
'Contrast': ...
},
'DominantColors': [
{
'Red': 123,
'Blue': 123,
'Green': 123,
'HexCode': 'string',
'CSSColor': 'string',
'SimplifiedColor': 'string',
'PixelPercent': ...
},
]
}
}
}
Response Structure
(dict) --
Labels (list) --
An array of labels for the real-world objects detected.
(dict) --
Structure containing details about the detected label, including the name, detected instances, parent labels, and level of confidence.
Name (string) --
The name (label) of the object or scene.
Confidence (float) --
Level of confidence.
Instances (list) --
If Label represents an object, Instances contains the bounding boxes for each instance of the detected object. Bounding boxes are returned for common object labels such as people, cars, furniture, apparel or pets.
(dict) --
An instance of a label returned by Amazon Rekognition Image ( DetectLabels ) or by Amazon Rekognition Video ( GetLabelDetection ).
BoundingBox (dict) --
The position of the label instance on the image.
Width (float) --
Width of the bounding box as a ratio of the overall image width.
Height (float) --
Height of the bounding box as a ratio of the overall image height.
Left (float) --
Left coordinate of the bounding box as a ratio of overall image width.
Top (float) --
Top coordinate of the bounding box as a ratio of overall image height.
Confidence (float) --
The confidence that Amazon Rekognition has in the accuracy of the bounding box.
DominantColors (list) --
The dominant colors found in an individual instance of a label.
(dict) --
A description of the dominant colors in an image.
Red (integer) --
The Red RGB value for a dominant color.
Blue (integer) --
The Blue RGB value for a dominant color.
Green (integer) --
The Green RGB value for a dominant color.
HexCode (string) --
The Hex code equivalent of the RGB values for a dominant color.
CSSColor (string) --
The CSS color name of a dominant color.
SimplifiedColor (string) --
One of 12 simplified color names applied to a dominant color.
PixelPercent (float) --
The percentage of image pixels that have a given dominant color.
Parents (list) --
The parent labels for a label. The response includes all ancestor labels.
(dict) --
A parent label for a label. A label can have 0, 1, or more parents.
Name (string) --
The name of the parent label.
Aliases (list) --
A list of potential aliases for a given label.
(dict) --
A potential alias of for a given label.
Name (string) --
The name of an alias for a given label.
Categories (list) --
A list of the categories associated with a given label.
(dict) --
The category that applies to a given label.
Name (string) --
The name of a category that applies to a given label.
OrientationCorrection (string) --
The value of OrientationCorrection is always null.
If the input image is in .jpeg format, it might contain exchangeable image file format (Exif) metadata that includes the image's orientation. Amazon Rekognition uses this orientation information to perform image correction. The bounding box coordinates are translated to represent object locations after the orientation information in the Exif metadata is used to correct the image orientation. Images in .png format don't contain Exif metadata.
Amazon Rekognition doesn’t perform image correction for images in .png format and .jpeg images without orientation information in the image Exif metadata. The bounding box coordinates aren't translated and represent the object locations before the image is rotated.
LabelModelVersion (string) --
Version number of the label detection model that was used to detect labels.
ImageProperties (dict) --
Information about the properties of the input image, such as brightness, sharpness, contrast, and dominant colors.
Quality (dict) --
Information about the quality of the image foreground as defined by brightness, sharpness, and contrast. The higher the value the greater the brightness, sharpness, and contrast respectively.
Brightness (float) --
The brightness of an image provided for label detection.
Sharpness (float) --
The sharpness of an image provided for label detection.
Contrast (float) --
The contrast of an image provided for label detection.
DominantColors (list) --
Information about the dominant colors found in an image, described with RGB values, CSS color name, simplified color name, and PixelPercentage (the percentage of image pixels that have a particular color).
(dict) --
A description of the dominant colors in an image.
Red (integer) --
The Red RGB value for a dominant color.
Blue (integer) --
The Blue RGB value for a dominant color.
Green (integer) --
The Green RGB value for a dominant color.
HexCode (string) --
The Hex code equivalent of the RGB values for a dominant color.
CSSColor (string) --
The CSS color name of a dominant color.
SimplifiedColor (string) --
One of 12 simplified color names applied to a dominant color.
PixelPercent (float) --
The percentage of image pixels that have a given dominant color.
Foreground (dict) --
Information about the properties of an image’s foreground, including the foreground’s quality and dominant colors, including the quality and dominant colors of the image.
Quality (dict) --
The quality of the image foreground as defined by brightness and sharpness.
Brightness (float) --
The brightness of an image provided for label detection.
Sharpness (float) --
The sharpness of an image provided for label detection.
Contrast (float) --
The contrast of an image provided for label detection.
DominantColors (list) --
The dominant colors found in the foreground of an image, defined with RGB values, CSS color name, simplified color name, and PixelPercentage (the percentage of image pixels that have a particular color).
(dict) --
A description of the dominant colors in an image.
Red (integer) --
The Red RGB value for a dominant color.
Blue (integer) --
The Blue RGB value for a dominant color.
Green (integer) --
The Green RGB value for a dominant color.
HexCode (string) --
The Hex code equivalent of the RGB values for a dominant color.
CSSColor (string) --
The CSS color name of a dominant color.
SimplifiedColor (string) --
One of 12 simplified color names applied to a dominant color.
PixelPercent (float) --
The percentage of image pixels that have a given dominant color.
Background (dict) --
Information about the properties of an image’s background, including the background’s quality and dominant colors, including the quality and dominant colors of the image.
Quality (dict) --
The quality of the image background as defined by brightness and sharpness.
Brightness (float) --
The brightness of an image provided for label detection.
Sharpness (float) --
The sharpness of an image provided for label detection.
Contrast (float) --
The contrast of an image provided for label detection.
DominantColors (list) --
The dominant colors found in the background of an image, defined with RGB values, CSS color name, simplified color name, and PixelPercentage (the percentage of image pixels that have a particular color).
(dict) --
A description of the dominant colors in an image.
Red (integer) --
The Red RGB value for a dominant color.
Blue (integer) --
The Blue RGB value for a dominant color.
Green (integer) --
The Green RGB value for a dominant color.
HexCode (string) --
The Hex code equivalent of the RGB values for a dominant color.
CSSColor (string) --
The CSS color name of a dominant color.
SimplifiedColor (string) --
One of 12 simplified color names applied to a dominant color.
PixelPercent (float) --
The percentage of image pixels that have a given dominant color.
{'Labels': {'Label': {'Aliases': [{'Name': 'string'}],
'Categories': [{'Name': 'string'}],
'Instances': {'DominantColors': [{'Blue': 'integer',
'CSSColor': 'string',
'Green': 'integer',
'HexCode': 'string',
'PixelPercent': 'float',
'Red': 'integer',
'SimplifiedColor': 'string'}]}}}}
Gets the label detection results of a Amazon Rekognition Video analysis started by StartLabelDetection .
The label detection operation is started by a call to StartLabelDetection which returns a job identifier (JobId ). When the label detection operation finishes, Amazon Rekognition publishes a completion status to the Amazon Simple Notification Service topic registered in the initial call to StartlabelDetection . To get the results of the label detection operation, first check that the status value published to the Amazon SNS topic is SUCCEEDED . If so, call GetLabelDetection and pass the job identifier (JobId ) from the initial call to StartLabelDetection .
GetLabelDetection returns an array of detected labels (Labels ) sorted by the time the labels were detected. You can also sort by the label name by specifying NAME for the SortBy input parameter.
The labels returned include the label name, the percentage confidence in the accuracy of the detected label, and the time the label was detected in the video.
The returned labels also include bounding box information for common objects, a hierarchical taxonomy of detected labels, and the version of the label model used for detection.
Use MaxResults parameter to limit the number of labels returned. If there are more results than specified in MaxResults , the value of NextToken in the operation response contains a pagination token for getting the next set of results. To get the next page of results, call GetlabelDetection and populate the NextToken request parameter with the token value returned from the previous call to GetLabelDetection .
See also: AWS API Documentation
Request Syntax
client.get_label_detection(
JobId='string',
MaxResults=123,
NextToken='string',
SortBy='NAME'|'TIMESTAMP'
)
string
[REQUIRED]
Job identifier for the label detection operation for which you want results returned. You get the job identifer from an initial call to StartlabelDetection .
integer
Maximum number of results to return per paginated call. The largest value you can specify is 1000. If you specify a value greater than 1000, a maximum of 1000 results is returned. The default value is 1000.
string
If the previous response was incomplete (because there are more labels to retrieve), Amazon Rekognition Video returns a pagination token in the response. You can use this pagination token to retrieve the next set of labels.
string
Sort to use for elements in the Labels array. Use TIMESTAMP to sort array elements by the time labels are detected. Use NAME to alphabetically group elements for a label together. Within each label group, the array element are sorted by detection confidence. The default sort is by TIMESTAMP .
dict
Response Syntax
{
'JobStatus': 'IN_PROGRESS'|'SUCCEEDED'|'FAILED',
'StatusMessage': 'string',
'VideoMetadata': {
'Codec': 'string',
'DurationMillis': 123,
'Format': 'string',
'FrameRate': ...,
'FrameHeight': 123,
'FrameWidth': 123,
'ColorRange': 'FULL'|'LIMITED'
},
'NextToken': 'string',
'Labels': [
{
'Timestamp': 123,
'Label': {
'Name': 'string',
'Confidence': ...,
'Instances': [
{
'BoundingBox': {
'Width': ...,
'Height': ...,
'Left': ...,
'Top': ...
},
'Confidence': ...,
'DominantColors': [
{
'Red': 123,
'Blue': 123,
'Green': 123,
'HexCode': 'string',
'CSSColor': 'string',
'SimplifiedColor': 'string',
'PixelPercent': ...
},
]
},
],
'Parents': [
{
'Name': 'string'
},
],
'Aliases': [
{
'Name': 'string'
},
],
'Categories': [
{
'Name': 'string'
},
]
}
},
],
'LabelModelVersion': 'string'
}
Response Structure
(dict) --
JobStatus (string) --
The current status of the label detection job.
StatusMessage (string) --
If the job fails, StatusMessage provides a descriptive error message.
VideoMetadata (dict) --
Information about a video that Amazon Rekognition Video analyzed. Videometadata is returned in every page of paginated responses from a Amazon Rekognition video operation.
Codec (string) --
Type of compression used in the analyzed video.
DurationMillis (integer) --
Length of the video in milliseconds.
Format (string) --
Format of the analyzed video. Possible values are MP4, MOV and AVI.
FrameRate (float) --
Number of frames per second in the video.
FrameHeight (integer) --
Vertical pixel dimension of the video.
FrameWidth (integer) --
Horizontal pixel dimension of the video.
ColorRange (string) --
A description of the range of luminance values in a video, either LIMITED (16 to 235) or FULL (0 to 255).
NextToken (string) --
If the response is truncated, Amazon Rekognition Video returns this token that you can use in the subsequent request to retrieve the next set of labels.
Labels (list) --
An array of labels detected in the video. Each element contains the detected label and the time, in milliseconds from the start of the video, that the label was detected.
(dict) --
Information about a label detected in a video analysis request and the time the label was detected in the video.
Timestamp (integer) --
Time, in milliseconds from the start of the video, that the label was detected. Note that Timestamp is not guaranteed to be accurate to the individual frame where the label first appears.
Label (dict) --
Details about the detected label.
Name (string) --
The name (label) of the object or scene.
Confidence (float) --
Level of confidence.
Instances (list) --
If Label represents an object, Instances contains the bounding boxes for each instance of the detected object. Bounding boxes are returned for common object labels such as people, cars, furniture, apparel or pets.
(dict) --
An instance of a label returned by Amazon Rekognition Image ( DetectLabels ) or by Amazon Rekognition Video ( GetLabelDetection ).
BoundingBox (dict) --
The position of the label instance on the image.
Width (float) --
Width of the bounding box as a ratio of the overall image width.
Height (float) --
Height of the bounding box as a ratio of the overall image height.
Left (float) --
Left coordinate of the bounding box as a ratio of overall image width.
Top (float) --
Top coordinate of the bounding box as a ratio of overall image height.
Confidence (float) --
The confidence that Amazon Rekognition has in the accuracy of the bounding box.
DominantColors (list) --
The dominant colors found in an individual instance of a label.
(dict) --
A description of the dominant colors in an image.
Red (integer) --
The Red RGB value for a dominant color.
Blue (integer) --
The Blue RGB value for a dominant color.
Green (integer) --
The Green RGB value for a dominant color.
HexCode (string) --
The Hex code equivalent of the RGB values for a dominant color.
CSSColor (string) --
The CSS color name of a dominant color.
SimplifiedColor (string) --
One of 12 simplified color names applied to a dominant color.
PixelPercent (float) --
The percentage of image pixels that have a given dominant color.
Parents (list) --
The parent labels for a label. The response includes all ancestor labels.
(dict) --
A parent label for a label. A label can have 0, 1, or more parents.
Name (string) --
The name of the parent label.
Aliases (list) --
A list of potential aliases for a given label.
(dict) --
A potential alias of for a given label.
Name (string) --
The name of an alias for a given label.
Categories (list) --
A list of the categories associated with a given label.
(dict) --
The category that applies to a given label.
Name (string) --
The name of a category that applies to a given label.
LabelModelVersion (string) --
Version number of the label detection model that was used to detect labels.