Introduction

This article will show you how to grab a camera preview frame and apply some effects to it.  Also, it describes how to apply the very same effect on the eventually taken picture using only the Tizen Native Camera and the Image Util C API.

In the presented example we want to add a sepia effect to a camera preview and the final picture. Sepia is an effect which makes the picture look like an old-fashioned monochromatic photograph, colorized with brown tones.

The point of our concept is to let the user see already in a camera preview how the final picture would look like.

Prerequisites

Before reading this article it is recommended that you familiarize yourself with the basic Native Camera API usage under the following link:

https://developer.tizen.org/development/tutorials/native-application/multimedia/camera

Then you should be able to prepare your own basic camera application which provides live preview and takes a picture. You can also download and familiarize with one of the SDK sample applications with the camera functionality.

The starting point is to have a simple, working camera app.

Using the instructions below you will be able to modify your application to provide a sepia effect on the preview and the final picture.

Steps to do

Considering we have already a working simple camera app (be it one of the sample Tizen camera apps or an app made by yourself), here is the brief summary of steps we need to do to make things work:

Step 1: Prepare a camera:

• configure it to capture pictures in a proper format (the same one as is used in the camera preview, to be able to apply the same effect to the preview and to the final picture)
• set a callback to get access to the camera preview frames

Step 2: Implement the function to modify preview frames:

• extract properly data from the camera preview data structure
• implement effect functions to modify the preview data

Step 3: Implement a function to modify the final picture:

• extract properly data from the camera image data structure
• re-use effect functions to modify the camera image data

Step 4: Finalize the work:

• encode the modified picture to JPEG
• insert the image to the media database in order to quickly reflect it in the Gallery and other apps using the media DB.

The expected result of the modifications will be as on the picture to the right:

Figure 1 – Picture taken using a camera app without a sepia effect (left) and picture taken using an application with the effect (right)

Code arrangement

To separate image modification code to be easily reused in the sample code, we will place all image modification functions in a pair of header and source files. Only three functions will have to be called “outside” in your camera app code: prepare_camera(), modify_final_image(), and finalize_taking_picture(). They are listed in the img-mods.h header which you have to include into your app.

Step 1: Prepare a camera

Sepia effect is a toning effect which influences the picture as a whole, making it actually a monochromatic picture, colorized with different shades of one color - brown. That's why the easiest way to apply it is to do it in the luminance-chrominance color space.

In Tizen the video stream is encoded in NV12, which is an encoding consisting of luminance data and two chrominance components. It’s a good format to directly apply the sepia effect, because we don't need any extra conversion to separate brightness and color information and modify them.

The camera preview frames are given to us in NV12 by default, so we don't need to set up the preview data encoding before starting the preview.

However to make the final picture encoded in NV12, we have to set the camera capture to NV12:

camera_set_capture_format(camera, CAMERA_PIXEL_FORMAT_NV12);

Also we have to register a callback to get informed about receiving the camera preview frames:

camera_set_preview_cb(camera, _camera_preview_callback, NULL);

As both of these functions can be called in the same moment, we can pack them in a convenient function:

// Function preparing camera:
// First, changing capture format to NV12 to easily modify the image
// using the same filters as the preview frame.
// Second, setting custom preview callback to grab and modify the preview frames

//…
#include <camera.h>
//…

int prepare_camera(camera_h camera)
{
    if ((camera_set_capture_format(camera, CAMERA_PIXEL_FORMAT_NV12) == CAMERA_ERROR_NONE)
		&& (camera_set_preview_cb(camera, _camera_preview_callback, NULL) == CAMERA_ERROR_NONE))
	{
		dlog_print(DLOG_INFO, LOG_TAG, "The camera has been prepared for an image modification.");
		return 0;
	}
	else
	{
		dlog_print(DLOG_ERROR, LOG_TAG, "Error occurred when preparing camera!");
		return -1;
	}
}

…which we call in the camera app code just after initiating the camera using camera_create() and before starting preview using camera_start_preview() APIs:

#include <camera.h> // Camera Native API header
#include "img-mods.h" // Our custom image modifications header

//=========================================
// CAMERA INITIALIZATION CODE
//=========================================

int ret = camera_create(CAMERA_DEVICE_CAMERA0, &camera);

//...
    	// Setting image to be captured in NV12
		// and setting custom preview callback to grab and modify the preview frames
		if (prepare_camera(view->camera) == 0)
		{
			dlog_print(DLOG_INFO, LOG_TAG, "Camera prepared!");
		}
		else
		{
			dlog_print(DLOG_ERROR, LOG_TAG, "Cannot prepare camera!");
		}		
//...

ret = camera_start_preview(camera);
//=========================================

Step 2: Implement a function to modify the preview frames

In step 1 we registered a callback to be executed on each camera preview frame we have received. Now we have to define that callback.

Before we start, we should examine the format of the preview frame.

In case of Samsung Z1 and Z3 devices, the preview frame is encoded in 2 -plane NV12. What does it mean?

NV12 is a specific case of Y’CbCr color space image encoding. You can read about Y’CbCr more on Wikipedia https://en.wikipedia.org/wiki/YCbCr. Y’CbCr is commonly called YUV (although it is not exactly the same format as original YUV as it could be found in analog television). Each pixel can be defined by the luma value (Y’), connected with luminance, and chroma component values: Cb (commonly called U) and Cr (commonly called V), defining color of a pixel. NV12 uses so called chroma subsampling. As you can see on the figure 2, if the image is encoded in NV12, each pixel has its own unique luma value, but four neighbouring pixels share one color value. The color is defined by a pair of chroma components. It is reasonable to subsample the color data and reduce its resolution in favour of luminance data, because the human eye is not as sensitive to color as it is to light.

Figure 2 - Luma and chroma values as reflecting sampled areas of the image

2-plane NV12 means the frame data is kept in 2 planes –one is Y plane storing information on luminance data, and the other – UV plane is storing information on chrominance data.

The camera_preview_cb type callback provides us with the pointer to the camera_preview_data_s structure which defines camera preview frame. 

Figure 3 - Representation of NV12 data in the camera preview frame

As the sepia effect converts the picture to the monochromatic color scale built on a brown color, it  can be easily done on luma and chroma planes separately. We can put our operations in two separate functions – one multiplying the luma value to provide old picture style overexposure effect, and the second one – bringing the brown tone to the picture, which in our case means replacing the chroma components’ values to constant ones.

// Callback function to be executed on each preview frame captured
// for modifying the NV12 preview frame
void _camera_preview_callback(camera_preview_data_s *frame, void *user_data)
{
    if (frame->format == CAMERA_PIXEL_FORMAT_NV12 && frame->num_of_planes == 2)
	{
		// We take the luminance (Y) component
		// and modify it to look like the old overexposed image:
		_luma_mod_blow_highlights(frame->data.double_plane.y, frame->data.double_plane.y_size);
		// We take the chrominance component and modify it to look like sepia:
		_chroma_mod_sepia(frame->data.double_plane.uv, frame->data.double_plane.uv_size);
	}
	else
	{
		dlog_print(DLOG_ERROR, LOG_TAG, "This preview frame format is not supported!");
		//we do nothing, the preview is left intact and displayed without modifications
	}
}

Usually, old pictures, especially amateur ones, are either over- or underexposed. In this example we will use a simple trick to blow out highlights in order to make an impression of an old photograph.

We can easily do it by multiplying the luma value by a constant, in the following example, by 1.5.

// Function for modifying luminance data of the image
// to make it look like the old picture with overexposed highlights.
void _luma_mod_blow_highlights(unsigned char* data, uint64_t size)
{
    uint64_t i = 0;

	for (i=0; i < size; i++)
	{
		int Yval = (int)(data[i]);
		Yval = (Yval*15)/10; // 150% of the original luma size
		data[i] = _clip_n_convert(Yval);
	}
};

To avoid exceeding the value range we create a simple function below. In our example the section clipping negative values will not be used (we only increase the luminance).

// Universal function to clip int values to the range [0..255]
// and convert it to unsigned char as stored in image buffers.
unsigned char _clip_n_convert(int val)
{
    if (val > 255)
	{
		return 255;
	}
	if (val < 0)
	{
		return 0;
	}
	return (unsigned char)val;
}

Modifying chroma comes down to simple setting the Cb and Cr composites to constant values, which define brown color (we chose Cb = 114, Cr = 144). As you have already seen on the figure presented before, odd bytes of data.double_plane.uv mean Cr values, while even bytes mean Cb values.

// Function for modifying chrominance data of the image
// to give it the sepia tone.
void _chroma_mod_sepia(unsigned char* data, uint64_t size)
{
    uint64_t i = 0;

	for (i=0; i<size; i++)
	{
		int Cval = (int)(data[i]);

		if (i%2 ==0) //Even byte reflects Cb chroma value
		{
			Cval = 114; //Setting the Cb value to look like sepia
		}
		else //Odd byte reflects Cr chroma value
		{
			Cval = 144; //Setting the Cr value to look like sepia
		}
		data[i] = (unsigned char)Cval;
	}
}

Step 3: Implement a function to modify the final picture

Now we can proceed to the modification of the picture finally taken by user.

As you probably know, capturing a picture is triggered by the camera_start_capture() API function.

//=========================================
// CAPTURING PICTURE WITH CALLBACK
//=========================================

int ret = camera_start_capture(camera, _capturing_cb, _capture_completed_cb, NULL);

To modify the picture before saving we should add some lines to the _capturing_cb callback.

Generally, we packed all that should be done in this step in two convenient functions: modify_final_image() and finalize_taking_picture(). The former, performs the operations on the image itself. The latter, saves the image as a file in a requested path of a storage memory and updates the media database and it will be explained in Step 4.

//=========================================
// CALLBACK ON PICTURE CAPTURED
//=========================================

static void _capturing_cb(camera_image_data_s *image, camera_image_data_s *postview, camera_image_data_s *thumbnail, void *user_data)
{
//...

//We modify the picture taken
    if (modify_final_image(image) == 0)
	{
		dlog_print(DLOG_INFO, LOG_TAG, "Captured photo modified!");
	}
	else
	{
		dlog_print(DLOG_ERROR, LOG_TAG, "Modifying photo failed!");
	}

//We convert it to JPEG, save and update the media db.
//This must be done when we already know the path of the file to be saved.
	if (finalize_taking_picture(image, filename) == 0)
	{
		dlog_print(DLOG_INFO, LOG_TAG, "Capturing finalized!");
	}
	else
	{
		dlog_print(DLOG_ERROR, LOG_TAG, "Capturing finalizing failed!");
	}

//...

}

The structure representing the captured NV12 image is a bit different than the preview frame structure we have seen before. The structure contains data string which contains both Y and UV(CbCr) data under one pointer. However the luma to chroma bytes’ proportions are the same – each pixel contains unique luma value and chroma values pair is subsampled for groups of 4 pixels.

Figure 4 - The camera image data format

That’s why, if providing properly set pointers as parameters, we can manipulate the luma and chroma data of the captured photo, using the same functions we have implemented for the preview frame modifications: _luma_mod_blow_highlights() and _chroma_mod_sepia().

// Wrapper function for modifying the final picture captured
int modify_final_image(camera_image_data_s *image)
{
    if (image->format == CAMERA_PIXEL_FORMAT_NV12)
	{
		dlog_print(DLOG_INFO, LOG_TAG, "We've got NV12 data - data [%p], length [%d], width [%d], height [%d]",
			image->data, image->size, image->width, image->height);
		int luma_size = image->width * image->height;
		int chroma_size = luma_size/2;
		_luma_mod_blow_highlights(image->data, luma_size);
		_chroma_mod_sepia((image->data)+luma_size, chroma_size);
		dlog_print(DLOG_INFO, LOG_TAG, "The modifications have been applied.");
		return 0;
	}
	else
	{
		dlog_print(DLOG_ERROR, LOG_TAG, "Wrong image format!");
		return -1;
	}
}

Step 4: Finalize the work

The last thing we have to do is to encode the modified picture to JPEG and save it to the device storage. It is done using the image_util_encode_jpeg() API function.

We could finish here, but it may be also useful to update the Media DB with the newly taken picture immediately. It is not essential to do this, however calling media_content_scan_file() will make the newly taken picture to appear in the apps such as the Gallery.

//...
#include <image_util.h>
//...
int finalize_taking_picture(camera_image_data_s *image, const char *filename)
{
    if (image->format == CAMERA_PIXEL_FORMAT_NV12
		&& image_util_encode_jpeg(image->data, image->width, image->height, IMAGE_UTIL_COLORSPACE_NV12, 100, filename) == IMAGE_UTIL_ERROR_NONE)
	{
		dlog_print(DLOG_INFO, LOG_TAG, "Image saved successfully.");

		// Updating media db...
		// This line is optional, use it when you want to update media database
		// instantly after taking a picture.
		_update_media_db(filename);

		return 0;
	}
	else
	{
		dlog_print(DLOG_ERROR, LOG_TAG, "Error occurred when saving image!");
		return -1;
	}
}

// Optional function for updating media database on the device
// instantly after taking a picture.
// It is not necessary for capturing the picture itself,
// but only to reflect the newly taken picture in the apps such as Gallery.

//...
#include <media_content.h> //when you want to update media db
//...


void _update_media_db(const char * filename)
{
    if (media_content_connect() == MEDIA_CONTENT_ERROR_NONE)
	{
		if (media_content_scan_file(filename) == MEDIA_CONTENT_ERROR_NONE)
		{
			dlog_print(DLOG_INFO, LOG_TAG, "Media database updated with the new file.");
		}
		media_content_disconnect();
	}
}

Summary

As you can see now, by using Tizen Native API and nothing more, you can grab and modify the camera preview and a captured picture. Now you can modify any camera app using the tips presented in this article as well as create your own more complex modifications to the camera preview and/or the picture.

The source code described in this article is located in the attachment.