CameraBin 2
CameraBin

CameraBin 2

Current camerabin is a single element and has some complex code with input/output-selectors for controlling the flow of buffers. Making it more modularized and with less -selectors would simplify the code a lot. We hope caps negotiation will be simplified, too. And more tricky features could be implemented inside still/video capturing modules(bins). We could also switch a whole sub-bins to ones with a different implementations. The multi-pad source design enables optimizations for hardware that can stream simultaneously on all pads.

Even though the new camerabin will provide the same basic functions as the current one (and have quite similar API), it will be built from ground up again with a different design. We should make it camerabin2 and take it in parallel with camerabin. This way we avoid major regressions and api changes hassle to applications using current camerabin. When they feel ready, the switch to camerabin2 wouldn't be hard. Also, we can keep bugfixing on old camerabin and keeping everything using it working.

Modules

?CameraBin2 will have 4 internal modules (bins), it should be possible to use any of them outside of camerabin2 (applications, like Empathy and Cheese could make use of those):

[[!img CameraBin2.png].

CameraSrc

It's a bin with 3 output pads for parallel flow (video recording, image capture and viewfinder)
Need to do some baseclass, and there should be an adapter element that would use current video source elements with a single srcpad internally (v4l2src, videotestsrc) and provide 3 output pads as ?CameraBin2 expects.
Might implement gstreamer's Photography interface The embedded world has complex camera drivers that offer more functionality that classic v4l2. E.g. they might have parallel outputs at the same time. The idea here is to make a more complex camerasource api. This could be implemented as a linux-camera-source (as a bin), reusing what we have in CameraBin, or be implemented on top of a different api providing these features as a monolithic block (e.g. gst-omx based camera-source).

This should be done as a baseclass (subclassing a ?GstBin), with the following features:

3 static source pads: image, viewfinder, video (as ?GstGhostpads)
Previewing using 2 sometimes source pads (image_preview, video_preview) or Messages
Do we want to proxy interfaces or application uses them directly from the elements? (photography, colorbalance, ...) This would be the potential class hierarchy:

GstBin
  GstBaseCameraSrc
    GstSimpleV4l2CameraSrc
    GstOmxCameraSrc
    ...

If the actual camera plugin is already implemented extending e.g. ?GstPushSrc, then one would still subclass a new element from ?GstBaseCameraSrc and insert the ?GstPushSrc based camera element there. In the most simple case the wrapper for a powerful hardware based camera source proxies gobject properties and links the internal element pads to the bins ghostpads. We would definitely want to have a ?GstSimpleV4l2CameraSrc as a reference implementation and also providing legacy/fallback support.

Image Capture Bin

Received the image buffers from the camera source, encodes them and saves to disk. Must have queues internally to allow further captures to be taken while others are encoding.

Handles eos/tags between buffers/image captures
Possibility of multiple implementations:
- A sequential encoding implementation using queues
- Parallel encoding (pool of pipelines)
- Other ones for trickier cases (e.g. panorama pictures)
- If we allow multiple implementations: How to guarantee a basic common API here? Do we need to guarantee it?
- Need to do tagging to multiple encoders. e.g. We use jifmux for jpg, what are we using for png?

Video Recording Bin

Responsible for recording video and audio. The video buffers are generated from the camera source, audio is recorded from a internal audio source element (that might be muted by the application).

There's an audio source element in this bin (be careful with the clock selection/distribution)
Newsegment/timestamp handling on recordings was tricky/buggy in current camerabin
Encodebin could work here

Viewfinder Bin

Contains the videosink and any image processing filters/analyzers the application might set
How useful is it outside of camerabin? It's simply a branch of elements. (Hot-swapping of effects?)

Use Cases

Basic picture capture: User sets capture resolution to camerabin2, that forwards it to the camerasrc. User also sends the capture filename to camerabin2, which forwards it to the image capture branch. User then sends the capture signal and camerabin forwards it to the camerasrc. Camerasrc pushes a buffer through its imagesrc and it gets to the image encoding branch. The image encoding branch encodes, puts tags into the image and sends an EOS to the filesink so that it finishes the file.
Basic video capture: User sets capture resolution and fps to camerabin2 and it forwards it to the camerasrc. User sets the capture filename, which is passed to the video recording bin. When the user sends the capture start signal, camerasrc starts pushing buffers through its videosrc pad. The audiosrc inside the videobin also starts recording the audio in sync with the video buffers. The viewfinder must still keep running during this. Recording continues until the user sends the capture stop signal. At this point, eos is pushed on the video recording bin and the file is finalized. It should be possible to pause and continue the recording. It should also be possible to mute audio recording.
Panorama: Panorama pictures are not planned to be supported natively on camerabin2, it could, however, be implemented as a image capture module and replace the default module whenever the user would like to take panorama pics. Currently an external application has to take care of the pictures merging. From camerabin2, an application needs to be able to control camera parameters (AWB, AF and AE) so that subsequent captures use the same values. This is possible by using the 'manual' mode for these properties in the Photography interface. It is also possible to set it to 'manual' after the first capture so that the 3A values are automatically set up for the first capture and maintained as that for the other captures.
Continuous shooting: In this mode, camerabin2 has little to do. This is handled by the camera source module, by setting it to 'continuous shooting' mode it will use faster algorithms to do 3A, thus reducing shot to shot time. The rest of camerabin2 will work as usual.

Other features

Digital zooming (with Photography interface or gstreamer elements)
Night mode adjustment (Use lower framerates to allow more exposure time and have a 'lighter' image)
Querying available capture formats
Autopick framerate
- Highest if not specified
- rounding (when user selected fps isn't available)
colorspace/scalers to guarantee compatibility between elements
Hot-swapping of effects elements.

Intentionally not handled

Cameras that generate non-raw input
Hot swapping of inputs/camera. This can be done by setting it to NULL and changing.

Open Questions

How to do effects hot swapping? cheese and empathy want it.
Do we want a explicit prepare-for-capture step? To be set when the user finishes setting up its preferences?
Use sometimes pads or messages for previewing on the source?
How to map ?CameraBin2 flags to the internal bins flags?
Detection of video sources? (Improve autovideosrc? This would be nice for Cheese)
Effects performance
- Camerabin can do effects on all branches with a single element. Camerabin2 would need one effect element per branch.
- Proper benchmarking needed here.
- Providing effects on the source bin of camerabin2 if it supports it would help here.

CameraBin

TODO

At some point we'd like to move CameraBin to gst-plugins-good. CameraBin already has unit tests and documentation. It is currently in use on Nokia N900 and is considered for cheese. This page lists things that should be done before declaring it stable.

dataflow

In current CameraBin we have a link from videobin back to viewfinder to so what we record. When we do the camera-source baseclass, we should move the input-selector to the ?GstSimpleV4l2CameraSrc implementation or even get rid of the selectors and provide paralell dataflow if needed. Keeping the viewfinder active is important for the cases where one has analyzers in e.g. the viewfinder-slot to get continuous analysis results in videorecording or continuous shooting.

Open items here:

in still capture mode we might want to set the queue in the still image camera source output to "leaky" and block its src pad. We would only unblock the on shutter press for the seleccted amount of images (burstmode). We could also use videorate or even a specialized videogate element here. What the element should do:
- only let n frames through (n=0...)
- drop frames to produce a certain output rate

focusing

We need more focus mode. Right now the API can start AF and get a signal when AF is done. There are also camera subsystems providing continuous focusing, where we would just lock the AF on half press, adjust frame and shot.

continuous shooting

We would set the queue to leaky and block the capture pad. In contrast to single capture mode, we would keep capture-setting frozen and let a frame through, each time the capture button is pressed.

bracketing

Lock settings, take various shots with some settings changed, merge single images with some filter. We can use the GstController for scheduling the parameter changes. Some issues with that though:

the camerabin pipeline is always running - we need a newsegment on the image capture pipeline, so that timestamps start with zero
the camerasrc is a live-src, thus it is hard to align the control-changes with frame times ( see bug 610338 and bug 616173).
v4l2 has no api yet to do timestamped CID changes

recording timestamps

Try pushing a newsegment to videobin, instead of rebasing timestamps with a buffer-probe.

Open Items

effect preview

When adding effects to imagebin or videobin, we might want to get a preview on viewfinder for those too. We either need to apply certain effect in the source already or add them twice.

isp internal effects and effect selection

Right now camerabin offers slots for setting effects on each of the imagebin, viewfinderbin and videobin. Some ISPs (image signal processor) come with internal effects (e.g. videostabilisation). Effect settings should then be exposed as gobject properties on the camera source, so that a camera ui can set a gstreamer plugin as an effect and link the ui to the gobject properties or activate the isp internal effect and connect the ui to the properties of that.

The effect parameters should not be added to ?GstPhotography to avoid it growing unbounded. ?GstPhotography is for capture settings. This is needed for post processing effects.

highspeed capture

We might need a videorate element infront of the viewfinder bin to reduce the framerate there.