We need a mechanism to transport extra data in a GstBuffer. Unlike caps the content is likely to change for each buffer.

Examples:

• Pan and scan (video subregion) position information
• interlacing markers (top field, bottom field)
• theora granulepos, instead of slapping it in the 'offset' field
• MPEG Decoding timestamps, which we currently never send along.
• Feature-Extraction elements can add feature data
• Some image capture sensors can provide histograms with the frames
  • Some formats (mpeg-ts from HDV cameras) provide date/gain/shutter-speed/scene-change at a regular interval but not for every frame (around 10 times per second).

• packet priority and timeout could be used by network plugins (such as DCCP) to send more important packets when congestion is detected. Ideas:

  1. GstStructure in GstBuffer
  2. GstTaglist in GstBuffer
  3. syncronized downstream events
  4. Store GstStructures in GstIndex GstStructure is also use in GstTaglist. The structure can be hierarchical. We could require that the producer uses its element name as a root tag. This way the taglist could carry multiple data-sets identified in a unique way. Elements that can use the data need to know the name of the element that created it.

Issues:

• Performance
  • copying, freeing buffers need to check/handle the presence of extra metadata
• Transformations
  • there might be transformation that render the data useless. e.g. if the data is a histogram, color changes invalidate it. if its a sub-region, size changes invalidate it.
• Mixing
  • if we go for (B) we could use the tag merge modes
• Muxing
  • using and/or dropping the data, it does not sound sensible to send it further
• Copies (tee)
  • make duplicates