[lkml]   [2018]   [Jul]   [24]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
Subject[PATCH 0/2] Document memory-to-memory video codec interfaces
This series attempts to add the documentation of what was discussed
during Media Workshops at LinuxCon Europe 2012 in Barcelona and then
later Embedded Linux Conference Europe 2014 in Düsseldorf and then
eventually written down by Pawel Osciak and tweaked a bit by Chrome OS
video team (but mostly in a cosmetic way or making the document more
precise), during the several years of Chrome OS using the APIs in

Note that most, if not all, of the API is already implemented in
existing mainline drivers, such as s5p-mfc or mtk-vcodec. Intention of
this series is just to formalize what we already have.

It is an initial conversion from Google Docs to RST, so formatting is
likely to need some further polishing. It is also the first time for me
to create such long RST documention. I could not find any other instance
of similar userspace sequence specifications among our Media documents,
so I mostly followed what was there in the source. Feel free to suggest
a better format.

Much of credits should go to Pawel Osciak, for writing most of the
original text of the initial RFC.

Changes since RFC:
- The number of changes is too big to list them all here. Thanks to
a huge number of very useful comments from everyone (Philipp, Hans,
Nicolas, Dave, Stanimir, Alexandre) we should have the interfaces much
more specified now. The issues collected since previous revisions and
answers leading to this revision are listed below.

General issues

Should TRY_/S_FMT really return an error if invalid format is set,
rather than falling back to some valid format? That would be
contradicting to the general spec.

Answer: Keep non-error behavior for existing spec compatibility, but
consider returning error for Request API.

The number of possible opens of M2M video node should not be
artificially limited. Drivers should defer allocating limited resources
(e.g. hardware instances) until initialization is attempted to allow
probing and pre-opening of video nodes. (Hans suggested vb2 queue setup

Answer: Allocate hardware resources in REQBUFS (or later).

How about colorimetry settings (colorspace, xfer function, etc.)?
Normally it is not needed for decoding itself, but some codecs can parse
it from the stream. If user space can parse by itself, should it set it
on OUTPUT queue? What should happen on CAPTURE queue if colorimetry can
be parsed, colorimetry can’t be parsed.

Answer: Mention copying colorimetry from OUTPUT to CAPTURE queue only.
Potentially extend for hardware that can do colorspace conversion later.

Decoder issues

Is VIDIOC_ENUM_FRAMESIZES mandatory? Coda doesn’t implement it, s5p-mfc

Answer: Make it mandatory. Otherwise nobody would implement it.

Should we support all the three specification modes of
VIDIOC_ENUM_FRAMESIZES (continuous, discrete and stepwise)? On both

Answer: Support all 3 size specification modes, not to diverge from
general specification.

Should ENUM_FRAMESIZES return coded or visible size?

Answer: That should be the value that characterizes the stream, so
coded size. Visible size is just a crop.

How should ENUM_FRAMESIZES be affected by profiles and levels?

Answer: Not in current specification - the logic is too complicated and
it might make more sense to actually handle this in user space. (In
theory, level implies supported frame sizes + other factors.)

Is VIDIOC_ENUM_FRAMEINTERVALS mandatory? Coda doesn’t implement it,
s5p-mfc either. What is the meaning of frame interval for m2m in

Answer: Do not include in this specification, because there is no way
to return meaningful values for memory-to-memory devices.

What to do for coded formats for which coded resolution can’t be parsed
(due to format or hardware limitation)? Current draft mentions setting
them on OUTPUT queue. What would be the effect on CAPTURE queue?
Should OUTPUT queue format include width/height? Would that mean coded
or visible size? If so, should they always be configured? Gstreamer
seems to pass visible size from the container.

Answer: If OUTPUT format has non-zero width and height, the driver must
behave as it instantly parsed the coded size from the stream, including
updating CAPTURE format and queuing source change event. If another
parameters are parsed later by hardware, a dynamic resolution change
sequence would be triggered. However, for hardware not parsing such
parameters from the stream, stateless API should be seriously

How about the legacy behavior of G_FMT(CAPTURE) blocking until queued
OUTPUT buffers are processed?

Answer: Do not include in the specification, keep in existing drivers for

Should we allow preallocating CAPTURE queue before parsing as an
optimization? If user space allocated buffers bigger than required, it
may be desirable to use them if hardware allows. Similarly, if a
decreasing resolution change happened, it may be desirable to avoid
buffer reallocation. Gstreamer seems to rely on this behavior to be
allowed and works luckily because it allocates resolutions matching what
is parsed later.

Answer: Yes. The client can setup CAPTURE queue beforehand. The driver
would still issue a source change event, but if existing buffers are
compatible with driver requirements (size and count), there is no need to
reallocate. Similarly for dynamic resolution change.

What is the meaning of CAPTURE format? Should it be coded format,
visible format or something else?

Answer: It should be a hardware-specific frame buffer size (>= coded
size), minimum needed for decoding to proceed.

Which selection target should be used for visible rectangle? Should we
also report CROP/COMPOSE_DEFAULT and COMPOSE_PADDED (the area that
hardware actually overwrites)? How about CROP_BOUNDS?

Answer: COMPOSE. Also require most of the other meaningful targets.
Make them default to visible rectangle and, on hardware without
crop/compose/scale ability, read-only.

What if the hardware only supports handling complete frames? Current
draft says that Source OUTPUT buffers must contain: - H.264/AVC: one or
more complete NALUs of an Annex B elementary stream; one buffer does not
have to contain enough data to decode a frame;

Answer: Defer to specification of particular V4L2_PIX_FMT (FourCC), to be
further specified later. Current drivers seem to implement support for
various formats in various ways (especially H264). Moreover, various
userspace applications have their own way of splitting the bitstream. We
need to keep all existing users working, so sorting this out will require
quite a bit of effort and should not be blocking the already de facto
defined part of the specification.

Does the driver need to flush its CAPTURE queue internally when a seek is
issued? Or the client needs to explicitly restart streaming on CAPTURE

Answer: No guarantees for CAPTURE queue from codec. User space needs to

Must all drivers support dynamic resolution change? Gstreamer parses the
stream itself and it can handle the change itself by resetting the

Answer: Yes, if it's a feature of the coded format. There is already
userspace relying on this. A hardware that cannot support this, should
likely use the stateless codec interface.

What happens with OUTPUT queue format (resolution, colorimetry) after
resolution change? Currently always 0 on s5p-mfc. mtk-vcodec reports
coded resolution.

Answer: Coded size on OUTPUT queue.

Can we allow G_FMT(CAPTURE) after resolution change before
REQBUFS(CAPTURE, 0)? This would allow keeping current buffer set if the
resolution decreased.

Answer: Yes, even before STREAMOFF(CAPTURE).

Should the client also read visible resolution after resolution change?
Current draft doesn’t mention it.

Answer: Yes.

Is there a requirement or expectation for the encoded data to be framed
as a single encoded frame per buffer, or is feeding in full buffer sized
chunks from a ES valid? It's not stated for the description of
V4L2_PIX_FMT_H264 etc either. Should we tie such requirements to
particular format (FourCC)?

Answer: Defer to specification of particular V4L2_PIX_FMT (FourCC), to be
further specified later. Similarly to the earlier issue with H264.

How about first frame in case of VP8, VP9 or H264_NO_SC? Should that
include only headers?

Answer: There is no separate header in case of VP8 and VP9. There are
only full frames. V4L2_PIX_FMT_H264_NO_SC implies user space splitting
headers (SPS, PPS) and frame data (slice) into separate buffers, due
to the nature of the format.

Should we have a separate format for headers and data in separate

Answer: As with the other format-specific issues - defer to format

How about timestamp copying between OUTPUT and CAPTURE buffers?
The draft says - buffers may become available on the CAPTURE queue
without additional buffers queued to OUTPUT (e.g. during flush or EOS)
What timestamps would those buffers have?

Answer: Those CAPTURE buffers would originate from an earelier OUTPUT
buffer, just being delayed. Timestamp would match those OUTPUT buffers.

Supposedly there are existing decoders that can’t deal with seek to a
non-resume point and end up returning corrupt frames.

Answer: There is userspace relying on this behavior not crashing the
system or causing a fatal decode error. Corrupt frames are okay. We can
extend the specification later with a control that gives a hint to the

Maybe we should state what happens to reference buffers, things like DPB.
Can we CMD_STOP, then V4L2_DEC_CMD_START and continue with the ref kept?

Answer: Refs lost - same as STREAMOFF(CAPTURE), STREAMON(CAPTURE), except
that buffers are successfully returned to user space.

After I streamoff, do I need to send PPS/SPS again after STREAMON, or
will the codec remember, and the following IDR is fine? (ndufresne: For
sure the DPB will be gone)

Answer: Decoder needs to keep PPS/SPS across STREAMOFF(OUTPUT),
STREAMON(OUTPUT). If we seek to another place in the stream that
references the same PPS/SPS, no need to queue the same PPS/SPS again
(since decoder needs to hold it). If we seek somewhere far, skipping
PPS/SPS on the way, we can’t guarantee anything. In practice most client
implementations already include PPS/SPS at seek before IDR.

Encoder issues

Is S_FMT() really mandatory during initialization sequence? In theory,
the client could just G_FMT() and use what’s already there. (tfiga: In
practice unlikely.)

Answer: Not mandatory, but it's the only thing that makes sense.

When does the actual encoding start? Once both queues are streaming?

Answer: When both queues start streaming.

When does the encoding stop/resets? As soon as one queue receives

Answer: STREAMOFF on CAPTURE. After restarting streaming on CAPTURE,
encoder will generate a stream independent of the stream generated
before. E.g. no references frames from before the restart (no H.264 long
term reference), any headers that must be included in a standalone stream
must be produced again. OUTPUT queue might be restarted on demand to
let the client change the buffer set or extended later to support
encoding streams with dynamic resolution changes.

How should we handle hardware that cannot control encoding parameters
dynamically? Should the driver internally stop, reconfigure and restart?
Or should we defer this to user space?

Answer: Disallow setting respective controls when streaming.

Which queue should be master, i.e. be capable of overriding settings on
the other queue?

Answer: CAPTURE, since coded format is likely to determine the list of
supported raw formats.

How should we describe the behavior of two queues?

Answer: Say that standard M2M principles apply. Also mention no direct
relation between order of raw frames being queued and encoded frames
dequeued, other than timestamp.

How should encoder controls be handled?

Answer: Keep up to the driver. Use Request API to set controls for exact

What should VIDIOC_STREAMON do on an already streaming queue, but after
says A read() or VIDIOC_STREAMON call sends an implicit START command to
the encoder if it has not been started yet.
says If VIDIOC_STREAMON is called when streaming is already in progress,
or if VIDIOC_STREAMOFF is called when streaming is already stopped, then
0 is returned. Nothing happens in the case of VIDIOC_STREAMON[...].

Answer: Nothing, as per the specification. Use V4L2_ENC_CMD_START for
resuming from pause.

Tomasz Figa (2):
media: docs-rst: Document memory-to-memory video decoder interface
media: docs-rst: Document memory-to-memory video encoder interface

Documentation/media/uapi/v4l/dev-decoder.rst | 872 +++++++++++++++++++
Documentation/media/uapi/v4l/dev-encoder.rst | 550 ++++++++++++
Documentation/media/uapi/v4l/devices.rst | 2 +
Documentation/media/uapi/v4l/v4l2.rst | 12 +-
4 files changed, 1435 insertions(+), 1 deletion(-)
create mode 100644 Documentation/media/uapi/v4l/dev-decoder.rst
create mode 100644 Documentation/media/uapi/v4l/dev-encoder.rst


 \ /
  Last update: 2018-07-24 16:07    [W:0.318 / U:2.536 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site