Systems and methods for deferring post-process effects in video encoding are disclosed. The systems and methods are capable of calculating the capability of client hardware to defer load, and summing a known load of one or more deferral candidates to evaluate how many post-process deferral candidates are capable of being deferred to client hardware. The systems and methods are also capable of sending an updated deferral list of post-processes to a remote server that can apply the list of deferred post-process candidates during the post-processing phase of a first video frame.
Systems and methods for reducing latency through motion estimation and compensation incorporate a client device that uses lookup tables from a remote server to match, tag, and sum motion vectors. When a remote server transmits encoded video frames to the client, the client decodes them and applies the summed motion vectors to estimate motion. The technology also generate motion vectors based on predetermined criteria and transmit them and invalidators to a client, which caches them. The server instructs the client to receive input and use it to match to cached motion vectors or invalidators. The technology also caches repetitive motion vectors and transmits previously generated motion vector libraries to a client. The server instructs the client to calculate a motion estimate and instructs the client to update the stored motion vector library, so that the client applies the stored library to initiate motion prior to receiving actual motion vector data.
ABSTRACT Systems and methods for improving computer technology related to the rendering and encoding of images are disclosed, preferably for use in a video-game environment. In certain embodiments, a codec is used to encode one or more reference images for a partial range of encoder settings and a renderer is used to generate one or more rendering quality-settings profiles, generate one or more reference images, calculate perceived qualities for each of the one or more reference images, re-render the one or more reference images for each of the one or more rendering quality-setting profiles, and calculate perceived qualities for each of the one or more re-rendered reference images. The renderer compares the perceived qualities of the reference images to the perceived qualities of the re-rendered images and matches them. Those matches result in an association of one or more encoder settings with their matching rendering quality-settings profiles into a look-up table. Date Recue/Date Received 2020-06-09
H04N 19/172 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
4.
SYSTEMS AND METHODS FOR RENDERING & PRE-ENCODED LOAD ESTIMATION BASED ENCODER HINTING
Systems and methods for hinting an encoder are disclosed in which a server monitors for information related to changes in frame encoding, calculates tolerance boundaries, rolling average frame time, and short-term trends in frame time, and uses those calculations to identify a frame time peak. The server then hints an encoder to modulate the quality settings of frame output in proportion to the size of the frame time peak. In certain embodiments, the calculations of tolerance boundaries, rolling average frame time, and short-term trends in frame time are used to identify high-entropy frames. In other embodiments, the server calculates a quality scaling value for a frame time outside of the tolerance boundaries, and uses that calculation to identify a frame time peak.
H04N 19/85 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
H04N 19/115 - Selection of the code volume for a coding unit prior to coding
H04N 19/15 - Data rate or code amount at the encoder output by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
H04N 19/154 - Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion
H04N 19/23 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
5.
SYSTEMS AND METHODS FOR RENDERING & PRE-ENCODED LOAD ESTIMATION BASED ENCODER HINTING
Systems and methods for hinting an encoder are disclosed in which a server monitors for information related to changes in frame encoding, calculates tolerance boundaries, rolling average frame time, and short-term trends in frame time, and uses those calculations to identify a frame time peak. The server then hints an encoder to modulate the quality settings of frame output in proportion to the size of the frame time peak. In certain embodiments, the calculations of tolerance boundaries, rolling average frame time, and short-term trends in frame time are used to identify high-entropy frames. In other embodiments, the server calculates a quality scaling value for a frame time outside of the tolerance boundaries, and uses that calculation to identify a frame time peak.
H04N 19/154 - Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion
H04N 19/137 - Motion inside a coding unit, e.g. average field, frame or block difference
H04N 19/172 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
H04N 19/51 - Motion estimation or motion compensation
6.
SYSTEMS AND METHODS FOR GAME-GENERATED MOTION VECTORS
Systems and methods for integrated graphics rendering are disclosed. In certain embodiments, the systems and methods utilize a graphics engine, a video encoding engine, and remote client coding engine to render graphics over a network. The systems and methods involve the generation of per-pixel motion vectors, which are converted to per-block motion vectors at the graphics engine. The graphics engine injects these per-block motion vectors into a video encoding engine, such that the video encoding engine may convert those vectors into encoded video data for transmission to the remote client coding engine.
H04N 19/503 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
A63F 13/355 - Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an MPEG-stream for transmitting to a mobile phone or a thin client
H04N 19/61 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
Systems and methods for integrated graphics rendering are disclosed. In certain embodiments, the systems and methods utilize a graphics engine, a video encoding engine, and remote client coding engine to render graphics over a network. The systems and methods involve the generation of per-pixel motion vectors, which are converted to per-block motion vectors at the graphics engine. The graphics engine injects these per-block motion vectors into a video encoding engine, such that the video encoding engine may convert those vectors into encoded video data for transmission to the remote client coding engine.
Systems and methods for hinting an encoder are disclosed in which a server monitors for information related to changes in frame rendering, calculates tolerance boundaries, rolling average frame time and/or short-term trends in frame time, using those calculations to identify a frame time peak. The server then hints a codec (encoder) to modulate the quality settings of frame output in proportion to the size of the frame time peak. In certain embodiments, a renderer records one or more playthroughs in a game environment, sorts a plurality of frames from one or more playthroughs into a plurality of cells on a heatmap, and collects the list of sorted frames. A codec may then encode one or more frames from the list of sorted frames to calculate an average encoded frame size for each cell in the heatmap, and associate each average encoded frame size with a per-cell normalized encoder quality setting.
H04N 19/137 - Motion inside a coding unit, e.g. average field, frame or block difference
H04N 19/154 - Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion
H04N 19/17 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
H04N 19/42 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals - characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
H04N 19/51 - Motion estimation or motion compensation
9.
SYSTEMS AND METHODS FOR ENCODER-GUIDED ADAPTIVE-QUALITY RENDERING
Systems and methods for improving computer technology related to the rendering and encoding of images are disclosed, preferably for use in a video-game environment. In certain embodiments, a codec is used to encode one or more reference images for a partial range of encoder settings and a renderer is used to generate one or more rendering quality-settings profiles, generate one or more reference images, calculate perceived qualities for each of the one or more reference images, re-render the one or more reference images for each of the one or more rendering quality-setting profiles, and calculate perceived qualities for each of the one or more re-rendered reference images. The renderer compares the perceived qualities of the reference images to the perceived qualities of the re-rendered images and matches them. Those matches result in an association of one or more encoder settings with their matching rendering quality-settings profiles into a look-up table.
10.
SYSTEMS AND METHODS FOR RENDERING & PRE-ENCODED LOAD ESTIMATION BASED ENCODER HINTING
ABSTRACT Systems and methods for hinting an encoder are disclosed in which a server monitors for information related to changes in frame encoding, calculates tolerance boundaries, rolling average frame time, and short-term trends in frame time, and uses those calculations to identify a frame time peak. The server then hints an encoder to modulate the quality settings of frame output in proportion to the size of the frame time peak. In certain embodiments, the calculations of tolerance boundaries, rolling average frame time, and short-term trends in frame time are used to identify high-entropy frames. In other embodiments, the server calculates a quality scaling value for a frame time outside of the tolerance boundaries, and uses that calculation to identify a frame time peak. Date Recue/Date Received 2021-01-19
H04N 19/85 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
H04N 19/115 - Selection of the code volume for a coding unit prior to coding
H04N 19/142 - Detection of scene cut or scene change
H04N 19/15 - Data rate or code amount at the encoder output by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
H04N 19/154 - Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion
11.
SYSTEMS AND METHODS FOR RENDERING & PRE-ENCODED LOAD ESTIMATION BASED ENCODER HINTING
Systems and methods for hinting an encoder are disclosed in which a server monitors for information related to changes in frame encoding, calculates tolerance boundaries, rolling average frame time, and short-term trends in frame time, and uses those calculations to identify a frame time peak. The server then hints an encoder to modulate the quality settings of frame output in proportion to the size of the frame time peak. In certain embodiments, the calculations of tolerance boundaries, rolling average frame time, and short-term trends in frame time are used to identify high-entropy frames. In other embodiments, the server calculates a quality scaling value for a frame time outside of the tolerance boundaries, and uses that calculation to identify a frame time peak.
H04N 19/85 - Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
A63F 13/355 - Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an MPEG-stream for transmitting to a mobile phone or a thin client
H04N 19/15 - Data rate or code amount at the encoder output by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
H04N 19/154 - Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion
12.
PLAYER INPUT MOTION COMPENSATION BY ANTICIPATING MOTION VECTORS
Systems and methods for reducing latency through motion estimation and compensation incorporate a client device that uses lookup tables from a remote server to match, tag, and sum motion vectors. When a remote server transmits encoded video frames to the client, the client decodes them and applies the summed motion vectors to estimate motion. The technology also generate motion vectors based on predetermined criteria and transmit them and invalidators to a client, which caches them. The server instructs the client to receive input and use it to match to cached motion vectors or invalidators. The technology also caches repetitive motion vectors and transmits previously generated motion vector libraries to a client. The server instructs the client to calculate a motion estimate and instructs the client to update the stored motion vector library, so that the client applies the stored library to initiate motion prior to receiving actual motion vector data.
13.
PLAYER INPUT MOTION COMPENSATION BY ANTICIPATING MOTION VECTORS
Systems and methods for reducing latency through motion estimation and compensation incorporate a client device that uses lookup tables from a remote server to match, tag, and sum motion vectors. When a remote server transmits encoded video frames to the client, the client decodes them and applies the summed motion vectors to estimate motion. The technology also generate motion vectors based on predetermined criteria and transmit them and invalidators to a client, which caches them. The server instructs the client to receive input and use it to match to cached motion vectors or invalidators. The technology also caches repetitive motion vectors and transmits previously generated motion vector libraries to a client. The server instructs the client to calculate a motion estimate and instructs the client to update the stored motion vector library, so that the client applies the stored library to initiate motion prior to receiving actual motion vector data.
patents
