A data dictionary generation system automatically populates and updates a data dictionary for listings offering shared data. A data dictionary includes metadata describing the shared data, including the individual objects, such as the individual tables, schemas, views, and functions. The shared data and each individual data object may be described in the data dictionary by a set of data fields that corresponds to the shared dataset or the object type of the individual object. The data dictionary can be presented to data consumers along with the description of the listing to provide data consumers with a comprehensive description of the shared data provided by a listing, including a high-level summary of the shared data and description of each individual object included in the shared data. The data dictionary allows data consumers to understand the contents of the shared data and how to use the shared data.
Embodiments of the present disclosure may provide a database optimization system that can generate computational values through a database compiler and assignment data for execution of a query by a plurality of nodes of a database system. The computational values and assignment data can be generated by one or more machine learning schemes. The machine learning schemes can be trained on previous computational values and previous assignment data.
Disclosed is an execution information sharing system that duplicates execution information to a provider target (and other targets) as it is being loaded to a consumer target. A consumer account of a data sharing platform executes an application shared with it by a provider account of the platform. Consumer and provider configurations indicating consumer and provider targets respectively are generated. The consumer configuration and provider configurations are provided to an event context to generate a first and second event unloaders respectively, wherein the event context maintains a mapping linking both the first event unloader and the second event unloader to the application. In response to receiving execution information from the application, the first event unloader and the second event unloader are retrieved. The execution information is then written to the consumer target and the provider target using the first event unloader and the second event unloader respectively.
The distributed database can implement unstructured data replication using an internal or external storage location. Metadata, such as a directory table that lists the unstructured files, can be replicated across different deployments, followed by replication of the staged data. Replicating the staged data can be implemented by replication of only the stage metadata or replication of the database files between the deployments.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
5.
PROJECTION CONSTRAINTS IN A QUERY PROCESSING SYSTEM
A constraint system enforces projection constraints on data values stored in specified columns of a shared dataset when queries are received by a database system. A projection constraint identifies that the data in a column may be restricted from being projected (e.g., presented, read, outputted) in an output to a received query, while allowing specified operations to be performed on the data and a corresponding output to be provided. For example, the projection constraint may indicate a context for a query that triggers the constraint, such as based on the user that submitted the query. Enforcing projection constraints on queries received at the database system allows for data to be shared and used anonymously by entities to perform various operations without the need to tokenize the data.
A data platform for managing an application as a first-class database object. The data object can include User Interface (UI) components. The data application can be shared by a provider account to a plurality of consumer accounts using a share object and based on grant commands. The consumer accounts can deploy and operate the UI component based on the share object.
In an embodiment, a data platform performs functions including creating a native platform application in a data-provider account of a data platform, the native platform application comprising one or more APIs corresponding to one or more functions; sharing provider data with the native platform application in the data-provider account; installing, in a data-consumer account of the data platform, an application instance of the native platform application, the application instance comprising one or more APIs corresponding to the one or more APIs in the native platform application in the data-provider account; sharing consumer data with the application instance in the data-consumer account; invoking one or more of the APIs of the application instance to execute respective associated functions, the respective associated functions not being visible to the data-consumer account; and saving output of the one or more respective associated functions locally within the data-consumer account.
G06V 10/80 - Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
G06V 20/56 - Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
B60W 50/00 - CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - Details of control systems for road vehicle drive control not related to the control of a particular sub-unit
B60W 50/14 - Means for informing the driver, warning the driver or prompting a driver intervention
8.
CROSS-CLOUD REPLICATION OF RECURRENTLY EXECUTING PIPELINES
In one aspect, a computer-implemented method includes detecting a committed version of recurrently executed tasks of a first data pipeline on a primary deployment that is hosted on a first cloud service, and replicating the committed version of the recurrently executed tasks to a second data pipeline on a secondary deployment that is hosted on a second cloud service.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
The subject technology receives a command to commit a table in a different table format on an external volume. The subject technology generates a first snapshot of the table on internal storage. The subject technology generates a first list of metadata files on the internal storage. The subject technology generates, based on the first list of metadata files, a first set of metadata files on the internal storage. The subject technology generates a second snapshot of the table on the external volume. The subject technology generates a second list of metadata files on the external volume. The subject technology generates, based on the second list of metadata files, a second set of metadata files on the external volume. The subject technology generates a first set of data files in a different file format on the external volume.
A system may include a processing device and a memory storing instructions that, when executed by the processing device, causes the processing device to discover one or more endpoints of a service in view of a name that is unique to the service. In response to receiving a request to resolve the name from a client, the processing device may obtain the one or more endpoints of that service in view of the name. The processing device may filter the one or more endpoints, in view of the name and return the one or more endpoints which are filtered, to the client.
An example apparatus includes a blockchain data circuit that interprets a blockchain description value, a blockchain index circuit that provides a blockchain index data structure in response to the blockchain description value, the blockchain index data structure including a number of attribute values for each of a number of entities associated with the blockchain description value, and a blockchain search circuit that exercises a user interface. The user interface allows for interpreting a user search value, and the blockchain search circuit provides a search description to the user interface in response to the user search value and the blockchain index data structure.
Provided herein are systems and methods for configuring replication group objects. A system includes at least one hardware processor coupled to memory and configured to decode a replication request received from a client device of a data provider. The replication request indicates a plurality of account objects for replication. The plurality of account objects associated with a corresponding plurality of account object types. A replication group object is generated based on the replication request. The replication group object includes a manifest. The manifest lists the plurality of account objects. A replication of the plurality of account objects is performed from a source account of the data provider into at least one target account. The replication is based on the manifest of the replication group object.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 16/25 - Integrating or interfacing systems involving database management systems
Embodiments of the present disclosure describe systems, methods, and computer program products for redacting sensitive data within a database. An example method can include receiving a data query referencing unredacted data of a database, responsive to the data query, executing, by a processing device, a redaction operation to identify sensitive data within the unredacted data of the database, and returning a redacted data set in which the sensitive data is replaced or removed to the data query.
Provided herein are systems and methods for query result data processing, including parallel fetching and processing of query result data. A system includes at least one hardware processor coupled to memory and configured to execute a query received from a first computing device, to obtain query result data. The query result data is stored at a storage location. Query result information associated with the query result data is generated. The query result information includes location information associated with the storage location and schema information associated with the query result data. The query result information is encoded for transmission to the first computing device.
Provided herein are systems and methods for a stream object configuration, including configuring and processing streams on views. For example, a method includes detecting a first stream object on a view. The view includes a query associated with a source table. A syntax tree of the query is determined based on a definition of the view. The view is expanded based on replacing the first stream object with the syntax tree. The syntax tree includes a second stream object on the source table. Stream expansion of the second stream object is performed based on computing changes on the source table.
Embodiments of the present disclosure provide systems and methods for defining database roles to allow sharing of the objects within a database in a more granular fashion. A database role is created within a database, and privileges are granted to the database role by a provider account. The database role may be granted to a share object, which would result in the share object being indirectly granted all privileges that have been granted to the database role. Once the share object has been hydrated in the consumer account, the local administrator can then decide which local roles are granted usage on the shared database role, thus ultimately allowing some level of filtering as to which shared objects can be accessed by which local roles.
The subject technology determines, after a period of time elapses over a periodic segment of time, an imbalance of cluster instances deployed in multiple zones based on a threshold value, the cluster instances including different types of clusters associated with compute service manager instances. The subject technology identifies a particular type of cluster instance to include in a particular zone from the multiple zones. The subject technology adds the particular type of cluster instance to the particular zone to meet a global balancing of cluster instances in the multiple zones. The subject technology determines, after a second period of time elapses over the periodic segment of time, that a number of cluster instances deployed in the multiple zones is below the threshold value indicating a current balance of cluster instances in the multiple zones.
Aspects of the present disclosure address systems, methods, and devices for tracking object dependencies in a cloud database system. An object dependency created between a referencing object and a referenced object is detected. Based on detecting the object dependency, a dependency record is generated. The dependency record includes dependency information describing the object dependency between the reference object and the referenced object. The dependency record is stored in a database of dependency records.
A file access system for user defined functions (UDFs) can be implemented on a distributed database system. The system can store UDF signatures and interfaces (e.g., classes, sub-classes) that can be called by other users. Upon a UDF being called, one or more interface objects (e.g., InputStream) can be created and requests transferred to a execution node via a network channel. The execution node can implement multiple threads that are authorized and download file data from a staging location (e.g., internal stage, external stage) concurrently.
Disclosed are systems, methods, and non-transitory computer- readable media for a providing stored procedures in a network-based database system. A database system executes a stored procedure within a sandbox process. The sandbox process separates the stored procedure from the other services and processes of the database system and may also limit the resources (e.g., storage, memory, etc.) and functionality available to the stored procedure. Stored procedures are commonly designed to execute database queries that are processed by other components of the database system. To provide for secure communication between the stored procedure and the other components of the database system, the sandbox process is loaded with java database connectivity (JDBC) libraries (e.g., JDBC instance) that is modified to limit the communication abilities of the stored procedure.
A query referencing an external table function provided by a remote software component is received. Requests to execute the external table function on input data are sent to a proxy service. A first request includes a batch of input rows from the input data. A first response to the first request received from the proxy service includes a first portion of result data and a pagination token. The pagination token indicates that at least a second portion of the result data corresponding to the first, batch of input rows is to be obtained from the remote software component. Based on the pagination token, a second request is sent to obtain the second portion of the result data. One or more responses are received from the proxy service that comprise at least the second portion of the result data. The result data is processed according to the query.
Disclosed herein are systems and methods for database replication to a remote deployment with automated fulfillment. In an embodiment, a data platform provisions a remote-deployment account of a data provider at a remote deployment of the data platform with a set of one or more replication- preparation objects associated with a primary-deployment database that is resident in a primary-deployment account of the data provider at a primary deployment of the data platform. After the provisioning, the data platform detects receipt of a database-replication request associated with a data consumer, requesting availability of a local instance of the primary-deployment database in the remote-deployment account of the data provider. The data platform responsively performs one or more remote-deployment-activation operations, which makes available to the data consumer a local instance of the primary- deployment database in the remote-deployment account of the data provider at the remote deployment of the data platform.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
G06F 16/21 - Design, administration or maintenance of databases
H04L 67/1095 - Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
H04L 69/10 - Streamlined, light-weight or high-speed protocols, e.g. express transfer protocol [XTP] or byte stream
Generating semantic names for a data set is described. An example method can include retrieving data from a data set, the data organized in a plurality of columns. The method may also include generating one or more candidate semantic categories for that column, wherein each of the one or more candidate semantic categories has a corresponding probability for each of the columns. The method may also further include creating a feature vector for each column from the one or more column candidate semantic categories and the corresponding probabilities. Additionally, the method may also include selecting, for each column, a column semantic category from the one or more candidate semantic categories using at least the feature vector and a trained machine learning model.
Systems and methods for managing input and output error of a machine learning (ML) model in a database system are presented herein. A set of test queries is executed on a first version of a database system to generate first test data, wherein the first version of the system comprises a ML model to generate an output corresponding to a function of the database system. An error model is trained based on the first test data and second test data generated based on a previous version of the system. The error model determines an error associated with the ML model between the first and previous versions of the system. The first version of the system is deployed with the error model, which corrects an output or an input of the ML model until sufficient data has been produced by the error model to retrain the ML model.
The subject technology receives a query directed to a set of source tables, each source table organized into a set of micro-partitions. The subject technology determines a set of metadata, the set of metadata comprising table metadata, query metadata, and historical data related to the query. The subject technology predicts, using a machine learning model, an indicator of an amount of computing resources for executing the query based at least in part on the set of metadata. The subject technology generates a query plan for executing the query based at least in part on the predicted indicator of the amount of computing resources. The subject technology executes the query based at least in part on the query plan.
A restricted view definition is received by a database system. The restricted view definition defines a view over a database table with one or more restrictions on use of the view. The view over the database table is generated based on the restricted view definition. A query directed at the view is received by the database system. The database system determines whether the query directed at the view is permitted based on the one or more restrictions on the use of the view.
Security credentials associated with a first account maintained by a database system are authenticated. Based on authenticating the security credentials, a user interface element is provided to enable sharing of customer data associated with the first account with an application. A request to share customer data with the application is received. Based on the request, third-party data that is accessible by the first account is identified. The third-party data corresponds to a second account maintained by the database system. The second account corresponds to a third-party data provider. The application is enabled to access cloud data associated with the first account based on the request. The cloud data comprises the customer data and the third-party data.
Row-level security (RLS) may provide fine-grained access control based on flexible, user-defined access policies to databases, tables, objects, and other data structures. A RLS policy may be an entity or object that defines rules for row access. A RLS policy may be decoupled or independent from any specific table. This allows more robust and flexible control. A RLS policy may then be attached to one or more tables. The RLS policy may include a Boolean-valued expression.
The subject technology receives, during a query compilation process, a query directed to a set of source tables, each source table from the set of source tables being organized into at least one micro-partition and the query including at least one pruning operation. The subject technology performs, during the query compilation process, a modification of the query for adjusting the at least one pruning operation, the modification being based on a set of statistics collected for previous pruning operations on at least a portion of the set of source tables and a set of heuristics, the set of statistics indicating at least an amount of execution time for each previous query associated with each of the previous pruning operations. The subject technology compiles the query including the modification of the query. The subject technology provides the compiled query to an execution node of a database system for execution.
A method of implementing object tagging framework starts with the processor receiving a tag creation command including a tag name. In response to the tag creation command, the processor creates a current tag. The processor then receives an association command, the tag name and a target object identifier. The processor determines a target object associated with the target object identifier. The target object includes a tag value. The processor associates the current tag with the target object. The processor identifies a first child object of the target object. The target object and the first child object are hierarchical objects. In response to determining that the first child object is tag-unassociated, the processor associates the current tag with the first child object. In response to receiving a query including the tag name, the processor generates an output based on the tag name. Other embodiments are also described herein.
A database export system exports data using a plurality of nodes that process the data to generate structured result files that are partitioned by an export parameter in an export request. The database export system distributes the data and merges the files to avoid small file creation and increase processing speed via parallelism. The database export system generates the result files of a specified maximum size in a final format, where the files are processed merged in a temporary file format. The parallel processing is optimized and constrained per the amount of processing nodes, available memory, requested final file sizes, and operation based ordering to complete data exports in a scalable multi-stage approach.
External functions can provide users of a database system the ability to invoke external services while operating from inside the database system. The techniques described herein address may detect throttling by the external system and automatically adjust or modify a request rate accordingly. Also, the techniques described herein may increase the request rate to optimize the use of the resources provided by the external system. These techniques allow the database system to improve efficiency when working with external services, which may or may not be elastic or scalable. Moreover, these techniques can decrease query execution times and reduce costs.
The subject technology receives a first transaction. The subject technology assigns a first read version to the first transaction, the first read version indicating a first version of the linearizable storage. The subject technology performs a read operation from the first transaction on a table in a database. The subject technology determines a first commit version identifier corresponding to first data resulting from the read operation. The subject technology, in response to determining that a particular write operation is absent from the first transaction, proceeding to execute a different transaction and foregoing to perform a commit process in connection with the first transaction.
Queries executed against a materialized view can execute up to orders of magnitude faster than equivalent queries on a source (or base) table. However, although a query can reference a materialized view directly, a user (e.g., query author) may not know about a relevant materialized view. Moreover, if a source table has multiple materialized views generated, the user may not know which materialized view to reference in the query. Thus, embodiments of the present disclosure provide techniques for automatically rewriting queries directed to a source table to utilize existing materialized views.
Provided herein are systems and methods for providing a secure and efficient way to manage a data exchange's availability and manage the visibility of data listings of the data exchange. For example, the method may comprise specifying, by an exchange admin of the data exchange, a set of regions where the data exchange is available, each of the set of regions comprising one or more remote deployments. The method may further comprise specifying by a data provider, one or more of the set of regions where a data listing owned by the data provider is visible. Upon receiving a request to access the data listing from a remote deployment of the one or more regions, the data provider may determine whether to reject or fulfill the request. In response to determining that the request should be fulfilled, replicating data of the data listing to the remote deployment.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Embodiments of the present disclosure may provide a data clean room allowing secure data analysis across multiple accounts, without the use of third parties. Each account may be associated with a different company or party. The data clean room may provide security functions to safeguard sensitive information. For example, the data clean room may restrict access to data in other accounts. The data clean room may also restrict which data may be used in the analysis and may restrict the output. The overlap data may be anonymized to prevent sensitive information from being revealed.
The subject technology receives first code in a first programming language, the first code including a reference to a first dataset. The subject technology converting the first code to: second code in a second programming language different than the first programming language, and at least one data query in a second format different than the first programming language. Further the subject technology sends the converted first code and the at least one data query to a data system for execution.
A database platform receives an object identifier from a client in association with a database session. The client is associated with a customer account of the database platform, and the database session is associated with the client. In response to receiving the object identifier, the database platform identifies a resolution namespace for the object identifier, where the resolution namespace for the object identifier is a namespace that is specified in the object identifier if the object identifier includes a specified namespace, and where the resolution namespace is otherwise a current account-level namespace of the database session. The database platform resolves the object identifier with reference to the identified resolution namespace for the object identifier, including identifying an object corresponding to the object identifier in the customer account.
The subject technology receives a query plan, the query plan comprising a set of query operations, the set of query operations including at least one aggregation. The subject technology analyzes the at least one aggregation to generate a modified query plan, the modified query plan including at least a top aggregation operator, an intermediate aggregation operator, and a bottom aggregation operator. The subject technology performs, with respect to the intermediate aggregation operator, at least one operation comprising: the subject technology receives an input intermediate data type; the subject technology performs an internalize operation on the input intermediate data type to generate an internal state; the subject technology performs an accumulate operation on the internal state to generate intermediate data; and the subject technology performs an externalize operation on the intermediate data to generate an output data type.
In an embodiment, a database platform receives a request from a client for creation of an attachable-and-detachable database session, and responsively creates the requested attachable-and-detachable database session for the client. The database platform sets the attachable-and-detachable database session as a current database session for the client at the database platform. The database platform determines that the client has detached from the attachable-and-detachable database session, and thereafter continues to maintain the attachable-and-detachable database session in data storage at the database platform.
G06F 16/14 - File systems; File servers - Details of searching files based on file metadata
G06F 16/17 - File systems; File servers - Details of further file system functions
H01L 29/06 - Semiconductor bodies characterised by the shapes, relative sizes, or dispositions of the semiconductor regions
H01L 29/08 - Semiconductor bodies characterised by the shapes, relative sizes, or dispositions of the semiconductor regions with semiconductor regions connected to an electrode carrying current to be rectified, amplified, or switched and such electrode being part of a semiconductor device which comprises three or more electrodes
H04L 29/06 - Communication control; Communication processing characterised by a protocol
H04L 29/08 - Transmission control procedure, e.g. data link level control procedure
Embodiments of the present disclosure may provide a streamlined process for performing operations, such as data sharing and data replication, using multiple accounts. A global identity (also referred to as an organization user) may be employed, where the global identity may have access to multiple accounts across the same or different deployments. The global identity may switch between accounts from its login session and perform various tasks in the context of different accounts without undergoing further authentication.
A query directed at a source table organized into a set of batch units is received. The query includes a pattern matching predicate that specifies a search pattern. A set of N-grams are generated based on the search pattern. A pruning index associated with the source table is accessed. The pruning index comprises a set of filters that index distinct N-grams in each column of the source table. The pruning index is used to identify a subset of batch units to scan for matching data based on the set of N- grams generated for the search pattern. The query is processed by scanning the subset of batch units.
A data platform provides support for unstructured files, semi-structured files, and structured files. In various different embodiments, the data platform provides functions for obtaining relative paths, absolute paths, and presigned URLs for files in internal and external stages; application programming interfaces (APIs) for integrating with external functions; built-in file-catalog tables; integration with cloud-storage platforms; customized views for flexible file-sharing options, stage-encryption options; notification integrations; table functions for generating file-catalog tables; streams on stages; streams on tables; and other features.
G06F 16/901 - Indexing; Data structures therefor; Storage structures
G06F 16/955 - Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
G06F 16/908 - Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
G06F 16/22 - Indexing; Data structures therefor; Storage structures
The subject technology receives, in a first computing process, a user defined function, the user defined function including code related to at least one operation to be performed. The subject technology sends a request based on the at least one operation to a second computing process to perform, the second computing process being different than the first computing process and comprising a sandbox for executing the at least one operation. The subject technology receives, by the second computing process, the request. The subject technology determines, using at least a security policy, whether performing the at least one operation is permitted. The subject technology performs, in the second computing process, the least one operation. The subject technology sends, by the second computing process, a result of the at least one operation to the first computing process.
G06F 21/53 - Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity, buffer overflow or preventing unwanted data erasure by executing in a restricted environment, e.g. sandbox or secure virtual machine
G06F 21/54 - Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity, buffer overflow or preventing unwanted data erasure by adding security routines or objects to programs
G06F 21/62 - Protecting access to data via a platform, e.g. using keys or access control rules
G06F 21/78 - Protecting specific internal or peripheral components, in which the protection of a component leads to protection of the entire computer to assure secure storage of data
Embodiments of the present disclosure may provide a dynamic query execution model. This query execution model may provide acceleration by scaling out parallel parts of a query (also referred to as a fragment) to additional computing resources, for example computing resources leased from a pool of computing resources. Execution of the parts of the query may be coordinated by a parent query coordinator, where the query originated, and a fragment query coordinator.
Provided herein are systems and methods for an efficient method of replicating share objects to remote deployments. For example, the method may comprise modifying a share object of a first account of a data exchange into a global object wherein the share object includes grant metadata indicating share grants to a set of objects of a database. The method may further comprise creating, in a second account of the data exchange, a local replica of the share object on the remote deployment based on the global object, wherein the second account is located in a remote deployment. The set of objects of the database may be replicated to a local database replica on the remote deployment and the share grants may be replicated to the local replica of the share object.
A network device communication system can configure network devices (e.g., a primary and secondary database) to send and receive sequences messages, such as replication data, over a channel comprising a plurality of private network nodes. The messages can be generated and encrypted using one or more key pairs and changing wrapping replication keys to send and receive the messages between different types of database deployments.
Embodiments of the present disclosure may provide cross cloud auto-ingestion techniques. A deployment may monitor multiple queues across different cloud providers and may classify the queues based on their cloud provider type. The deployment may receive notifications from those queues regarding new data ready for ingestion. The deployment may maintain a pool of credentials and may assign appropriate credentials to each queue. The deployment may route the notifications to appropriate receivers based on cloud provider types. The receivers may then auto-ingest new data in the corresponding queue.
The subject technology determines a set of shards corresponding to an external data source accessible via a network, the external data source being hosted by an external system separate from a network-based database system. The subject technology determines, using a stream object, a set of offsets of each shard of the set of shards. The subject technology identifies an operation to perform on the set of shards, the operation comprising a read operation or a write operation. The subject technology, based on the set of shards and the set of offsets, performs the operation on the external data source.
A database platform can track changes using a stream that records changes made by database transactions. Streams on views can be implemented by applying a stream to source data, and applying the view to the streamed data. Users of the database platform can query the stream on the view to review recent changes to the view, such as changes made since the last database transaction.
The subject technology receives a task for executing in a network-based data warehouse system. The subject technology determines a number of execution nodes to execute the task. The subject technology, based on the number of execution nodes, selects a virtual warehouse that includes at least the number of execution nodes. The subject technology sends the task to the virtual warehouse for execution. The subject technology receives an indication that execution of the task is complete. Further, the subject technology provides an indication that the virtual warehouse is idle.
Sharing data in a data exchange across multiple cloud computing platforms and/or cloud computing platform regions is described. An example computer-implemented method can include receiving data sharing information from a data provider for sharing a data set in a data exchange from a first cloud computing entity to a set of second cloud computing entities. In response to receiving the data sharing information, the method may also include creating an account with each of the set of second cloud computing entities. The method may also further include sharing the data set from the first cloud computing entity with the set of second cloud computing entities using at least the corresponding account of that second cloud computing entity.
Using container-centric managed access, an administrator is enabled to define a set of future grants for each object that will be created in the future in a container managed by the administrator. When a user creates a database object, the system checks the future grants to determine if any apply to the user, the database object, or the combination. Any applicable future grants are applied to the database object before the user is allowed to modify it. As a result, the administrator is enabled to control the privileges associated with the database object even before the database object is created, while restricting individual object owners from managing privileges on their owned objects.
G06F 21/62 - Protecting access to data via a platform, e.g. using keys or access control rules
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
54.
PRUNING INDEXES TO ENHANCE DATABASE QUERY PROCESSING
A source table organized into a set of micro-partitions is accessed by a network-based data warehouse. A pruning index is generated based on the source table. The pruning index comprises a set of filters that indicate locations of distinct values in each column of the source table. A query directed at the source table is received at the network-based data warehouse. The query is processed using the pruning index. The processing of the query comprises pruning the set of micro-partitions of the source table to scan for data matching the query, the pruning of the plurality of micro-partitions comprising identifying, using the pruning index, a sub-set of micro-partitions to scan for the data matching the query.
A shared database platform can interface with a cluster computing platform over a network through a connector. The data transferred over the network can include metadata result packages that can be distributed to worker nodes of the cluster computing platform, which receive the metadata objects and access the result data for further processing on a staging platform, such as a scalable storage platform.
A shared database platform implements dynamic masking on data shared between users where specific data is masked, transformed, or otherwise modified based on preconfigured functions that are associated with user roles. The shared database platform can implement the masking at runtime dynamically in response to users requesting access to a database object that is associated with one or more masking policies.
G06F 16/22 - Indexing; Data structures therefor; Storage structures
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
G06F 16/25 - Integrating or interfacing systems involving database management systems
57.
EXTERNAL CREDENTIAL-LESS STAGES FOR DATABASE INTEGRATIONS
A storage integration object is created in a database of a data warehouse system. The storage integration object identifies a storage location in a storage platform of a cloud storage provider system and a cloud identity object maintained by a network-based data warehouse system. The cloud identity object is associated, at the storage platform of the cloud storage provider, with a proxy identity object granted permission to access the storage location. An external stage object is created based on the storage integration object. The external stage object identifies the storage location and includes an association with the storage integration object. A command to load or unload data at the storage location is received. In response to the command, the data is loaded or unloaded, via the proxy identity object, at the storage location using the external stage object.
A query referencing a function associated with a remote software component is received by a network-based data warehouse system. Temporary security credentials corresponding to a role at a cloud computing service platform are obtained. The role has permission to send calls to a web endpoint corresponding to the remote software component. A request comprising input data and electronically signed using the temporary security credentials is sent to a web Application Programming Interface (API) management system of the cloud computing service platform. The request, when received by the web API management system, causes the web API management system to invoke external functionality provided by the remote software component at the web endpoint with respect to the input data. A response comprising a result of invoking the external functionality is received from the web API management system, and the result data is processed according to the query.
The subject technology receives a query plan, the query plan comprising a set of query operations, the set of query operations including at least one aggregation and at least one join operation. The subject technology analyzes the query plan to identify an aggregation that is redundant. The subject technology removes the aggregation based at least in part on the analyzing. The subject technology determines at least one aggregation property corresponding to at least one query operation of the query plan. The subject technology inserts at least one adaptive aggregation operator in the query plan based at least in part on the at least one aggregation property. The subject technology provides a modified query plan based at least in part on the inserted at least one adaptive aggregation operator in the query plan.
Providing a private data exchange is described. An example computer-implemented method can include providing a data exchange by a cloud computing service on behalf of an entity. The data exchange may comprise several data listings provided by one or more data providers. The data listings reference one or more data sets stored in a data storage platform associated with the cloud computing service. The method may also include designating a data exchange administrator account of the data exchange. The data exchange administrator account may be associated with the entity and may be capable of: granting and denying requests from data consumers to access the data exchange; and granting and denying requests from data providers to publish data listings on the data exchange.
Systems, methods, and devices for automatically retrying a query. A method includes receiving a query directed to database data and assigning execution of the query to one or more execution nodes of a database platform. The method includes determining that execution of the query was unsuccessful. The method includes assigning a first retry execution of the query on the first version of the database platform and assigning a second retry execution of the query on a second version of the database platform.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Systems, methods, and devices for sharing materialized views in multiple tenant database systems. A method includes defining a materialized view over a source table that is associated with a first account of a multiple tenant database. The method includes defining cross-account access rights to the materialized view to a second account such that that second account can read the materialized view without copying the materialized view. The method includes modifying the source table for the materialized view. The method includes identifying whether the materialized view is stale with respect to the source table by merging the materialized view and the source table.
A data warehouse storing databases for a plurality of users, including service providers hosting data for other users of the data warehouse may implement a data exchange. A provider of data creates a listing in the exchange including metadata. A consumer browses the metadata and select a listing to add to consumed shares of the consumer. Browsing may be filtered to implement a private exchange. Adding of data may be limited by access controls requiring verification of the consumer's identity. The added data may be filtered to include only data that relates to the consumer. Sharing may be bi-directional such that a request to add a listing for first data of a provider may also reference second data of the consumer such that the first data is added to the consumed shares of the consumer and the second data is added to the consumed shares of the provider.
Systems, methods, and devices for generating and updating cross-account materialized views in multiple tenant database systems. A method includes defining a share object in a first account wherein the share object includes data associated with the first account. The method includes granting cross-account access rights to the share object to a second account such that the second account has access to the share object without copying the share object. The method includes generating a materialized view over the share object. The method includes updating the data associated with the first account. The method includes identifying whether the materialized view is stale with respect to the share object by merging the materialized view and the share object.
Systems, methods, and devices for storing database data in journal tables comprising a snapshot and a log table. A method includes defining a journal table comprising a snapshot and a log table, the snapshot comprising an up-to-date representation of data in the journal table at a point in time. The method includes assigning a timestamp to the snapshot indicating when the snapshot was generated. The method includes receiving a request to execute a transaction on the journal table to modify the data in the journal table, the transaction comprising one or more of an insert, a delete, an update, or a merge. The method includes inserting a new row into the log table in lieu of executing the transaction on the snapshot of the journal table, the new row comprising an indication of a change requested to be made to the journal table based on the transaction.
G06F 7/00 - Methods or arrangements for processing data by operating upon the order or content of the data handled
G06F 15/16 - Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
Systems, methods, and devices for querying over an external table are disclosed. A method includes connecting a database platform to an external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes receiving a query comprising a predicate, the query directed at least to data in the external table. The method includes determining, based on metadata, one or more partitions in the external table comprising data satisfying the predicate. The method includes pruning, based on the metadata, all partitions in the external table that do not comprise any data satisfying the predicate. The method includes generating a query plan comprising a plurality of discrete subtasks. The method includes assigning, based on the metadata, the plurality of discrete subtasks to one or more nodes in an execution platform.
Systems, methods, and devices for automated maintenance of external data tables are disclosed. An external table is defined by a database platform based on content stored in an external data storage platform that is separate from the database platform and to which the database platform does not have write access. Metadata is generated for the external table. The metadata is stored on a shared storage platform that is associated with the database platform and to which the database platform has write access. One or more materialized views are generated over the external table and stored in the shared storage platform. Notifications are received of modifications made to the content. The metadata for the external table is refreshed in response to the notifications. The one or more materialized views are refreshed continuously in the background to reflect a current state of the external table.,
Systems, methods, and devices for transitioning a client connection for a database from a first deployment of the database to a second deployment of the database. A method includes maintaining the client connection between a client and the first deployment such that database traffic occurs at the first deployment. The method includes generating a unique identification that references a first connection object of the first deployment and a second connection object of a second deployment. The method includes receiving a notification that the first deployment is unavailable and providing an external connection group unique identification to the second deployment for the second deployment to determine whether the client should be connected to the second deployment. The method includes receiving a uniform resource locator from the second deployment if the client should be connected to the second deployment.
G06F 15/16 - Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
G06F 16/00 - Information retrieval; Database structures therefor; File system structures therefor
Systems, methods, and devices for multiple level metadata organization in database systems. A system includes a plurality of shared storage devices collectively storing database data across a plurality of tables that each comprise one or more micro-partitions constituting immutable storage devices. The system includes a resource manager configured to manage the storage of database data stored in a table across one or more of the plurality of shared storage devices. The resource manager is configured to manage the storage of metadata for the table. The metadata includes a column expression property with information about data stored in a column, a micro-partition expression property with information about data stored in a micro-partition, a grouping expression property with information about data stored in two or more micro-partitions, and cumulative table metadata with global information about all micro-partitions of the table.
Systems, methods, and devices for generating a transactional change tracking summary for a database. A method includes executing a transaction on a table of a database, wherein the table includes a micro-partition and the transaction is executed on the micro-partition. The method includes, in response to the transaction being fully executed, generating a change tracking entry comprising an indication of one or modifications made to the table by the transaction. The method includes storing the change tracking entry in a change tracking stream, wherein the change tracking stream includes one or more sequential change tracking entries that each correspond to a different transaction that is fully executed on the table.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Systems, methods, and devices for executing a task on database data in response to a trigger event are disclosed. A method includes executing a transaction on a table comprising database data, wherein executing the transaction comprises generating a new table version. The method includes, in response to the transaction being fully executed, generating a change tracking entry comprising an indication of one or more modifications made to the table by the transaction and storing the change tracking entry in a change tracking stream. The method includes executing a task on the new table version in response to a trigger event.
Systems, methods, and devices for tracking a series of changes to database data are disclosed. A method includes executing a transaction to modify data in a micro-partition of a table of a database by generating a new micro-partition that embodies the transaction. The method includes associating transaction data with the new micro-partition, wherein the transaction data comprises a timestamp when the transaction was fully executed, and further includes associating modification data with the new micro-partition that comprises an indication of one or more rows of the table that were modified by the transaction. The method includes joining the transaction data with the modification data to generate joined data and querying the joined data to determine a listing of intermediate modifications made to the table between a first timestamp and a second timestamp.
G06F 7/00 - Methods or arrangements for processing data by operating upon the order or content of the data handled
G06F 11/34 - Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation
G06F 15/16 - Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
Systems, methods, and devices for tracking changes to database data. A method includes determining a change to be executed on a micro-partition of a table of a database and executing the change on the table by generating a new micro-partition that embodies the change. The method includes updating a table history that includes a log of changes made to the table, wherein each change in the log of changes includes a timestamp, and wherein updating the table history includes inserting the change into the log of changes.
Systems, methods, and devices for incrementally refreshing a materialized view are disclosed. A method includes generating a materialized view based on a source table. The method includes merging the source table and the materialized view to generate a merged table to identify whether an update has been executed on the source table that is not reflected in the materialized view. The method includes, in response to detecting an update made to the source table that is not reflected in the materialized view, applying the update to the materialized view.
Systems, methods, and devices for implementing secure views for zero-copy data sharing in a multi-tenant database system are disclosed. A method includes generating a share object in a first account comprising a share role. The method includes associating view privileges for the share object such that an underlying detail of the share object comprises a secure user-defined function definition. The method includes granting, to a second account, cross-account access rights to the share role or share object in the first account. The method includes receiving a request from the second account to access data or services of the first account and providing a response to the second account based on the data or services of the first account. The method is such that the underlying detail of the share object that comprises the secure user-defined function definition is hidden from the second account and visible to the first account.
Systems, methods, and devices for implementing secure views for zero-copy data sharing in a multi-tenant database system are disclosed. A method includes generating a share object in a first account comprising a share role. The method includes associating view privileges for the share object such that an underlying detail of the share object comprises a secure view definition. The method includes granting, to a second account, cross-account access rights to the share role or share object in the first account. The method includes receiving a request from the second account to access data or services of the first account and providing a response to the second account based on the data or services of the first account. The method is such that the underlying detail of the share object that comprises the secure view definition is hidden from the second account and visible to the first account.
Automatic clustering of a database table is disclosed. A method for automatic clustering of a database table includes receiving an indication that a data modification task has been executed on a table and determining whether the table is sufficiently clustered. The method includes, in response to determining the table is not sufficiently clustered, selecting one or more micro-partitions of the table to be reclustered. The method includes assigning each of the one or more micro-partitions to an execution node to be reclustered.
Replication and failover of database data is disclosed. A method includes replicating database data stored in a primary deployment such that the database data is further stored in a secondary deployment. The method includes executing one or more updates to the database data at the secondary deployment when the primary deployment is unavailable and propagating the one or more updates to the primary deployment when the primary deployment becomes available again. The method includes executing queries on the database data at the primary deployment when the primary deployment is available.
Systems, methods, and devices, for managing data skew during a join operation are disclosed. A method includes computing a hash value for a join operation and detecting data skew on a probe side of the join operation at a runtime of the join operation using a lightweight sketch data structure. The method includes identifying a frequent probe-side join key on the probe side of the join operation during a probe phase of the join operation. The method includes identifying a frequent build-side row having a build-side join key corresponding with the frequent probe-side join key. The method includes asynchronously distributing the frequent build-side row to one or more remote servers.
patents
