Sharing data in a data exchange across multiple cloud computing platforms and/or cloud computing platform regions is described. An example computer-implemented method can include creating a listing in a data exchange, the listing including a data set hosted by a first cloud computing entity. The data set can be shared with a second cloud computing entity. The method further includes receiving a request associated with a customer account of the second cloud computing entity to access the data set of the listing hosted by the first cloud computing entity and replicating at least a subset of the data set of the listing from the first cloud computing entity to a provider account at the second cloud computing entity to be accessible by the customer account at the second cloud computing entity.
H04L 41/50 - Network service management, e.g. ensuring proper service fulfilment according to agreements
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
H04L 67/51 - Discovery or management thereof, e.g. service location protocol [SLP] or web services
H04L 67/53 - Network services using third party service providers
2.
REAL-TIME STREAMING DATA INGESTION INTO DATABASE TABLES
A streaming ingest platform can improve latency and expense issues related to uploading data into a cloud data system. The streaming ingest platform can organize the data to be ingested into per-table chunks and per-account blobs. This data may be committed and may be made available for query processing before it is ingested into the target source tables. This significantly improves latency issues. The streaming ingest platform can also accommodate uploading data from various sources with different processing and communication capabilities, such as Internet of Things (IOT) devices.
Systems and methods for inferring a schema for a text file are provided. The systems and methods perform operations including: accessing a file comprising a plurality of textual records, each textual record of the plurality of textual records being associated with one or more columns of data; sampling a set of textual records from the plurality of textural records; obtaining a hierarchy comprising a plurality of levels of schema types; determining whether an individual column of the one or more columns of data corresponding to the set of textual records is successfully associated with a first level of the plurality of levels of the schema types and, in response, associating a schema type represented by the first level with the individual column of the one or more columns of data corresponding to the plurality of textual records.
A method of implementing object tagging framework starts with the processor receiving a tag creation command including a tag name. In response to the tag creation command, the processor creates a current tag. The processor then receives an association command, the tag name and a source object identifier. The processor determines a source object associated with the source object identifier. The source object includes a tag value. The processor associates the current tag with the source object. The processor receives a replication command including the source object and a target object. The processor causes replication of the source object to the target object that comprises replicating the current tag with the tag name and the tag value in the source object to the target object. Other embodiments are also described herein.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
G06F 16/28 - Databases characterised by their database models, e.g. relational or object models
5.
PREDICTIVE RESOURCE ALLOCATION FOR DISTRIBUTED QUERY EXECUTION
The subject technology receives a query directed to a set of source tables, each source table organized into a set of micro-partitions. The subject technology determines a set of metadata, the set of metadata comprising table metadata, query metadata, and historical data related to the query. The subject technology predicts, using a machine learning model, an indicator of an amount of computing resources for executing the query based at least in part on the set of metadata. The subject technology generates a query plan for executing the query based at least in part on the predicted indicator of the amount of computing resources. The subject technology executes the query based at least in part on the query plan.
Embodiments of the present disclosure describe systems, methods, and computer program products for redacting sensitive data within a database. An example method can include receiving a masking policy for a column of a database, the masking policy identifying a category of sensitive data, examining a column of a database to identify a category of sensitive data in a first location of the column, and, in response to a data query accessing the column, the first location of the column exceeding a threshold probability of comprising sensitive data, executing a redaction operation to redact the category of sensitive data from the first location of the column to generate redacted data for a response to the data query.
A method for identity resolution and data enrichment is performed by at least one hardware processor and includes detecting at an account of a data provider, a shared data object that is shared by an account of a data consumer with the account of the data provider. An application executing at the account of the data consumer is enabled for an identity resolution process based on the detecting of the shared data object. A request for source data received from the application is detected at the account of the data provider. The source data is managed by the account of the data provider. The source data is communicated to the application executing at the account of the data consumer, based on a verification that the application is enabled for the identity resolution process. The identity resolution process is performed at the account of the data consumer using the source data.
A system for key-rotation in a data platform. A computing system receives key-rotator application configuration instructions and generates a key-rotator application based on the key key-rotator application configuration instructions. A key-rotation process using the key-rotation application is initiated based on a predetermined schedule where the key-rotation process includes accessing a data platform based on an account identification of the key-rotator application configuration instructions, determining a private key and public key, storing the private key and the public key in a datastore, assigning the public key to a user profile of the data platform where the user profile included in the key-rotator application configuration instructions, updating one or more data platform services with the private key, and deleting a prior public key from the user profile.
A data dictionary generation system automatically populates and updates a data dictionary for listings offering shared data. A data dictionary includes metadata describing the shared data, including the individual objects, such as the individual tables, schemas, views, and functions. The shared data and each individual data object may be described in the data dictionary by a set of data fields that corresponds to the shared dataset or the object type of the individual object. The data dictionary can be presented to data consumers along with the description of the listing to provide data consumers with a comprehensive description of the shared data provided by a listing, including a high-level summary of the shared data and description of each individual object included in the shared data. The data dictionary allows data consumers to understand the contents of the shared data and how to use the shared data.
A data dictionary generation system utilizes a background service that is programmed to automatically populate and update a data dictionary for listings offering shared data. A data dictionary includes metadata describing the shared data overall as well as the individual objects included in the listing, such as the individual tables, schemas, views, and functions. To generate the data dictionary, the data dictionary generation system analyzes the shared data to identify objects, identifies a set of data fields associated with each identified object and populates the set of data fields associated with each identified object based on the shared data offered by the listing. To ensure that a data dictionary for each listing remains up to date, the data dictionary generation system periodically scans the listings to identify any changes to share access granted to the listings.
A data dictionary generation system automatically populates and updates a data dictionary for listings offering shared data. The data listing distribution component distributes the data dictionaries to various remote deployments in a data exchange by using a global messaging framework and replication method. For example, the data listing distribution component replicates a data dictionary generated for the listing and its shared data from a source deployment to one or more destination deployments associated with various geographic regions. The data listing distribution component distributes the listing to the various remote deployments to allow for the listing, including its shared data and data dictionary, to be accessed by users within the geographic region associated with the remote deployment.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
G06Q 30/0201 - Market modelling; Market analysis; Collecting market data
12.
SYSTEMS AND METHODS FOR EFFICIENTLY QUERYING EXTERNAL TABLES
Disclosed herein are systems and methods for efficiently querying external tables. In an embodiment, a database platform receives a query that is directed at least in part to external data in an external table stored on a data storage platform that is external to the database platform. The external table includes a plurality of partitions. The database platform identifies, from external-table metadata, a subset of the plurality of partitions of the external table as including data that potentially satisfies the query. The external-table metadata is stored by the database platform. The database platform identifies data that satisfies the query by scanning the identified subset of the partitions, and responds to the query at least in part with the identified data that satisfies the query.
A method, apparatus, and system for join operations of a plurality of relations that are distributed over a plurality of storage locations over a network of computing components.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
H04L 67/1095 - Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
H04L 67/568 - Storing data temporarily at an intermediate stage, e.g. caching
14.
EVENT DRIVEN TECHNIQUE FOR CONSTRUCTING TRANSACTION LOCK WAIT HISTORY
Techniques for constructing transaction lock wait history showing blocker queries are described. A first transaction referencing a resource saved in a network-based data warehouse is received where the first transaction being blocked due to second transaction accessing the resource. A first telemetry event based on the first transaction being blocked is transmitted. After acquiring lock ownership of the resource by the first transaction, a second telemetry event based on acquiring lock ownership by the first transaction is transmitted.
The subject technology generates a nested object based on a set of metadata, the set of metadata including information linking the nested object to a table object associated with a base table. The subject technology generates a second table object associated with the nested object, the second table object representing a secondary index of the base table, the second table object including information linking the second table object to the nested object. The subject technology generates a second nested object based on a particular set of metadata, the particular set of metadata including information linking the second nested object to the table object. The subject technology generates a third table object associated with the second nested object, the third table object representing a particular secondary index of the base table, the third table object including information linking the third table object to the second nested object.
The subject technology determines a derived encryption key using a cryptographic hash function applied to a hybrid tenant master encryption key and a local random generated identifier. The subject technology encrypts a record value and a key value associated with a transaction using the derived encryption key. The subject technology determines a non-leaf node using a tenant prefix of a tenant. The subject technology inserts the encrypted record value at a leaf node below a non-leaf node of a tree structure associated with the tenant. The subject technology receives a second transaction for performing a read operation on a distributed database. The subject technology retrieves a set of encryption keys based at least in part on an account and the tenant. The subject technology decrypts, using the set of encryption keys, data from the distributed database. The subject technology provides the decrypted data as a result of the second transaction.
G06F 16/22 - Indexing; Data structures therefor; Storage structures
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
H04L 9/06 - Arrangements for secret or secure communications; Network security protocols the encryption apparatus using shift registers or memories for blockwise coding, e.g. D.E.S. systems
H04L 9/14 - Arrangements for secret or secure communications; Network security protocols using a plurality of keys or algorithms
H04L 9/32 - Arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system
17.
SYSTEMS AND METHODS FOR ATTACHABLE-AND-DETACHABLE DATABASE SESSIONS
In an embodiment, a database platform maintains a first account and a second account, where the second account has stored therein an attachable-and-detachable database session. The database platform receives, from a second-account user in the second account, a request to grant, to a first-account user in the first account, access to the attachable-and-detachable database session, and responsively grants the requested access. The database platform receives, from the first-account user, an attachment request requesting that the first-account user attach to the attachable-and-detachable database session, and responsively sets the attachable-and-detachable database session as a current database session for the first-account user. The database platform executes at least one command received from the first-account user with respect to the attachable-and-detachable database session.
A differentially private security system communicatively coupled to a database storing restricted data receives a database query from a client. The database query includes an operation, a target accuracy, and a maximum privacy spend for the query. The system performs the operation to produce a result, then injects the result with noise sampled from a Laplace distribution to produce a differentially private result. The system iteratively calibrates the noise value of the differentially private result using a secondary distribution different from the Laplace distribution and a new fractional privacy spend. The system ceases to iterate when an iteration uses the maximum privacy spend or a relative error of the differentially private result is determined to satisfy the target accuracy, or both. The system sends the differentially private result to the client.
A method includes detecting, by at least one hardware processor, a change request for a table of a database, the table comprising a plurality of micro-partitions. A transaction associated with the change request is executed at a first timestamp. The transaction causes replacement of a first micro-partition of the plurality of micro-partitions with a second micro-partition. A change tracking column is generated in the second micro-partition. The change tracking column comprises metadata for the transaction. A delta for the table between the first timestamp and a second timestamp is generated using the metadata in the change tracking column. The delta indicates changes made to one or more rows of the table between the first time stamp and the second timestamp.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Row-level security (RLS) may provide fine-grained access control based on flexible, user-defined access policies to databases, tables, objects, and other data structures. A RLS policy may be an entity or object that defines rules for row access. A RLS policy may be decoupled or independent from any specific table. This allows more robust and flexible control. A RLS policy may then be attached to one or more tables. The RLS policy may include a Boolean-valued expression.
A process of fulfilling a database deployment request for a data platform. A compute service manager of the data platform scans one or more accounts of a consumer region of the data platform for a pending listing fulfillment request, where the pending listing request includes a request for deployment of a consumer database and an associated share of grant metadata of the consumer database within the consumer region. When the compute service manager determines that an account of the one or more accounts has a pending listing fulfillment request, the compute service manager determines a listing for the pending listing fulfillment request based on listing data of the account. The compute service manager determines that no other fulfillment task is scheduled for the pending listing fulfillment request and schedules a background fulfillment task to perform the fulfillment process for the pending listing fulfillment request.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Systems and methods for generating transient object references are provided. The systems and methods perform operations including establishing a session between a first entity and a second entity. The operations include identifying an object that the first entity is authorized to access according to a first set of access privileges. The operations include generating a reference associated with the object. The operations include temporarily authorizing the second entity to access the object using the reference according to a second set of access privileges, the second set of access privileges being derived from the first set of access privileges.
Systems, methods, and devices for retrying a query. A method includes receiving a query directed to database data and assigning execution of the query to one or more execution nodes of an execution platform, the one or more execution nodes configured to execute the query on a first version of a database platform. The method includes determining that execution of the query was unsuccessful. The method includes assigning a first retry execution of the query to the one or more execution nodes of the execution platform and determining whether a regression or an intermittent fault caused the execution of the query to be unsuccessful based at least in part on whether the first retry execution of the query was successful or unsuccessful.
Disclosed are techniques for deduplicating files to be ingested by a database. A bloom filter may be built for each of a first set of files to be ingested into a data exchange to generate a set of bloom filters, wherein each of the set of bloom filters is built with a number of hash functions that is based on a desired false positive rate. The set of bloom filters may be stored in the metadata storage of the data exchange. In response to receiving a set of candidate files to be ingested, identifying using the set of bloom filters, candidate files from the set of candidate files that are duplicative of a file in the first set of files and pruning from the set of candidate files, each candidate file identified as being duplicative of a file in the first set of files using the set of bloom filters.
Example systems and methods for cloning catalog objects are described. In one implementation, a method includes creating a second catalog object by mapping a second portion of second metadata of the second catalog object to same data of a same data file as a first portion of first metadata of a first catalog object, and, in response to a data storage or data retrieval request directed to the second catalog object, deleting data associated with the second portion of the second metadata from the second catalog object independently of the first catalog object.
G06F 16/00 - Information retrieval; Database structures therefor; File system structures therefor
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
26.
Two-way data sharing between private and public clouds
Methods, systems, and computer programs are presented for enabling automated secure data sharing from a private cloud region to a public cloud region and vice versa. A cloud data platform confirms a relationship establishment procedure between a provider and a consumer is recorded with a cloud data platform, the provider being associated with a private cloud deployment and the consumer being associated with a public cloud deployment in a public region. The cloud data platform enables disabling of a firewall policy that is preventing data traffic between the private cloud deployment and the public cloud deployment and enables data sharing between the private cloud deployment and the public cloud deployment. The cloud data platform enables data sharing in a database of the cloud data platform.
Methods, systems, and computer programs are presented for enabling any sandboxed user-defined function code to securely access the Internet via a cloud data platform. A remote procedure call is received by a cloud data platform from a user-defined function (UDF) executing within a sandbox process. The UDF includes code related to at least one operation to be performed. The cloud data platform provides an overlay network to establish a secure egress path for UDF external access. The cloud data platform enables the UDF executing in the sandbox process to initiate a network call.
Embodiments of the present disclosure may provide dynamic and fair assignment techniques for allocating resources on a demand basis. Assignment control may be separated into at least two components: a local component and a global component. Each component may have an active dialog with each other; the dialog may include two aspects: 1) a demand for computing resources, and 2) a total allowed number of computing resources. The global component may allocate resources from a pool of resources to different local components, and the local components in turn may assign their allocated resources to local competing requests. The allocation may also be throttled or limited at various levels.
A system for improving task scheduling on a cloud data platform is provided. A task to be executed using resources of a computing cluster is received. A task execution plan is generated and information about data to be used for the ask is accessed. Resource requirements for executing the task are predicted by applying machine learning to the task execution plan and the information about the data. Assignment data is generated to execute the task on the resources by applying machine learning information about a current state of the resources and predicted resource requirements.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
30.
COLUMN DATA ANONYMIZATION BASED ON PRIVACY CATEGORY CLASSIFICATION
An approach is disclosed herein that retrieves data from a data set that includes first column data comprising a first data type and a second data type. The approach structures the first column data into second column data and third column data based on the first data type and the second data type. The approach determines a first semantic category and a second semantic category for the first data type and the second data type, and then determines a first privacy category and a second privacy category based on the first semantic category and the second semantic category. The approach anonymizes the second column data and the third column data to produce anonymized data based on the first privacy category and the second privacy category, respectively. In turn, the approach generates an anonymized view of the data set using the anonymized data.
An example method of tuning a machine learning operation can include receiving a data query comprising a reference to an input data set of a database, generating a plurality of hyperparameter sets based on the input data set, in response to receiving the data query, training a plurality of machine learning models using the plurality of hyperpararneter sets, selecting a first mathine learning model of the plurality of machine learning models based on an accuracy of an output of the first machine learning model, and in response to receiving the data query, returning the output of the first machine learning model.
Systems and methods for generating a machine-learning (ML) model for extracting information from one or more electronic documents, where the ML model can be used as a data object, which can be part of a database command or as part of a document information extraction process that is continuously running (e.g., document information extraction pipeline).
Provided herein are systems and methods for global data objects on a data platform where the global data objects are accessible at an organization level. In particular, an organization-level global data object provided by various embodiments can be used as a generic organization object that is owned by a specific organization, and can be managed (e.g., created, deleted, or modified) by use of a leader-based model.
Techniques for creating and using error tables to track errors associated with a base table are described. A command to perform an operation on a base table stored in a network-based data system can be received and executed, causing at least one error. At least one error record corresponding to the at least one error can be inputted into an error table, which is nested with the base table. Contextual information can be added to the at least one error record.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/22 - Indexing; Data structures therefor; Storage structures
35.
Inexact timestamp range matching join for time series data (AS OF JOIN)
A method includes parsing a query to determine a plurality of data processing operations associated with the query and including an AS OF JOIN operation between first time series data in a first table and second time series data in a second table. A query plan of the query is generated. The query plan includes a plurality of nodes corresponding to the plurality of data processing operations. At least one of the plurality of nodes corresponding to the AS OF JOIN operation is modified to generate a modified query plan of the query. The modifying is based on applying a UNION operation on at least a first portion of column data in the first table and the second table to obtain a combined table. Execution of the query by at least one of a plurality of computing nodes is scheduled based on the modified query plan.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
A restricted view definition is received by a database system. The restricted view definition defines a view over a database table with one or more restrictions on use of the view. The view over the database table is generated based on the restricted view definition. A query directed at the view is received by the database system. The database system determines whether the query directed at the view is permitted based on the one or more restrictions on the use of the view.
The subject technology receives, at a first execution node, a first transaction, the first transaction to be executed on linearizable storage. The subject technology determines whether the first execution node corresponds to a rank indicating a leader worker. The subject technology, in response to the first execution node corresponding to the rank indicating the leader worker, performs, by the first execution node, an initialization process for executing the first transaction. The subject technology broadcasts a first read timestamp associated with the first transaction to a set of execution nodes, the set of execution nodes being different than the first execution node. The subject technology executes, by the first execution node, at least a first operation from the first transaction.
The subject technology receives first party training data provided by an end-user of a baseline machine learning model. The subject technology determines a first set of common features based on the first party training data. The subject technology receives, from at least one data source. The subject technology determines a second set of common features based on the set of datasets. The subject technology trains, using the first set of common features and the second set of common features, a second machine learning model, the second machine learning model incorporating additional training data from the external data supplier during training compared to the baseline machine learning model. The subject technology generates a boosted machine learning model based at least in part on the training, the boosted machine learning model comprising the trained second machine learning model.
Disclosed is an execution information sharing system that writes execution information to a provider target (and other targets) in a secure manner. Execution information generated by an application may be written to a consumer stage, wherein the application is shared by a provider account of a data exchange with a consumer account that executes the application. A consumer exchange service (ES) of the data exchange may send a request to a copy service of the data exchange to copy the execution information from the consumer stage to the provider stage, wherein the consumer ES is a part of the data exchange and is protected from actions of the consumer account. A copy operation may be executed to copy the execution information from the consumer stage to the provider stage using the copy service of the data exchange. The execution information is ingested from the provider stage to a provider table.
Using container-centric managed access, an administrator is enabled to define a set of future grants for each object that will be created in the future in a container managed by the administrator. When a user creates a database object, the system checks the future grants to determine if any apply to the user, the database object, or the combination. Any applicable future grants are applied to the database object before the user is allowed to modify it. As a result, the administrator is enabled to control the privileges associated with the database object even before the database object is created, while restricting individual object owners from managing privileges on their owned objects.
G06F 21/62 - Protecting access to data via a platform, e.g. using keys or access control rules
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Embodiments of the present disclosure may provide a streamlined process for performing operations, such as data sharing and data replication, using multiple accounts. A global identity (also referred to as an organization user) may be employed, where the global identity may have access to multiple accounts across the same or different deployments. The global identity may switch between accounts from its login session and perform various tasks in the context of different accounts without undergoing further authentication.
H04L 67/02 - Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
H04L 41/50 - Network service management, e.g. ensuring proper service fulfilment according to agreements
H04L 41/5041 - Network service management, e.g. ensuring proper service fulfilment according to agreements characterised by the time relationship between creation and deployment of a service
H04L 67/10 - Protocols in which an application is distributed across nodes in the network
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
H04L 67/59 - Providing operational support to end devices by off-loading in the network or by emulation, e.g. when they are unavailable
H04L 67/60 - Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
42.
SHARING EVENTS AND OTHER METRICS IN NATIVE APPLICATIONS
Disclosed is an execution information sharing system that duplicates execution information to a provider target (and other targets) as it is being loaded to a consumer target. A first log information object and a second log information object are generated. The first and second log information objects comprise information indicating a consumer target and information indicating a provider target respectively where execution information generated by an application shared with a consumer account of a data exchange is written. A first event unloader and a second event unloader are generated based on the first and second log information objects respectively, wherein the first and second event unloaders are both linked to the application using a mapping. In response to receiving execution information from the application, the execution information is forwarded to the consumer target and the provider target using the first event unloader and the second event unloader respectively.
Techniques described herein can optimize usage of computing resources in a data system. Dynamic throttling can be performed locally on a computing resource in the foreground and autoscaling can be performed in a centralized fashion in the background. Dynamic throttling can lower the load without overshooting while minimizing oscillation and reducing the throttle quickly. Autoscaling may involve scaling in or out the number of computing resources in a cluster as well as scaling up or down the type of computing resources to handle different types of situations.
A database system can configure network devices, such as a primary database in a multi-tenant deployment and a secondary database in a private deployment, to send and receive sequence messages, such as input data indicative of a selection of a link. The database system can create a secure share area in the private deployment in response to receiving the input data indicative of the selection of the link. The database system can replicate the data from the multi-tenant deployment to the secure share area in the private deployment and share the replicated data from the secure share area to the secondary database hosted in the private deployment.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Provided herein are systems and methods for configuring trace events. A system includes at least one hardware processor coupled to a memory and configured to instantiate a user code runtime to execute user-defined function (UDF) code. The user code runtime is instantiated within a sandbox process of an execution node. An application programming interface (API) call is detected during execution of the UDF code. The API call includes one or more configurations of a trace event. Telemetry information is collected based on the one or more configurations. The telemetry information is associated with the trace event using a telemetry API. The telemetry API corresponds to the API call. The telemetry information is formatted using the telemetry API, to generate structured telemetry information. The at least one hardware processor causes ingestion of the structured telemetry information into an event table.
Techniques for identity resolution and data enrichment include configuring, during an onboarding process at an account of a data provider, at least one parameter associated with access to identity resolution functions by an account of a data consumer. A first shared data object is generated at the account of the data provider. The first shared data object corresponds to a second shared data object at the account of the data consumer. The second shared data object at the account of the data consumer is enabled for sharing of log data associated with an application executing at the account of the data consumer. The application is enabled for an identity resolution process based on the detecting of the second shared data object. Source data associated with the identity resolution functions is encoded for communication to the application at the account of the data consumer based on the enabling.
Systems and methods for an organization-level account for an organization on a data platform, users of which can possess administrative or management privileges with respect to the organization and across one or more others accounts of the organization.
A method of implementing sub-table replication starts with the processor detecting an update to an entitlements table. The processor performs filtering of a data table based on the update to the entitlements table. The data table including an entitlements column. The processor detects an update to the entitlements column and performs incremental replication of the data table by causing a version-based replication to be executed. Other embodiments are also described herein.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
G06F 16/22 - Indexing; Data structures therefor; Storage structures
Systems and methods for managing membership in a private data exchange are provided herein. In one embodiment, the method includes receiving a request for access to a first listing of a data exchange, each listing of the data exchange comprising version metadata. The method further includes, in response to the request, accessing a first version of a data set referenced by the first listing, wherein the first version of the data set comprises a first structure defined by first version metadata, a second listing of the data exchange references a second version of the data set, the second version of the data set comprising a second structure defined by second version metadata, and the second structure is incompatible with the first structure.
H04L 41/22 - Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks comprising specially adapted graphical user interfaces [GUI]
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
H04L 51/212 - Monitoring or handling of messages using filtering or selective blocking
Provided herein are systems and methods to provide a way to share metrics regarding shared data access and accesses associated with data providers for different data listings of the data exchange. For example, the method may comprise detecting one or more client interactions with a set of data listings of a data exchange, the set of data listings associated with one or data providers. The method may further comprise collecting metrics corresponding to the one or more client interactions. In addition, the method may share metrics relevant to the one or more data providers with the one or more data providers.
H04L 67/1095 - Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
G06F 16/21 - Design, administration or maintenance of databases
H04L 67/12 - Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
A system may include a processing device and a memory storing instructions that, when executed by the processing device, causes the processing device to obtain a health check instruction that is specific to a name of a service that is associated with one or more endpoints, including performing a lookup with the name to obtain the health check instruction that is specific to the name. The processing device performs the one or more actions of the health check instruction to determine a health status of the one or more endpoints, and stores the health status of the one or more endpoints. In response to receiving a request to resolve the name from a client, the processing device returns the one or more endpoints based at least on the health status of the one or more endpoints.
H04L 43/0805 - Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
H04L 43/0817 - Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking functioning
H04L 67/133 - Protocols for remote procedure calls [RPC]
H04L 67/02 - Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
H04L 67/1036 - Load balancing of requests to servers for services different from user content provisioning, e.g. load balancing across domain name servers
H04L 61/4511 - Network directories; Name-to-address mapping using standardised directory access protocols using domain name system [DNS]
52.
Secure predicate derivation of queries using metadata
The subject technology provides embodiments for enabling derivation of predicates not only from other predicates but also from metadata such as expression properties. In examples, predicates are derived, avoiding unwanted impact on cardinality estimation. In other examples, predicates are derived, avoiding artificial runtime errors and providing a way to avoid security issues with secure views.
Embodiments of the present disclosure relate to sharing data using database roles. Database roles are generated within a database container of a provider account. Grants to a particular subset of the plurality of data objects of the database container may be assigned to each of the database roles, and each of the database roles are granted to a share object. The share object is mounted within a consumer account to generate an imported copy of each of the database roles. The imported copy of one or more of the database roles is granted to each of one or more account level roles of the consumer account. When a new object is added to a particular database role, it is immediately available for consumption by any account level roles to which the imported copy of the particular database role has been granted.
An in-database application package and application instance for a data platform. The data platform creates an application instance of an application package having a versioned schema, creates one or more system roles for the application instance, creates a user role and an administrator role for the application instance, creates one or more objects of the application instance based on a versioned schema, and grants one or more use privileges to the one or more roles. Application instances of the application package are upgraded or patched on the data platform based on application package versions. To ensure a proper upgrade or patch, the data platform tracks versions of executing objects of application instances in a call context.
Disclosed herein are systems and methods for query processing with restrictions in a database clean room. In an embodiment, a system receives a query directed to a combination of a first source dataset from a first database account of a distributed database and a second source dataset from a second database account of the distributed database. The system generates an approved statements table that contains database statement language that can be executed against the combination of the first and second source datasets. Based on determining that the approved statements table includes the query, the system executes the query to produce results data, and stores the results data in the first database account.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
G06F 21/62 - Protecting access to data via a platform, e.g. using keys or access control rules
Disclosed herein are methods and systems for secure data comparison using data clean rooms. In an embodiment, a computer system generates a replica database based on a provider database, which stores a cross reference table that cross references a client dataset of a client database and a provider dataset of the provider database. The system receives, at the replica database, a table that is generated by the client database using the cross-reference table. The system transmits, from the replica database, the table to the provider database. The system receives, at the replica database, a results dataset that is generated by the provider database by applying a database statement to the provider database using the table generated by the client database. The system shares, from the replica database, the results dataset with the client database.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
57.
SCHEMA EVOLUTION FOR KEY COLUMNAR DATA INTO ROW-ORGANIZED SEQUENCES
The subject technology generates, by a compute service manager, a schema hash value for a new schema version associated with a new schema version value, the schema hash value based on determining a sum of hash values of a set of attributes of value columns, the set of attributes comprises a column identifier, and a logical type of a column. The subject technology stores a mapping of the schema hash value to the new schema version value for a table in a metadata database. The subject technology stores a new schema entry based on the schema hash value, the new schema version value, and a new column for the table in the metadata database, the metadata database storing multiple entries for different schema versions, each entry including a particular schema hash value for mapping to a corresponding schema version from the different schema versions.
A data platform that implements memoizable functions for database objects. The data platform detects a first execution of a memoizable function and generates a first key based on metadata of one or more database objects operated on by the memoizable function and generates a first result for the memoizable function based on the one or more database objects. The data platform detects a second execution of the memoizable function and generates a second key based on the metadata of the one or more database objects operated on by the memoizable function. When the first key and the second key are equal, the data platform reuses the first result of the memoizable function. When the first key and second key do not match, the data platform generates a second result for the second execution of the memoizable function.
G06F 21/53 - Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity, buffer overflow or preventing unwanted data erasure by executing in a restricted environment, e.g. sandbox or secure virtual machine
Embodiments of the present disclosure may provide a dynamic query execution model. This query execution model may provide acceleration by scaling out parallel parts of a query (also referred to as a fragment) to additional computing resources, for example computing resources leased from a pool of computing resources. Execution of the parts of the query may be coordinated by a parent query coordinator, where the query originated, and a fragment query coordinator.
A top K query directed at a table is received. The table is organized into multiple storage units. The top K query comprises a first clause to sort a result set in order and a second clause that specifies a limit on a number of results provided in response to the query. A table scan operator identifies a first set of rows from the table based on a scan set determined for the table and provides the first set of rows to a top K operator. The top K operator determines a current boundary based on the first set of rows and provides the current boundary to the table scan operator. The table scan operator prunes the scan set based on the current boundary and identifies a second set of rows from the table based on the pruning.
A data platform for running a subset of cells in a notebook is provided. The data platform receives a run cells message from a notebook user interface (UI) application specifying the subset of cells to run. For each cell in the subset, the data platform runs the cell to generate a set of results, generates a cell execution stream using the results, stores the stream, and transmits the stream to the notebook UI application. The notebook UI application generates a display for the user using the cell execution stream. The data platform provides an efficient way to run specific cells in a notebook and display the results to the user.
Embodiments of the present disclosure may provide a data clean room allowing secure data analysis across multiple accounts, without the use of third parties. Each account may be associated with a different company or party. The data clean room may provide security functions to safeguard sensitive information. For example, the data clean room may restrict access to data in other accounts. The data clean room may also restrict which data may be used in the analysis and may restrict the output. The overlap data may be anonymized to prevent sensitive information from being revealed.
Systems, methods, and devices for incrementally refreshing a materialized view are disclosed. A method includes generating a materialized view based on a source table. The method includes merging the source table and the materialized view to generate a merged table to identify whether an update has been executed on the source table that is not reflected in the materialized view. The method includes, in response to detecting an update made to the source table that is not reflected in the materialized view, applying the update to the materialized view.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
G06F 16/22 - Indexing; Data structures therefor; Storage structures
64.
SERIALIZATION OF DATA IN A CONCURRENT TRANSACTION PROCESSING DISTRIBUTED DATABASE
The subject technology serializes, by at least one hardware processor, non-primary key data of column-organized data into compressed serialized value data that is in a row-organized sequence, the compressed serialized value data compressed using at least one bitmap, the non-primary key data comprising a schema identifier, the column-organized data being stored in a columnar database system, the column-organized data comprising primary key data and the non-primary key data. The subject technology stores the compressed serialized value data in a key-value data store of a key-value database system, the key-value database system processing key-value data in a key-value format. The subject technology receives a query by the columnar database system. The subject technology deserializes a portion of the compressed serialized value data that corresponds to the query. The subject technology processes the query using the columnar database system.
G06F 16/28 - Databases characterised by their database models, e.g. relational or object models
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/22 - Indexing; Data structures therefor; Storage structures
Example caching systems and methods are described. In one implementation, a method receives a query, at an execution platform, directed to data stored across a plurality of shared storage devices, the execution platform comprising one or more execution nodes, an execution node comprising a plurality of processors. The method processes the query using the one or more execution nodes of the execution platform, and in response to a determination of a change in a number of execution nodes of the execution platform, wherein the change is creating a new execution node, wherein a first subset of the plurality of processors comprises a minimal cache and a second subset of the plurality of processors comprises a cache providing faster input-output operations, reassigns processing of the query, among the changed number of execution nodes of the execution platform.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
G06F 9/50 - Allocation of resources, e.g. of the central processing unit [CPU]
G06F 16/14 - File systems; File servers - Details of searching files based on file metadata
G06F 16/21 - Design, administration or maintenance of databases
G06F 16/22 - Indexing; Data structures therefor; Storage structures
G06F 9/48 - Program initiating; Program switching, e.g. by interrupt
H04L 67/1095 - Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
66.
Differentially Private Processing and Database Storage
A hardware database privacy device is communicatively coupled to a private database system. The hardware database privacy device receives a request from a client device to perform a query of the private database system and identifies a level of differential privacy corresponding to the request. The identified level of differential privacy includes privacy parameters (ε,δ) indicating the degree of information released about the private database system. The hardware database privacy device identifies a set of operations to be performed on the set of data that corresponds to the requested query. After the set of data is accessed, the set of operations is modified based on the identified level of differential privacy such that a performance of the modified set of operations produces a result set that is (ε,δ)-differentially private.
A system to identify optimal cloud resources for executing workloads. The system deduplicates historical client queries based on a workload selection configuration to determine a grouping of historical client queries. The system generates a workload based on at least a portion of the grouping of historical client queries. The system repeatedly executes a test run of the workload using resources of a cloud environment to determine whether there is a performance difference in the test run. The system, in response to determining that there is no performance difference, identifies one or more sets of decreased resources of the cloud environment. The system re-executes the test run using the one or more sets of decreased resources of the cloud environment to determine whether there is a performance difference in the test run that is attributed to the one or more sets of decreased resources of the cloud environment.
An external function system can be implemented on a database to perform processing on one or more external network services. The external function system can comprise a particular external function for a particular external service, an outbound serializer function, and an inbound serializer function that are linked with the particular external function. The outbound serializer function can be configured to transform the data of a query from a database format to a different format of the particular external network service. The inbound deserializer function can be configured to receive data returned from the external service and transform the data to the format of the database.
G06F 16/25 - Integrating or interfacing systems involving database management systems
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Example resource management systems and methods are described. In one implementation, a resource manager is configured to manage data processing tasks associated with multiple data elements. An execution platform is coupled to the resource manager and includes multiple execution nodes configured to store data retrieved from multiple remote storage devices. Each execution node includes a cache and a processor, where the cache and processor are independent of the remote storage devices. A metadata manager is configured to access metadata associated with at least a portion of the multiple data elements.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
G06F 9/50 - Allocation of resources, e.g. of the central processing unit [CPU]
G06F 16/14 - File systems; File servers - Details of searching files based on file metadata
G06F 16/21 - Design, administration or maintenance of databases
G06F 16/22 - Indexing; Data structures therefor; Storage structures
G06F 9/48 - Program initiating; Program switching, e.g. by interrupt
H04L 67/1095 - Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
This disclosure provides methods and techniques of referencing row access policy (RAP) protected mapping tables in a RAP for a data table are disclosed herein. An example method of referencing a mapping table in a data table using nested RAP includes defining, by a processing device, a first access policy for the mapping table to control access by specific users or under specific conditions. The processing device further defines a second access policy attached to the data table referencing the mapping table. The processing device in response to a query, executes the second access policy of the data table to provide a response or operation of data associated with the data table and the mapping table. Executing the second access policy invokes executing the first access policy of the mapping table.
A data platform provides for encryption of secrets. During operation, an application of the data platform receives a secret and communicates the secret to an encryption client of the data platform. The encryption client generates an encrypted secret using a Data Encryption Key (DEK) and the secret. The encryption client communicates the DEK to an encryption server of the data platform while retaining the encrypted secret. The encryption server generates an encrypted DEK using a Transit Encryption Key TEK. The encryption server communicates the encrypted DEK to the encryption client and the encryption client generates a binary large object (blob) using the retained encrypted secret and the encrypted DEK. The application stores the blob on a data storage device.
A differentially private security system communicatively coupled to a database storing restricted data receives a database query from a client. The database query includes a relation specifying a set of data in the database upon which to perform the query and privacy parameters associated with the query. The differentially private security system determines a worst-case privacy spend for the query based on the privacy parameters and the relation. The differentially private security system performs the query upon the set of data specified by the relation and decrements the determined worst-case privacy spend from a privacy budget associated with the client. The differentially private security system records the worst-case privacy spend and the query at a log and determines a privacy budget refund based on queries recorded in the log. The differentially private security system applies the determined privacy budget refund to the privacy budget associated with the client.
A system for data object replication includes at least one hardware processor and at least one memory storing instructions. The instructions cause the at least one hardware processor to perform operations including parsing a replication request to obtain a data object and identification information identifying a set of databases at a first deployment of a data provider. A dependency of the data object to one or more additional data objects stored in the set of databases is detected. A sequential replication of the data object and the one or more additional data object from the first deployment to a second deployment of the data provider is performed. The second deployment is identified by the replication request. A sequence of the sequential replication is based on the detected dependency.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
G06F 16/185 - Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof
G06F 16/11 - File system administration, e.g. details of archiving or snapshots
G06F 16/13 - File access structures, e.g. distributed indices
74.
HEURISTIC SEARCH FOR K-ANONYMIZATION IN A GENERALIZATION LATTICE
An approach is disclosed that computes a path through a generalization lattice comprising a plurality of levels. For each of the levels, the approach uses a scoring function to compute one or more values from a node on a first level of the generalization lattice to one or more neighboring nodes on a second level of the generalization lattice. The approach then adds a best node from the neighboring nodes to the path based on the values. At the completion of computing scoring functions on the generalization lattice, the path comprises a best node from each of the plurality of levels. The approach then selects an optimal node from the best nodes in the path.
Techniques for identity resolution and data enrichment include configuring, at an account of a data consumer, an outbound share. The outbound share is designating a share at an account of a data provider as a receiving share. An identity resolution application is instantiated at the account of the data consumer. An instruction originating from the account of the data provider is decoded at the account of the data consumer. The instruction is generated based on the configuring of the outbound share. The instruction enables the identity resolution application for an identity resolution process. Source data is retrieved from the account of the data provider at the account of the data consumer. The source data is associated with the identity resolution process.
Systems, methods, and devices for sharing materialized views in multiple tenant database systems. A method includes defining a materialized view over a source table that is associated with a first account of a multiple tenant database. The method includes defining cross-account access rights to the materialized view to a second account such that that second account can read the materialized view without copying the materialized view. The method includes modifying the source table for the materialized view. The method includes identifying whether the materialized view is stale with respect to the source table by merging the materialized view and the source table.
This disclosure provides methods and techniques of data replication involving cross replication group (RG) references. Example methods, systems, and techniques are disclosed regarding batch database replication (e.g., backup) and failover (e.g., automatic transition to a backup) between multiple database deployments or database providers. For example, a system causes database data to be stored in a primary deployment and replicated in one or more secondary deployments. In the event that data in the primary deployment is unavailable, transactions may be executed on one or more of the secondary deployments. When the original primary deployment becomes available again, any transactions executed on secondary deployments may be propagated to the primary deployment. The system may be configured such that queries on the database data are executed on the primary deployment at any time when the primary deployment is available.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/25 - Integrating or interfacing systems involving database management systems
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
78.
QUERY PROCESSING OF STREAM OBJECTS USING STREAM EXPANSION
Provided herein are systems and methods for a stream object configuration, including query processing of stream objects using stream expansion. For example, a method includes decoding a query to obtain a first data processing operation and a first stream object. The first stream object is associated with a view on a base table. A first stream expansion on the first stream object is performed. The first stream expansion is based on generating a second stream object on the base table. A second stream expansion of the second stream object is performed. The second stream expansion is based on replacing the second stream object with at least a second data processing operation. The query is executed based on completing the first data processing operation and the at least a second data processing operation.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
G06F 16/22 - Indexing; Data structures therefor; Storage structures
In an embodiment, a data platform creates an application in a data-provider account. The application includes one or more APIs corresponding to one or more underlying code blocks. The data platform shares provider data with the application in the data-provider account, and also installs, in a data-consumer account, an application instance of the application. The application instance includes one or more APIs corresponding to the one or more APIs in the application in the data-provider account. The data platform shares consumer data with the application instance in the data-consumer account, and invokes one or more of the APIs of the application instance to execute respective associated underlying code blocks, which are not visible to the data-consumer account. The data platform also saves output of the one or more respective associated underlying code blocks locally within the data-consumer account.
A method includes installing, in a consumer database account, a shared-instance database that includes a shared instance of a provider-account database that resides in a provider database account. The shared-instance database includes a first schema that includes provider-account training data, provider-account scoring data, a training function, and a scoring function. The method also includes invoking the training function from the consumer database account, which results in creation in the consumer database account of a second schema that includes a machine-learning-model instance of a machine learning model, and which also results in training the machine-learning model instance with at least the provider-account training data. Additionally, the method includes generating consumer-account scoring data by inputting, into the trained machine-learning-model instance, consumer-account input data that is stored in the consumer database account. The method also includes storing the consumer-account scoring data in the consumer database account.
A system for a materialized table (MT) refresh using multiple processing pipelines includes at least one hardware processor coupled to memory storing instructions. The instructions cause the at least one hardware processor to perform operations including determining dependencies among a plurality of intermediate MTs generated from a source MT. The source MT uses a table definition with a query on one or more base tables and a lag duration value. A graph snapshot of dependencies among the plurality of intermediate MTs is generated. Processing pipelines are configured. Each of the processing pipelines corresponds to a subset of the plurality of intermediate MTs indicated by the graph snapshot. Responsive to detecting an instruction for a refresh operation on the source MT, refreshes on corresponding intermediate MTs of the plurality of intermediate MTs in each processing pipeline of the processing pipelines are performed to complete the refresh operation on the source MT.
G06F 7/14 - Merging, i.e. combining at least two sets of record carriers each arranged in the same ordered sequence to produce a single set having the same ordered sequence
82.
PROCESSING FUNCTIONALITY TO STORE SPARSE FEATURE SETS
The subject technology generates, by a database system, cell data for a particular table based on values from a source table, the values being based on raw input data, the source table comprising multiple rows and multiple columns, the raw input data comprising values in a first format, the values comprising input features corresponding to datasets included in the raw input data for machine learning models, the source table being provided by an external environment, the external environment comprising an external system from the database system. The subject technology performs a database operation to generate the particular table including table metadata, column metadata, and the generated cell data, the generated particular table comprising a second format that causes more efficient processing of data by the database system using a single query on the particular table compared to processing the raw input data from the source table.
Embodiments of the present disclosure may provide a data protection system that performs identification of errors from queries on a database. The data protection system can further identify corrupted data from additional errors, are difficult to detect, and occur between layers of data in the database system. The data protection system can perform corrections of the error data by rebuilding database data or removing the corrupted data.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 11/07 - Responding to the occurrence of a fault, e.g. fault tolerance
G06F 16/215 - Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
84.
PROJECTION CONSTRAINTS ENFORCED IN A DATABASE SYSTEM
A system for enforcing projection constraints on data values stored in specified variables of a shared dataset of a cloud data platform. A request is received from a first account of the cloud data platform that identifies a first operation to be performed on the shared dataset. A first set of data, including data accessed from a first variable, is accessed from the shared dataset to use in performing the first operation. A projection constraint policy attached to the first variable of the shared dataset is determined, and the projection constraint policy is further determined to be enforced based on the request. Based on the first set of data and the first operation, an output to the first request is generated.
A data platform for developing and deploying a data application. The data platform receives from a first user the data application and provider granted privileges including a consumer usage privilege and a consumer access to data privilege. The data platform authorizes the second user to access the data platform based on one or more consumer account privileges included in a set of account privileges. The data platform authorizes the second user to execute the data application based on the consumer usage privilege. During execution, the data platform authorizes the data application to access the provider database object based on the consumer access to data privilege, and authorizes the data application to access the consumer database object based on a provider access to data privilege provided by the second user.
The disclosure relates generally to methods, systems, and apparatuses for managing network connections. An example method includes receiving one or more messages from a plurality of computing devices connected through a network, the one or more messages indicating actual connections among the plurality of computing devices. The method also includes comparing the actual connections to a list of expected connections indicated by a connections master file that comprises connection information for the plurality of computing devices. The method also includes identifying an unexpected connection based on one of the actual connections having no matching entry in the list of expected connections and updating the connections master file by adding the unexpected connection to the list of expected connections indicated by the connections master file.
H04L 43/026 - Capturing of monitoring data using flow identification
H04L 41/22 - Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks comprising specially adapted graphical user interfaces [GUI]
H04L 43/00 - Arrangements for monitoring or testing data switching networks
H04L 43/062 - Generation of reports related to network traffic
H04L 41/0604 - Management of faults, events, alarms or notifications using filtering, e.g. reduction of information by using priority, element types, position or time
H04L 43/0811 - Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking connectivity
G06F 21/57 - Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
G06F 21/56 - Computer malware detection or handling, e.g. anti-virus arrangements
87.
PROCESSING USER-DEFINED FUNCTIONS (UDFs) USING MULTIPLE EXECUTION ENVIRONMENTS
A method includes detecting, by at least one hardware processor, an upload of a user application within a database system. The user application includes user-defined function (UDF) code. A plurality of dependencies of the user application is determined by the at least one hardware processor. A plurality of execution environments corresponding to the plurality of dependencies is generated by the at least one hardware processor. The plurality of execution environments is associated with a corresponding plurality of data types of the UDF. A database query is decoded. The database query specifies database data of a data type of the plurality of data types of the UDF. The database query is processed using at least one of the plurality of execution environments to generate results data. The at least one of the plurality of execution environments corresponds to the data type of the plurality of data types of the UDF.
G06F 16/22 - Indexing; Data structures therefor; Storage structures
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
This disclosure provides methods and techniques of data replication involving cross replication group (RG) references. The present disclosure avoids automatic replication failing when an entity in an RG refers to another entity external to the RG. The entity to be replicated within the RG is referred to as the “referring entity,” and the entity as the dangling reference is referred to as the “referred entity.” Although the referring and referred entities are not replicated together in a replication operation, the referred entity may have already been replicated to the target account in another replication operation on a different replication group. In such cases, the data replication procedure may, according to aspects of the present disclosure, check if the referred entity has already been replicated, and if so, proceed to replicate the referring entity without fail, and link the referring and referred entities to enable normal functioning of the referring entity.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/25 - Integrating or interfacing systems involving database management systems
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
A data platform for managing an application as a first-class database object. The data platform includes at least one processor and a memory storing instructions that cause the at least one processor to perform operations including detecting a data request from a browser for a data object located on the data platform, executing a stored procedure, the stored procedure containing instructions that cause the at least one processor to perform additional operations including instantiating a User Defined Function (UDF) server, an application engine, and the application within a security context of the data platform based on a security policy determined by an owner of the data object. The data platform then communicates with the browser using the application engine as a proxy server.
Techniques are described for budget tracking in a differentially private security system. A request to perform a query of a private database system is received by a privacy device from a client device. The request is associated with a level of differential privacy. A privacy budget corresponding to the received request is accessed by the privacy device. The privacy budget includes a cumulative privacy spend and a maximum privacy spend, the cumulative privacy spend representative of previous queries of the private database system. A privacy spend associated with the received request is determined by the privacy device based at least in part on the level of differential privacy associated with the received request. If a sum of the determined privacy spend and the cumulative privacy spend is less than the maximum privacy spend, the query is performed. Otherwise a security action is performed based on a security policy.
Disclosed herein are embodiments of systems and methods for synchronizing file-catalog table with a file stage. In an embodiment, a data platform receives a notification of a modification to one or more files in a file stage. The file stage includes data storage having a storage location. The data platform updates, based on receiving the notification of the modification, a first file-catalog table for the file stage based on the modification. The first file-catalog table includes a row for each file in the file stage, as well as a column for each of one or more metadata properties of the one or more files in the file stage.
G06F 16/22 - Indexing; Data structures therefor; Storage structures
G06F 16/908 - Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
92.
Projection constraints in a query processing system
A constraint system enforces projection constraints on data values stored in specified columns of a shared dataset when queries are received by a database system. A projection constraint identifies that the data in a column may be restricted from being projected (e.g., presented, read, outputted) in an output to a received query, while allowing specified operations to be performed on the data and a corresponding output to be provided. For example, the projection constraint may indicate a context for a query that triggers the constraint, such as based on the user that submitted the query. Enforcing projection constraints on queries received at the database system allows for data to be shared and used anonymously by entities to perform various operations without the need to tokenize the data.
The distributed database can implement unstructured data replication using an internal or external storage location. Metadata, such as a directory table that lists the unstructured files, can be replicated across different deployments, followed by replication of the staged data. Replicating the staged data can be implemented by replication of only the stage metadata or replication of the database files between the deployments.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
G06F 16/25 - Integrating or interfacing systems involving database management systems
A data platform for developing and deploying a user application within a unified security context. The data platform authorizes a first user to use an editor to access source code of a user application based on security policies of a security context and authorizes the first user to use an application and data manager to set usage privileges for a second user to use the user application based on the security policies of the security context. To provide the user application to the second user, the data platform deploys the user application by instantiating a User Defined Function (UDF) server and an application engine of the UDF server within the security context, instantiating the user application as an application of the application engine within the security context, and authorizing access by the user application to databased on the security policies of the security context.
In an embodiment, an application is created on a data-provider platform. The application includes one or more application programming interfaces (APIs) corresponding to one or more underlying code blocks. Provider data is shared with the application on the data-provider platform. An application instance of the application is installed in a trusted execution environment (TEE). The application instance includes one or more APIs corresponding to the one or more APIs in the application on the data-provider platform. Consumer data is shared with the application instance from a data-consumer platform. One or more of the APIs of the application instance are invoked to execute, on the TEE, respective associated underlying code blocks that are not visible on the TEE. The output of the one or more respective associated underlying code blocks is saved to the data-consumer platform.
G06F 21/62 - Protecting access to data via a platform, e.g. using keys or access control rules
G06F 21/53 - Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity, buffer overflow or preventing unwanted data erasure by executing in a restricted environment, e.g. sandbox or secure virtual machine
Using an attributes model of a time series forecasting model, determine a set of features based on time series data, the set of features including periodic components. The time series data may be divided into a set of segments. For each segment of the set of segments, a weight may be assigned using an age of the segment, resulting in a set of weighted segments of time series data. Using a trend detection model of the time series forecasting model, trend data from the set of weighted segments of time series data may be determined. A time series forecast may be generated by combining the set of features and the trend data.
The subject technology receives, from a metadata database, information related to a base table. The subject technology determines a table object associated with the base table, the table object including a first set of metadata. The subject technology generates a nested object based on a second set of metadata, the second set of metadata including information linking the nested object to the table object. The subject technology generates a second table object associated with the nested object, the second table object representing a secondary index of the base table, the second table object including information linking the second table object to the nested object. The subject technology establishes a link between the second table object to the base table based on the nested object. The subject technology stores, in the metadata database, the nested object and the second table object.
A data platform for managing an application as a first-class database object. The data object can include User Interface (UI) components. The data application can be shared by a provider account to a plurality of consumer accounts using a share object and based on grant commands. The consumer accounts can deploy and operate the UI component based on the share object.
The subject technology receives a command to commit a table in a different table format on an external volume. The subject technology generates a first snapshot of the table on internal storage. The subject technology generates a first list of metadata files on the internal storage. The subject technology generates, based on the first list of metadata files, a first set of metadata files on the internal storage. The subject technology generates a second snapshot of the table on the external volume. The subject technology generates a second list of metadata files on the external volume. The subject technology generates, based on the second list of metadata files, a second set of metadata files on the external volume. The subject technology generates a first set of data files in a different file format on the external volume.
A method includes retrieving a plurality of materialized tables (MTs). Each of the plurality of MTs includes a lag duration and refers to a corresponding base table of a plurality of base tables. The lag duration indicates a maximum time period that a result of a prior refresh of a query on the corresponding base table can lag behind a current time instance. A plurality of time instances for the MT is determined based on the lag duration and a number of prior refreshes of the corresponding base table. A plurality of aligned time instances for the plurality of MTs is determined based on the plurality of time instances for each of the plurality of MTs. Refresh operations are scheduled for the plurality of MTs at one or more of the plurality of aligned time instances that are within the maximum time period.
patents
