A differentially private security system communicatively coupled to a database storing restricted data receives a database query from a client. The database query includes an operation, a target accuracy, and a maximum privacy spend for the query. The system performs the operation to produce a result, then injects the result with noise sampled from a Laplace distribution to produce a differentially private result. The system iteratively calibrates the noise value of the differentially private result using a secondary distribution different from the Laplace distribution and a new fractional privacy spend. The system ceases to iterate when an iteration uses the maximum privacy spend or a relative error of the differentially private result is determined to satisfy the target accuracy, or both. The system sends the differentially private result to the client.
G06F 16/90 - Information retrieval; Database structures therefor; File system structures therefor - Details of database functions independent of the retrieved data types
A differentially private security system communicatively coupled to a database storing restricted data receives a database query from a client. The database query includes a relation indicative of data to perform the query upon and at least one privacy parameter indicative of a level of differential privacy with which to perform the query. The differentially private security system determines a noise type for the query. The differentially private security system determines a representation of probabilistic privacy loss for the query based on the determined noise type. The differentially private security system determines a privacy spend for the query using the generated representation of probabilistic privacy loss. The differentially private security system determines whether the determined privacy spend exceeds a privacy budget associated with the client.
A differentially private security system communicatively coupled to a database storing restricted data receives a database query from a client. The database query includes a relation specifying a set of data in the database upon which to perform the query and privacy parameters associated with the query. The differentially private security system determines a worst-case privacy spend for the query based on the privacy parameters and the relation. The differentially private security system performs the query upon the set of data specified by the relation and decrements the determined worst-case privacy spend from a privacy budget associated with the client. The differentially private security system records the worst-case privacy spend and the query at a log and determines a privacy budget refund based on queries recorded in the log. The differentially private security system applies the determined privacy budget refund to the privacy budget associated with the client.
Techniques are described for budget tracking in a differentially private security system. A request to perform a query of a private database system is received by a privacy device from a client device. The request is associated with a level of differential privacy. A privacy budget corresponding to the received request is accessed by the privacy device. The privacy budget includes a cumulative privacy spend and a maximum privacy spend, the cumulative privacy spend representative of previous queries of the private database system. A privacy spend associated with the received request is determined by the privacy device based at least in part on the level of differential privacy associated with the received request. If a sum of the determined privacy spend and the cumulative privacy spend is less than the maximum privacy spend, the query is performed. Otherwise a security action is performed based on a security policy.
Systems, methods, and devices for batch ingestion of data into a table of a database. A method includes determining a notification indicating a presence of a user file received from a client account to be ingested into a database. The method includes identifying data in the user file and identifying a target table of the database to receive the data in the user file. The method includes generating an ingest task indicating the data and the target table. The method includes assigning the ingest task to an execution node of an execution platform, wherein the execution platform comprises a plurality of execution nodes operating independent of a plurality of shared storage devices collectively storing database data. The method includes registering metadata concerning the target table in a metadata store after the data has been fully committed to the target table by the execution node.
A method for a database system includes storing table data for a database, the table data including information in rows and columns of one or more database tables. The method includes storing metadata on immutable storage, the metadata including information about the table data for the database. In one embodiment, mutable metadata may be periodically consolidated in the background to create new versions of metadata files and which allows for deletions of old metadata files and old data files.
A method for sharing data in a multi-tenant database includes generating a share object in a first account comprising a share role. The method includes associating one or more access rights with the share role, wherein the one or more access rights indicate which objects in the first account are accessible based on the share object. The method includes granting, to a second account, cross-account access rights to the share role or share object in the first account. The method includes receiving a request from the second account to access data or services of the first account. The method further includes providing a response to the second account based on the data or services of the first account.
G06F 16/90 - Information retrieval; Database structures therefor; File system structures therefor - Details of database functions independent of the retrieved data types
G06F 21/62 - Protecting access to data via a platform, e.g. using keys or access control rules
H04L 9/32 - Arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system
A method includes storing table data for a table in a plurality of partitions and for maintaining approximate or good enough clustering. The method includes creating one or more new partitions based on changes to the table, wherein at least one of the one or more new partitions overlap with each other or previous partitions resulting in a decrease in a degree of clustering of the table. The method includes determining that a degree of clustering of the table data is below a clustering threshold. The method further includes reclustering one or more partitions of the table to improve the degree of clustering of the table in response to one or more of: determining that the degree of clustering has fallen below the clustering threshold, an explicit user command from a user, and/or as part of a DML command. Reclustering may be performed in incremental steps to iteratively improve clustering.
Example systems and methods for cloning catalog objects are described. In one implementation, a method identifies an original catalog object associated with a set of data and creates a duplicate copy of the original catalog object without copying the data itself. The method allows access to the set of data using the duplicate catalog object and supports modifying the data associated with the original catalog object independently of the duplicate catalog object. The duplicate catalog object can be deleted upon completion of modifying the data associated with the original catalog object.
G06F 16/90 - Information retrieval; Database structures therefor; File system structures therefor - Details of database functions independent of the retrieved data types
A method for a multi-cluster warehouse includes allocating a plurality of compute clusters as part of a virtual warehouse. The compute clusters are used to access and perform queries against one or more databases in one or more cloud storage resources. The method includes providing queries for the virtual warehouse to each of the plurality of compute clusters. Each of the plurality of compute clusters of the virtual warehouse receives a plurality of queries so that the computing load is spread across the different clusters. The method also includes dynamically adding compute clusters to and removing compute clusters from the virtual warehouse as needed based on a workload of the plurality of compute clusters.
The disclosure relates generally to methods, systems, and apparatuses for managing network connections. A system for managing network connections includes a storage component, a decoding component, a rule manager component, and a notification component. The storage component is configured to store a list of expected connections for a plurality of networked machines, wherein each connection in the list of expected connections defines a start point and an end point for the connection. The decoding component is configured to decode messages from the plurality of networked machines indicating one or more connections for a corresponding machine. The rule manager component is configured to identify an unexpected presence or absence of a connection on at least one of the plurality of network machines based on the list of expected connections. The notification component is configured to provide a notification or indication of the unexpected presence or absence.
A hardware database privacy device is communicatively coupled to a private database system. The hardware database privacy device receives a request from a client device to perform a query of the private database system and identifies a level of differential privacy corresponding to the request. The identified level of differential privacy includes privacy parameters (e,d) indicating the degree of information released about the private database system. The hardware database privacy device identifies a set of operations to be performed on the set of data that corresponds to the requested query. After the set of data is accessed, the set of operations is modified based on the identified level of differential privacy such that a performance of the modified set of operations produces a result set that is (8,5)-differentially private.
Provided are methods, apparatuses, and systems for managing semi-structured data. For example, an implementation of a method for managing semi-structured data may receive semi- structured data elements from a data source, and may perform statistical analysis on collections of the semi-structured data elements as they are added to the database. Additionally, common data elements from within the semi-structured data may be identified and may further combine the common data elements from the data source into separate pseudo-columns stored in cache memory. The implementation may further make metadata and statistics corresponding to the pseudo-columns available to a computer based query generator, and may store non-common data elements in an overflow serialized column in computer memory.
G06F 16/80 - Information retrieval; Database structures therefor; File system structures therefor of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML
G06F 17/18 - Complex mathematical operations for evaluating statistical data
14.
IMPLEMENTATON OF SEMI-STRUCTURED DATA AS A FIRST-CLASS DATABASE ELEMENT
G06F 16/80 - Information retrieval; Database structures therefor; File system structures therefor of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML
G06F 17/18 - Complex mathematical operations for evaluating statistical data
Example caching systems and methods are described. In one implementation, a method identifies multiple files used to process a query and distributes each of the multiple files to a particular execution node to execute the query. Each execution node determines whether the distributed file is stored in the execution node's cache. If the execution node determines that the file is stored in the cache, it processes the query using the cached file. If the file is not stored in the cache, the execution node retrieves the file from a remote storage device, stores the file in the execution node's cache, and processes the query using the file.
Example resource management systems and methods are described. In one implementation, a resource manager is configured to manage data processing tasks associated with multiple data elements. An execution platform is coupled to the resource manager and includes multiple execution nodes configured to store data retrieved from multiple remote storage devices. Each execution node includes a cache and a processor, where the cache and processor are independent of the remote storage devices. A metadata manager is configured to access metadata associated with at least a portion of the multiple data elements.
Example data management systems and methods are described. In one implementation, a method identifies multiple files to process based on a received query and identifies multiple execution nodes available to process the multiple files. The method initially creates multiple scansets, each including a portion of the multiple files, and assigns each scanset to one of the execution nodes based on a file assignment model. The multiple scansets are processed by the multiple execution nodes. If the method determines that a particular execution node has finished processing all files in its assigned scanset, an unprocessed file is reassigned from another execution node to the particular execution node.
Example resource provisioning systems and methods are described. In one implementation, an execution platform accesses multiple remote storage devices. The execution platform includes multiple virtual warehouses, each of which includes a cache to store data retrieved from the remote storage devices and a processor that is independent of the remote storage devices. A resource manager is coupled to the execution platform and monitors received data processing requests and resource utilization. The resource manager also determines whether additional virtual warehouses are needed based on the data processing requests and the resource utilization. If additional virtual warehouses are needed, the resource manager provisions a new virtual warehouse.
G06F 9/50 - Allocation of resources, e.g. of the central processing unit [CPU]
G06F 16/90 - Information retrieval; Database structures therefor; File system structures therefor - Details of database functions independent of the retrieved data types
Example resource provisioning systems and methods are described. In one implementation, an execution platform accesses multiple remote storage devices. The execution platform includes multiple virtual warehouses, each of which includes a cache to store data retrieved from the remote storage devices and a processor that is independent of the remote storage devices. A resource manager is coupled to the execution platform and monitors received data processing requests and resource utilization. The resource manager also determines whether additional virtual warehouses are needed based on the data processing requests and the resource utilization. If additional virtual warehouses are needed, the resource manager provisions a new virtual warehouse.
G06F 9/50 - Allocation of resources, e.g. of the central processing unit [CPU]
G06F 16/90 - Information retrieval; Database structures therefor; File system structures therefor - Details of database functions independent of the retrieved data types
patents
