What is Gretel Benchmark?

Today we’re announcing the release of Gretel Benchmark, a Python library for you to compare any model that generates synthetic data, with a set of standardized tests to evaluate those algorithms for synthetic data quality, runtime, and other machine learning use cases. 

Getting started 

Want to jump right in? You can use Gretel Benchmark by installing Gretel-trainer.

Get started with the quickstart Benchmark notebook and Benchmark documentation. 

Keep reading to learn about important features and the detailed Benchmark evaluation report.

Models

Your custom models

It’s easy to define custom models, so you can use any algorithm, not just Gretel models, for synthetic data generation to compare in Benchmark. 

To provide your own model implementation, define a Python class that meets this interface:

class MyCustomModel:
    def train(self, source: str, **kwargs) -> None:
        # your training code here
    def generate(self, **kwargs) -> pd.DataFrame:
        # your generation code here

Learn more here about creating your custom model. Make sure to install any third-party libraries you use as dependencies wherever you are running Benchmark.

Gretel models

We’ve also made it easy for you to use Gretel models in Benchmark. Here’s a nifty summary of all the available Gretel models with default configurations:

GretelAuto

• This model will automatically pick the best solution between Gretel LSTM and Gretel CTGAN for your dataset (see more below on the two models). This can be helpful if you want the Gretel engine to select the best model based on the characteristics of your dataset.

‍Gretel LSTM 

• This model works for a variety of synthetic data tasks and works with time-series, tabular, and text data. Gretel LSTM is generally most useful for datasets with a few thousand records and upward. Datasets can include a mix of categorical, continuous, and numerical values. 

Gretel ACTGAN

• This model works well for high-dimensional, largely numeric data. You can use Gretel CTGAN for datasets with more than 20 columns and/or 50,000 rows.
• Data requirements: Not ideal if the dataset contains free text fields.

Gretel GPT 

• This model is useful for natural language or plain text datasets such as reviews, tweets, and conversations. 
• Data requirements: Dataset must be single-column.

Gretel Amplify

• This model is great for generating lots of data quickly. 
• Note: Gretel Amplify is not a neural network model but instead uses statistical means to generate large amounts of data from an input dataset. The Synthetic Data Quality Score (SQS) for data generated using Gretel Amplify may be lower.
  

You can also easily modify the Gretel model configs with: 

class CustomizedLSTM(GretelModel):
    config = {...} # define configuration here

Find out more about how to use Gretel model classes in the Benchmark documentation. 

Data

Benchmark allows you to compare the synthetic data quality and runtime of multiple models (whether custom or Gretel models) on multiple datasets. 

To use your own data in Benchmark, you can follow the instructions for  `make_dataset` in the docs or check out the Benchmark notebook. 

If you need test data, we also provide a list of publicly available datasets that are popular for synthetic data use cases like those in finance, e-commerce, healthcare, and more. You can view and select datasets in Benchmark using these functions: 

list_gretel_datasets(datatype: Optional[Union[Datatype, str]] = None, tags: Optional[List[str]] = None) -> List[Dataset]
"""Returns a list of Gretel-curated datasets matching the specified datatype and tags. Uses “and” semantics—i.e. only returns datasets that match all supplied values.
`datatype` (optional): Datatype to filter on
`tags` (optional): Tags to filter on. Various tags are applied to Gretel-curated datasets, see below"""

get_gretel_dataset(name: str) -> Dataset
"""Fetches a Gretel-curated dataset from Gretel’s S3 bucket
`name` (required): The name of the dataset.
This function will raise an exception if no dataset exists with the supplied name"""

Evaluations

We created a set of standard tests that make up the Gretel Benchmark evaluations on algorithms for synthetic data generation. The Benchmark report shows: 

• Data type 
• Data shape 
• Synthetic Data Quality Score (SQS): an evaluation, developed by Gretel, of the quality of synthetic data. Learn more about SQS on our SQS FAQ. 
• Train time 
• Generate time 
• Total runtime 

The Benchmark report

Want to evaluate Gretel models on your industry use case? For a quick and easy look into how different Gretel models perform on popular machine learning datasets, check out our Benchmark report below. When you run Benchmark, you’ll also see an evaluation report like this one.

You can use a Benchmark report like the one shown here to evaluate which Gretel model is best for your synthetic data goals. For example, Gretel LSTM consistently generates synthetic data with a high Synthetic Data Quality Score (SQS) on multiple types of tabular or complex data. As seen in the results below, Gretel CTGAN is great for particularly long or wide datasets and generally has a faster runtime. If you’re looking to quickly generate lots of data, Gretel Amplify produces results in 1/10 of the time (check out the fast train and generate times!). Gretel GPT-X generates high-quality synthetic data for natural language datasets. Depending on your specific goals with synthetic data or constraints, you may find particular Gretel models to be best suited for your use case. You can reference the Benchmark report below to guide how you evaluate Gretel models, or of course, try Benchmark yourself!  

Learn more

You can find out more in the Benchmark documentation. Questions or comments? We’re always available in our Discord community - send us a note! Happy synthesizing!

‍