Introducing Gretel Benchmark
What is Gretel Benchmark?
Today we’re announcing the release of Gretel Benchmark, a Python library for you to compare any model that generates synthetic data, with a set of standardized tests to evaluate those algorithms for synthetic data quality, runtime, and other machine learning use cases.
Want to jump right in? You can use Gretel Benchmark by installing Gretel-trainer.
Keep reading to learn about important features and the detailed Benchmark evaluation report.
Your custom models
It’s easy to define custom models, so you can use any algorithm, not just Gretel models, for synthetic data generation to compare in Benchmark.
To provide your own model implementation, define a Python class that meets this interface:
Learn more here about creating your custom model. Make sure to install any third-party libraries you use as dependencies wherever you are running Benchmark.
We’ve also made it easy for you to use Gretel models in Benchmark. Here’s a nifty summary of all the available Gretel models with default configurations:
- This model will automatically pick the best solution between Gretel LSTM and Gretel CTGAN for your dataset (see more below on the two models). This can be helpful if you want the Gretel engine to select the best model based on the characteristics of your dataset.
- This model works for a variety of synthetic data tasks and works with time-series, tabular, and text data. Gretel LSTM is generally most useful for datasets with a few thousand records and upward. Datasets can include a mix of categorical, continuous, and numerical values.
- This model works well for high-dimensional, largely numeric data. You can use Gretel CTGAN for datasets with more than 20 columns and/or 50,000 rows.
- Data requirements: Not ideal if the dataset contains free text fields.
- This model is useful for natural language or plain text datasets such as reviews, tweets, and conversations.
- Data requirements: Dataset must be single-column.
- This model is great for generating lots of data quickly.
- Note: Gretel Amplify is not a neural network model but instead uses statistical means to generate large amounts of data from an input dataset. The Synthetic Data Quality Score (SQS) for data generated using Gretel Amplify may be lower.
You can also easily modify the Gretel model configs with:
Find out more about how to use Gretel model classes in the Benchmark documentation.
Benchmark allows you to compare the synthetic data quality and runtime of multiple models (whether custom or Gretel models) on multiple datasets.
If you need test data, we also provide a list of publicly available datasets that are popular for synthetic data use cases like those in finance, e-commerce, healthcare, and more. You can view and select datasets in Benchmark using these functions:
We created a set of standard tests that make up the Gretel Benchmark evaluations on algorithms for synthetic data generation. The Benchmark report shows:
- Data type
- Data shape
- Synthetic Data Quality Score (SQS): an evaluation, developed by Gretel, of the quality of synthetic data. Learn more about SQS on our SQS FAQ.
- Train time
- Generate time
- Total runtime
The Benchmark report
Want to evaluate Gretel models on your industry use case? For a quick and easy look into how different Gretel models perform on popular machine learning datasets, check out our Benchmark report below. When you run Benchmark, you’ll also see an evaluation report like this one.
You can use a Benchmark report like the one shown here to evaluate which Gretel model is best for your synthetic data goals. For example, Gretel LSTM consistently generates synthetic data with a high Synthetic Data Quality Score (SQS) on multiple types of tabular or complex data. As seen in the results below, Gretel CTGAN is great for particularly long or wide datasets and generally has a faster runtime. If you’re looking to quickly generate lots of data, Gretel Amplify produces results in 1/10 of the time (check out the fast train and generate times!). Gretel GPT-X generates high-quality synthetic data for natural language datasets. Depending on your specific goals with synthetic data or constraints, you may find particular Gretel models to be best suited for your use case. You can reference the Benchmark report below to guide how you evaluate Gretel models, or of course, try Benchmark yourself!
You can find out more in the Benchmark documentation. Questions or comments? We’re always available in our Discord community - send us a note! Happy synthesizing!