Introduction

Last year, we introduced the Gretel Synthetic Report, which compares synthetic and real-world datasets when you train a model, and provides a high-level score and evaluation metrics to help you assess the accuracy of your Gretel generated synthetic data. But what about evaluating the quality of existing synthetic data? We’ve heard your feedback and have expanded our tools to allow you to evaluate the quality of any synthetic datasets, regardless of the source. 

We’re excited to announce the release of Gretel Evaluate, which generates a Synthetic Data Quality Score (SQS) report for any synthetic dataset by comparing it against real-world data. Now, there are no restrictions around evaluating synthetic data only trained with Gretel. The Evaluate API is available via our CLI and SDK and can be run in the Gretel Cloud or locally to generate the Gretel Synthetic Report you know and love. As a fully managed service, you do not need any additional infrastructure to set up or manage. 

In this blog, we’ll walk through a simple example of how Gretel Evaluate works in our SDK. If you are interested in using Evaluate with the Gretel CLI, check out our docs. You can follow along with our Gretel Evaluate demo notebook located here or here:

‍Load and Preview Data

After inputting our Gretel API key, we start by specifying the real-world and synthetic datasets we want to evaluate. These can be local files, web locations, or Pandas DataFrames. In this example, our real-world data is a United States Census dataset on adult income from Kaggle and our synthetic data is the corresponding dataset trained by Gretel Synthetics. Here, we are using Gretel generated synthetic data, but Evaluate can be used on any synthetic dataset.

real_data = "https://gretel-public-website.s3.us-west-2.amazonaws.com/datasets/USAdultIncome5k.csv"
real_df = pd.read_csv(real_data)
real_df

synth_data = "https://gretel-public-website.s3.us-west-2.amazonaws.com/datasets/USAdultIncome5kGenerated.csv"
synth_df = pd.read_csv(synth_data)
synth_df

The notebook then shows a preview of the datasets. Simply looking does not tell us the accuracy of the synthetic data when compared to the real-world data it was trained on. This is where Evaluate comes in.

Create a Quality Report

Once we have specified our datasets, we use Evaluate to generate a Gretel Synthetic Report. Now, we task a worker running in the Gretel cloud to evaluate our data and produce the quality report using a temporary project. The QualityReport class takes our synthetic data as the data_source parameter and our real-world data as the ref_data parameter. Here, we have not specified a project parameter, so a temporary project is created and deleted after the Evaluate job finishes and delivers our results.

report = QualityReport(data_source=synth_data, ref_data=real_data)
report.run()

View Results

Evaluate returns the Gretel Synthetic Report as both HTML and a JSON dictionary. Once the Evaluate job is complete, we can use the peek method to see our resulting SQS.

report.peek()

{'grade': 'Excellent', 'raw_score': 91.42962962962963, 'score': 91}.

Our Synthetic Data Quality Score (SQS), as shown below, is 91 which is classified as excellent. 

This is the same report you see when you train a model using Gretel synthetics, now created as a standalone job. For detailed information on the components of the report, what they mean, and how they are calculated, you can read our Synthetic Data Quality page. 

Conclusion

As demonstrated above, Gretel Evaluate is a simple and powerful tool for analyzing the quality of any synthetically-generated data. By creating a way to benchmark accuracy of synthetic data from any source, Evaluate empowers you to confidently choose the synthetic tool that best fits your use case. If your Synthetic Data Quality Score isn’t as high as you’d like it to be, check out our tips for improving accuracy and try them out on a new Gretel synthetic model. For more information about Gretel Evaluate, check out our docs.

We are super excited to see how you use Gretel Evaluate. Feel free to reach out at hi@gretel.ai or join our Slack Community and share your thoughts there.