Data Curation paper - Reads

Paper #1 - A Survey on Data Cleaning Methods for Improved Machine Learning Model Performance

• Two aspects of data cleaning: what to clean and how to clean

Key Notes

• SampleClean: Simulated Clean Data Instances - SampleClean suggests a solution to sampling the raw data that can better present clean data instances.
• Approximate Query Processing (AQP). The AQP consists of two steps: first, in Direct Estimate (DE), a set of k rows is sampled randomly and cleaned, and the training result is returned independently of the dirty data. The correction step is used to reweight the sample based on the contribution of the cleaned data
• ActiveClean: Incremental Data Cleaning in Convex Models. ActiveClean gradually cleans a dirty dataset to learn a convex-loss model, such as Logistic Regression and Support Vector Machine (SVM).
• HoloClean: Holistic Data Repairs With Probabilistic Inference
• AlphaClean: Generate-Then-Search Parallel Data Cleaning
• CPClean: Reusable Computation in Data Cleaning

ML Papers - Learning-with-Label-Noise

Paper #2 - Advancing Data Curation With Metadata and Statistical Relational Learning

Key Notes

• We refer to data science as an umbrella term gathering algorithms and techniques from several disciplines, such as statistics, software engineering, and machine learning
• Data is inconsistent, duplicated, stale, incomplete, and/or inaccurate. Data errors, such as outliers, duplicates, missing values, and inconsistencies.
• Mapping Metadata to Data Quality Issues
• Error Detection
• Joint Error Detection and Repair Suggestion

Data Quality fundamentals

• The Consistency dimension refers to the validity and integrity of values and tuples with respect to defined inter- and intra-relational constraints that exist within either single or multiple relations
• The accuracy dimension identifies correct and true values of the entities presented by data.
• Completeness is a degree to which values are included in a data collection
• Timeliness dimension reflects the change and update of data by identifying the most current value of an entity in a database
• Core data quality dimensions, the violation of Accuracy, Consistency,
• Uniqueness, Completeness and Timeliness lead to data quality issues

• Metadata is "structured information that describes, explains, locates, or otherwise makes it easier to retrieve, use, or manage an information resource"

Single-Column Profiling Tasks

• Cardinalities refers to the counts of values
• Number of rows: the number of entities which are available in the table;
• Distinctness: the number of distinct values of the single attribute;
• Uniqueness: the ratio of the number of distinct values to the number of rows

Value Distribution refers to the distribution of values on the column. This category includes:

• Constancy: the ratio between the most frequent value count and the number of rows;
• Extreme values: minimum and maximum values in numeric columns; shortest and
• longest strings in categorical, alphanumeric or text columns;
• Histogram: values distribution summary on an attribute
• Quartiles: three points that divide numeric distribution into four equal groups;
• Inverse distribution: an inverse frequency distribution (a distribution of the frequency distribution);

Patterns

• Patterns refers to the syntactic properties on the values of the individual column.
• Lengths, which specifies the descriptive statistics of the column value lengths
• Decimals, which determines the number of decimals in numeric columns

Multi-Column Profiling Tasks

• Functional dependencies
• What. The first dimension captures common data quality issues and typical data cleaning tasks, which had been found in the literature.
• How. The second dimension reflects differently focused data cleaning approaches.

Rule-Based Approaches

• Data cleaning rules or integrity constraints to detect and repair various error types in the dataset.

Statistical Approaches

• DEC (DetectExplore-Clean) framework [22] uses statistical and other analytical techniques, such as the Fleiss’ kappa measure, to compute the glitch score, which identifies and scores the data glitches

Probabilistic and Machine Learning-Based Approaches

• The BoostClean system [141] addresses the domain value violations while cleaning training data for predictive models
• The HoloClean system [202] considers error detection as a black-box component and expects the specification of integrity constraints-aligned data quality rules to make probabilistic suggestions on how to repair erroneous data values.
• Interactive Data Cleaning
• Numerous data cleaning systems use crowdsourcing for duplicate detection and resolution

Supervised Error Detection with Metadata

1) an Error Detection Suite, which includes pluggable error detection systems that function as black boxes to our system.

2) a Metadata Profiler Suite, which extracts various metadata categories, and 

3) an Aggregation Suite, which combines the output of the error detection suite and the profiler. In the following, we describe each of the components.

Keep Exploring!!!

Data Curation Paper Reads - Data Quality - Data Cleaning

Paper #1 - Auto-Detect: Data-Driven Error Detection in Tables

Key Notes

• Values in a column not conforming to patterns associated with a data-type are flagged as errors.
• Formulas inconsistent with other formulas in the region 
• Text clustering feature that groups together similar values in a column
• Single-column approaches detect errors only based on values within an input column.
• When certain multi-column data quality rules (e.g. function-dependencies and other types of first-order logic)

Methods

• Fixed-Regex (F-Regex)
• dBoost
• Compression-based dissimilarity measure (CDM)
• Support vector data description (SVDD)
• Distance-based outlier detection (DBOD)
• Local outlier factor (LOF)
• Multi-column error detection using rules
• Single-column error detection
• Numeric error detection
• Outlier detection
• Application-driven error correction. Recent approaches such as BoostClean  and ActiveClean

Record Linkage

I like this technique for data merging

• Similarity between two words 
• Match between numbers
• Match between First Name
• Match between Last Name

Similarity distance function

Deep learning for ER

BoostClean selects an ensemble of methods (statistical and logic rules) for error detection and for repair combinations using statistical boosting.

• ActiveDetect - detects and prioritizes the most important data errors in a dataset.
• Sample clean - A Sample-and-Clean Framework for Fast and Accurate Query Processing on Dirty Data
• AlphaClean -declaratively synthesizes data cleaning programs
• A Data Quality Metric (DQM)
• Data Cleaning for Data Science - PrivateClean, ActiveClean, and BoostClean.

More Reads

• BoostClean: Automated Error Detection and Repair for Machine Learning
• Machine Learning-Based Data Cleaning : Current Solutions and Challenges 
• BoostClean: Automated Error Detection and Repair for Machine Learning
• Auto-Data Cleaning
• A Demonstration of DBWipes: Clean as You Query
• SampleClean: Fast and Reliable Analytics on Dirty Data
• ActiveClean: Interactive Data Cleaning For Statistical Modeling
• AlphaClean: Automatic Generation of Data Cleaning Pipelines
• Towards Automated Data Cleaning Workflows

Keep Exploring!!!

Data Quality - Algorithm Fairness - Data Curation Papers

Paper #1 - Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI

Key Notes

• Poor data quality in high-stakes domains can have outsized effects on vulnerable communities and context
• Data Cascades: compounding events causing negative, downstream effects from data issues, resulting in technical debt over time
• Many researchers have pointed to the undervalued human labour that powers AI models 
• Practitioners often work with a set of assumptions about their data during analysis and visualisation
• Other frameworks to discover data bugs and clean data include ActiveClean and BoostClean
• Data cascades are complex, long-term, occur frequently and persistently
• Under-valuing of data work is common to all of AI development
• Practitioners viewed data as operations, moved fast, hacked model performance (through hyperparameters rather than data quality)
• Everyone wants to do the model work, not the data work
• It was difficult to get buy-in from clients and funders to invest in good quality data collection and annotation work
• Lack of adequate training on AI data quality
• Cascades triggered by ‘hardware drifts’
• Cascades triggered by ‘environmental drifts’

Paper #2 - Re-imagining Algorithmic Fairness in India and Beyond

Key Notes

• While Indians are part of the AI workforce, a majority work in services, and engineers do not entirely represent marginalities,limiting re-mediation of distances
• While other axes of discrimination and injustices such as disability status
• Algorithmic powerful in India, where the distance between models and oppressed communities is large
• “rich people problems like cardiac disease and cancer, not poor people’s Tuberculosis, prioritised in AI"

More Reads

• Non-portability of Algorithmic Fairness in India
• Auto-Detect: Data-Driven Error Detection in Tables
• BoostClean: Automated Error Detection and Repair for Machine Learning
• Machine Learning-Based Data Cleaning : Current Solutions and Challenges 
• BoostClean: Automated Error Detection and Repair for Machine Learning
• Auto-Data Cleaning
• ActiveClean: Interactive Data Cleaning For Statistical Modeling

Keep Thinking!!!

Data Curation Reads

Data curation – the process of discovering, integrating, and cleaning data. Data Curation needs to be guided by Data Governance

Paper - Data Curation with Deep Learning [Vision]

• (Towards Automatic Curation)
• Diverse DC tasks (such as deduplication, error detection, data repair) 
• DC problems, such as data discovery [17] and entity resolution
• Series of techniques (e.g., unsupervised representation learning, data augmentation, synthetic data generation, weak supervision, domain adaptation, and crowdsourcing)

An Approach Adapted from Word Embeddings

• map words to a dense high dimensional vector such that semantically related words are close to each other
• A big difference between databases and documents is that databases have many data dependencies (or integrity constraints), within tables

Combining Word and Graph Embeddings

• Treat each relation as a heterogeneous network
• Learn distributed representations for the cells over the entire data ocean, not only on one relation

Experiment for different levels

• Column Embeddings (Column2Vec)
• Table Embeddings (Table2Vec) 
• Database Embeddings (Database2Vec)
• Contextual Embeddings for DC

Entity Matching

• Entity matching is a key problem in data integration

DeepER, applies DL techniques for ER.

DeepMatcher [35] proposes a template based architecture for entity matching

Paper - A Survey on Data Collection for Machine Learning

Key Notes

More Reads

• Data provenance, curation and quality in metrology
• CleanML: A Study for Evaluating the Impact of Data Cleaning on ML Classification Tasks
• Data Acquisition for Improving Machine Learning Models
• “You might also like this model”: Data Driven Approach for Recommending Deep Learning Models for Unknown Image Datasets
• Data Curation with Deep Learning
• Data Loss Prevention Samples

Keep Reading!!!