Outlier Detection for Shale Gas Production Data: An Integrated Approach of Enhanced Clustering and Domain Knowledge

Abstract Shale gas production data are inherently non-stationary, characterized by distinct decline trends across various production stages. These datasets are frequently contaminated with outliers stemming from operational interventions, such as well shut-ins and choke adjustments, as well as sensor malfunctions and human-induced collection errors. Consequently, robust outlier detection is a prerequisite for reliable production data analysis. While unsupervised learning algorithms, such as Local Outlier Factor (LOF), Angle-Based Outlier Detection (ABOD), and Isolation Forest (IF), are currently employed, their efficacy is often hindered by the subjectivity involved in defining manual thresholds for anomaly partitioning. This study develops an enhanced unsupervised classification algorithm, termed IF-K-Medoids, by integrating the IF algorithm with K-Medoids clustering. This hybrid algorithm addresses the initialization sensitivity of K-Medoids by leveraging IF to optimally select initial cluster centers, thereby eliminating the need for random or manual selection. Furthermore, the K-Medoids enables automated differentiation between anomalous and normal samples, overcoming the subjectivity of manual thresholding. This algorithm is then incorporated into a two-stage outlier detection approach that fuses machine learning with domain knowledge. IF-K-Medoids performs an initial screening of outliers, which are then further reviewed using domain knowledge. This two-stage outlier detection approach enhances both the accuracy and reliability of identifying outliers in production data. The IF-K-Medoids algorithm was first validated using datasets from the University of California-Irvine (UCI) Machine Learning Repository, achieving an Area Under the Curve (AUC) exceeding 0.88, which outperforms standalone IF and Gaussian Mixture Model (GMM) algorithms. The proposed two-stage outlier detection approach was subsequently validated using field data from two shale gas wells. Results demonstrate that the proposed approach produced physically interpretable detection results, thereby facilitating improvements in the accuracy and precision of production forecasting, reserves estimation, and rate transient analysis for shale gas wells.