Anamoly / Outlier Detection packages

 

Ref - Link

• Pyod - Link
• Timeseries anomaly detection using an Autoencoder
• Autoencoder For Anomaly Detection Using Tensorflow Keras
• keras-anomaly-detection
• SRCNN Anamoly Detection

Keep Exploring!!!

Anamoly Detection

Paper #1 - A review on outlier/anomaly detection in time series data

Key Notes

• An observation which deviates so much from other observations as to arouse suspicions that it was generated by a different mechanism

• (Univariate time series) A univariate time series X = {xt }t ∈T is an ordered set of real-valued observations, where each observation is recorded at a specific time
• (Multivariate time series) A multivariate time series X = {xt }t ∈T is defined as an ordered set of kdimensional vectors, each of which is recorded at a specific time

Outliers Type

• Point outliers. A point outlier is a datum that behaves unusually in a specific time instant 
• Subsequence outliers. This term refers to consecutive points in time whose joint behavior is unusual, although each observation individually is not necessarily a point outlier

• multivariate time series is composed of more than one time-dependent variable a univariate analysis can be performed for each variable to detect univariate point outliers

Paper #2 - A Survey on GANs for Anomaly Detection

Notes

• GANs are a framework for the estimation of generative models via an adversarial process in which two models, a discriminator D and a generator G, are trained simultaneously
• The generator G aim is to capture the data distribution, while the discriminator D estimates the probability that a sample came from the training data rather than G

Paper #3 - Anomaly Detection of Time Series with Smoothness-Inducing Sequential Variational Auto-Encoder

Notes

• time series anomaly detection can be in general divided into two settings: 
• i) subsequence or whole sequence level anomaly whereby a subsequence xm,t1:t2 is labeled as an anomaly; 
• ii) point level anomaly for which a measurement xm,t at time t in sequence m is treated as an anomaly. 

Note - Overview of GAN Structure

Notes

• The generator learns to generate plausible data. The generated instances become negative training examples for the discriminator.
• The discriminator learns to distinguish the generator's fake data from real data. The discriminator penalizes the generator for producing implausible results.

More Reads

• Scalable and Decentralized Algorithms for Anomaly Detection via Learning-Based Controlled Sensing
• Autoencoders for unsupervised anomaly detection in high energy physics

Keep Exploring!!!

Anamoly Reads

Ref - Link

Summary

• Point Anomalies - Value is far outside the entirety of the data set
• Conditional Outliers - With respect to context, Same value may not be anamoly in another time 
• Collective Outliers - Set of 1 or more points that deviate from dataset

Telemanom 

Paper - Machine Learning for Time Series Anomaly Detection

Key Notes

• Clustering methods do not require the data to be labeled, making it a good fit for our unsupervised task. Very sensitive to outlier data points

Two-Step Process

• The number of clusters can be set to 2 (one anomalous and one normal)
• Summarized by taking averages across an interval of one hour
• Rolling Window Sequences

On the Nature and Types of Anomalies: A Review of Deviations in Data

Patent - FAST AUTOMATED DETECTION OF ( 56 ) SEASONAL PATTERNS IN TIME SERIES DATA WITHOUT PRIOR KNOWLEDGE OF SEASONAL PERIODICITY 

Key Notes

• Calculate Automatic correlation based on timeseries values
• Identify local maxima
• The seasonal trend identification module
• Data store for Normal data, Anamoly data
• Scoring module
• Human in loop feedback system

Sklearn Models for Supervised Anomaly Detection. Some popular scikit-learn models for supervised anomaly detection include:

• KNeighborsClassifier
• SVC (SVM classifier)
• DecisionTreeClassifier
• RandomForestClassifier
• Interquartile Range
• Isolation Forest
• Median Absolute Deviation
• K-Nearest Neighbours

More Reads

Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding

Machine Learning for Time Series Anomaly Detection

Anomaly Detection Learning Resources.

Anomaly Detection in Time Series Data Using LSTMs and Automatic Thresholding

SYSTEMAND METHOD FOR TRANSFORMING OBSERVED METRICS INTO DETECTED AND SCORED ANOMALIES

Keep Reading!!!

Research Paper Reads - Logs Monitoring

Ideas need a birds-eye view of the landscape to understand existing work. Papers are the only way to understand that. Bookmarking few notes for future reference

Paper #1 - Log-based software monitoring: a systematic mapping study

Key Notes

• The Lifecycle of Log

• Possible components would be Elasticsearch, Logstash, and Kibana
• Kibana provides an interface for visualization, query, and exploration of log data

• LOGGING - 1) empirical studies on logging practices, (2) requirements for application logs, and (3) implementation of log statements
• LOG INFRASTRUCTURE -  (1) log parsing, and (2) log storage.
• LOG ANALYSIS: : (1) anomaly detection, (2) security and privacy, (3) root cause analysis, (4) failure prediction, (5) quality assurance, (6) model inference and invariant mining, (7) reliability and dependability, and (8) log platforms
• Log Parsing - “textual similarity” between the log messages.
• Each log is converted to a binary vector, with each element representing whether the log contains that keyword
• Transformer - TEMPLATE2VEC (as an alternative to WORD2VEC) to represent extracted templates from logs and LSTMs to learn common sequences of log sequences

Root Cause Analysis

• By correlating log messages and resource consumption, their
• approach builds relationships between changes in resource consumption and application events.
• They propose a technique based on the correlation of console logs and resource usage information to link jobs with anomalous behavior and erroneous nodes.

Failure Prediction

• Utilize system logs to predict failures by mining recurring event sequences that are correlated

Paper #2 - Multi-Source Anomaly Detection in Distributed IT Systems

Key Notes

• Three categories-modalities: metrics, application logs, and distributed traces
• Word frequencies and metrics derived from the logs (e.g TF-IDF)
• Decompose the trace in its building blocks, the events/spans, and predict the next span in the sequence

Paper #3 -  LogBERT: Log Anomaly Detection via BERT

Key Notes

• LogBERT leverages the Transformer encoder to
• model log sequences and is trained by novel self-supervised tasks to capture the patterns of normal sequences. 

Baselines

• Principal Component Analysis (PCA) [19]. PCA builds counting matrix based on the frequency of log keys sequences and then reduces the original counting matrix into a low dimensional space to detect anomalous sequences
• One-Class SVM (OCSVM) [14]. One-Class SVM is a well-known one-class classification model and widely used for log anomaly detection [5,16] by only observing the normal data.
• IsolationForest (iForest) [7]. Isolation forest is an unsupervised learning algorithm for anomaly detection by representing features as tree structures.
• LogCluster [6]. LogCluster is a clustering based approach, where the anomalous log sequences are detected by comparing with the existing clusters.
• DeepLog [2]. DeepLog is a state-of-the-art log anomaly detection approach.
• DeepLog adopts recurrent neural network to capture patterns of normal log sequences and further identifies the anomalous log sequences based on the performance of log key predictions.
• LogAnomaly [23]. Log Anomaly is a deep learning-based anomaly detection approach and able to detect sequential and quantitative log anomalies.

Paper #4 - A Survey on Automated Log Analysis for Reliability Engineering

• Log event sequence: A sequence of log events recording system’s activities.
• Log event count vector: A feature vector recording the log events occurrence

Analysis Insights / Thoughts

• The query for selected values vs Bulk Upload of Data
• Usage patterns segmented for weekday/weekend / Trading hours
• Usage patterns across different time zones
• Usage patterns across different sections of applications
• Number of ad-hoc queries
• Restrict bulk upload to certain timezones / non-peak hours
• Two-stage commit - Upload and commit at a later stage

Big picture Notes

• Limit users to App Access during peak hours (5 calls during peak hours)
• Limit users to App Access during peak hours (10 calls during non-peak hours)
• Refer to replicated data in case of data that has stop-gap 5 hours delay
• Pagination of results
• Cache/reuse of results
• Identify maximum reported errors
• Patterns of errors over a weekday 
• User login activities and queries
• User value - Application usage vs Revenue
• User Action predictions 
• Take top 100 users, Plot the sequence of usage and see common flow/patterns

Diagnosis Perspective

• What is the blocking that happens between
• Page load query vs Search query
• Search query vs Data upload query
• Data upload vs Report download query
• Measure potential data conflicts that cause issues

Prediction Algorithms for User Actions

Execution model

• Understand problem statement
• Understand data sources
• Understand data access / permissions
• Frame NLP / Data / User level details
• Initial Analysis Scope
• Application Understanding
• Connects / Feedback

Diagnosis

• User based - Create APIs / Read APIs / Update APIs - Simple / Bulk / Delete APIs - Single / Bulk
• Do we track at UserId, Numberofcalls,Avgtime
• Nature of transactions - Realtime vs Reporting vs Bulk inserts vs Bulk Updates
• API calls across day by time
• %% Mix of workflow and common tables mapped / accessed by them - Time dimension added for pattern
• A,B,C @ Time T1
• A,B at Time T2

More Reads

• Detecting Anomalies in Software Execution Logs with Siamese Network
• How to Predict the Next Best Action to Progress Sales with TensorFlow 2
• Markov Chains in Python: Beginner Tutorial

Keep Thinking!!!