query: when is a data point considered an outlier

pos: Outlier by using a hierarchical Bayes model, or a mixture model. Outlier In statistics, an outlier is an observation point that is distant from other observations. An outlier may be due to variability in the measurement or it may indicate experimental error; the latter are sometimes excluded from the data set. An outlier can cause serious problems in statistical analyses. Outliers can occur by chance in any distribution, but they often indicate either measurement error or that the population has a heavy-tailed distribution. In the former case one wishes to discard them or use statistics that are robust to outliers, while

pos: Outlier Outlier In statistics, an outlier is an observation point that is distant from other observations. An outlier may be due to variability in the measurement or it may indicate experimental error; the latter are sometimes excluded from the data set. An outlier can cause serious problems in statistical analyses. Outliers can occur by chance in any distribution, but they often indicate either measurement error or that the population has a heavy-tailed distribution. In the former case one wishes to discard them or use statistics that are robust to outliers, while in the latter case they indicate that the distribution has

neg: Dixon's Q test sample size and confidence level, then reject the questionable point. Note that only one point may be rejected from a data set using a Q test. Consider the data set: Now rearrange in increasing order: We hypothesize that 0.167 is an outlier. Calculate "Q": With 10 observations and at 90% confidence, "Q" = 0.455 > 0.412 = "Q", so we conclude 0.167 is indeed an outlier. However, at 95% confidence, "Q" = 0.455 < 0.466 = "Q" 0.167 is not considered an outlier. This means that for this example we can be 90% sure that 0.167 is an outlier, but

neg: Outlier average is determined. Next the absolute deviation between each data point and the average are determined. Thirdly, a rejection region is determined using the formula: where formula_10 is the critical value from the Student distribution with "n"-2 degrees of freedom, "n" is the sample size, and s is the sample standard deviation. To determine if a value is an outlier: Calculate formula_11. If "δ" > Rejection Region, the data point is an outlier. If "δ" ≤ Rejection Region, the data point is not an outlier. The modified Thompson Tau test is used to find one outlier at a time (largest

neg: Outlier of the measurement process, such as when an experiment is not entirely capable of measuring such extreme values, resulting in censored data. In regression problems, an alternative approach may be to only exclude points which exhibit a large degree of influence on the estimated coefficients, using a measure such as Cook's distance. If a data point (or points) is excluded from the data analysis, this should be clearly stated on any subsequent report. The possibility should be considered that the underlying distribution of the data is not approximately normal, having "fat tails". For instance, when sampling from a Cauchy distribution,

neg: Local outlier factor well in numerous setups, often outperforming the competitors, for example in network intrusion detection and on processed classification benchmark data. The LOF family of methods can be easily generalized and then applied to various other problems, such as detecting outliers in geographic data, video streams or authorship networks. The resulting values are quotient-values and hard to interpret. A value of 1 or even less indicates a clear inlier, but there is no clear rule for when a point is an outlier. In one data set, a value of 1.1 may already be an outlier, in another dataset and parameterization (with

neg: Outlier value of "δ" is removed if it is an outlier). Meaning, if a data point is found to be an outlier, it is removed from the data set and the test is applied again with a new average and rejection region. This process is continued until no outliers remain in a data set. Some work has also examined outliers for nominal (or categorical) data. In the context of a set of examples (or instances) in a data set, instance hardness measures the probability that an instance will be misclassified ( formula_12 where is the assigned class label and represent the

neg: Outlier k-nearest neighbors to label observations as outliers or non-outliers. The modified Thompson Tau test is a method used to determine if an outlier exists in a data set. The strength of this method lies in the fact that it takes into account a data set’s standard deviation, average and provides a statistically determined rejection zone; thus providing an objective method to determine if a data point is an outlier. Note: Although intuitively appealing, this method appears to be unpublished (it is "not" described in Thompson (1985)) and one should use it with caution. How it works: First, a data set's

neg: Local outlier factor > 1 means Lower density than neighbors (Outlier) Due to the local approach, LOF is able to identify outliers in a data set that would not be outliers in another area of the data set. For example, a point at a "small" distance to a very dense cluster is an outlier, while a point within a sparse cluster might exhibit similar distances to its neighbors. While the geometric intuition of LOF is only applicable to low-dimensional vector spaces, the algorithm can be applied in any context a dissimilarity function can be defined. It has experimentally been shown to work very

neg: Influential observation parameter formula_6. Also define formula_7, the projection matrix of formula_3. Then we have the following measures of influence: An outlier may be defined as a surprising data point. Leverage is a measure of how much the estimated value of the dependent variable changes when the point is removed. There is one value of leverage for each data point. Data points with high leverage force the regression line to be close to the point. In Anscombe's quartet, only the bottom right image has a point with high leverage. Influential observation In statistics, an influential observation is an observation for a statistical

neg: Local outlier factor Local outlier factor In anomaly detection, the local outlier factor (LOF) is an algorithm proposed by Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng and Jörg Sander in 2000 for finding anomalous data points by measuring the local deviation of a given data point with respect to its neighbours. LOF shares some concepts with DBSCAN and OPTICS such as the concepts of "core distance" and "reachability distance", which are used for local density estimation. The local outlier factor is based on a concept of a local density, where locality is given by formula_1 nearest neighbors, whose distance is used to

neg: Quartile data lying outside these defined bounds can be considered an outlier. Anything below the Lower fence or above the Upper fence can be considered such a case. The fences provide a guideline by which to define an outlier, which may be defined in other ways. The fences define a "range" outside of which an outlier exists; a way to picture this is a boundary of a fence, outside of which are "outsiders" as opposed to outliers. Quartile A quartile is a type of quantile. The first quartile ("Q") is defined as the middle number between the smallest number and the

neg: Local outlier factor strong local fluctuations) a value of 2 could still be an inlier. These differences can also occur within a dataset due to the locality of the method. There exist extensions of LOF that try to improve over LOF in these aspects: Local outlier factor In anomaly detection, the local outlier factor (LOF) is an algorithm proposed by Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng and Jörg Sander in 2000 for finding anomalous data points by measuring the local deviation of a given data point with respect to its neighbours. LOF shares some concepts with DBSCAN and OPTICS such as

neg: Outlier the cause. Some estimators are highly sensitive to outliers, notably estimation of covariance matrices. Even when a normal distribution model is appropriate to the data being analyzed, outliers are expected for large sample sizes and should not automatically be discarded if that is the case. The application should use a classification algorithm that is robust to outliers to model data with naturally occurring outlier points. Deletion of outlier data is a controversial practice frowned upon by many scientists and science instructors; while mathematical criteria provide an objective and quantitative method for data rejection, they do not make the practice more

neg: K-nearest neighbors algorithm larger the distance to the "k"-NN, the lower the local density, the more likely the query point is an outlier. To take into account the whole neighborhood of the query point, the average distance to the "k"-NN can be used. Although quite simple, this outlier model, along with another classic data mining method, local outlier factor, works quite well also in comparison to more recent and more complex approaches, according to a large scale experimental analysis. A confusion matrix or "matching matrix" is often used as a tool to validate the accuracy of "k"-NN classification. More robust statistical methods such

neg: Outlier variety of datasets, indicating that the causative mechanism for the data might differ at the extreme end (King effect). There is no rigid mathematical definition of what constitutes an outlier; determining whether or not an observation is an outlier is ultimately a subjective exercise. There are various methods of outlier detection. Some are graphical such as normal probability plots. Others are model-based. Box plots are a hybrid. Model-based methods which are commonly used for identification assume that the data are from a normal distribution, and identify observations which are deemed "unlikely" based on mean and standard deviation: It is proposed

neg: Quartile 40, 41 As there are an even number of data points, all three methods give the same results. There are methods by which to check for outliers in the discipline of statistics and statistical analysis. As is the basic idea of descriptive statistics, when encountering an outlier, we have to explain this value by further analysis of the cause or origin of the outlier. In cases of extreme observations, which are not an infrequent occurrence, the typical values must be analyzed. In the case of quartiles, the Interquartile Range (IQR) may be used to characterize the data when there may

neg: Cochran's C test (H): The sample variance of data series "j" is considered an outlier at significance level "α" if "C" exceeds the upper limit critical value C. C depends on the desired significance level "α", the number of considered data series "N", and the number of data points ("n") per data series. Selections of values for C have been tabulated at significance levels α = 0.01, α = 0.025, and α = 0.05. "C" can also be calculated from: Here: The C test can be generalized to include unbalanced designs, one-sided lower limit tests and two-sided tests at any significance level "α",

neg: OPTICS algorithm lead to quadratic complexity, since every neighborhood query returns the full data set. Even when no spatial index is available, this comes at additional cost in managing the heap. Therefore, formula_6 should be chosen appropriately for the data set. OPTICS-OF is an outlier detection algorithm based on OPTICS. The main use is the extraction of outliers from an existing run of OPTICS at low cost compared to using a different outlier detection method. The better known version LOF is based on the same concepts. DeLi-Clu, Density-Link-Clustering combines ideas from single-linkage clustering and OPTICS, eliminating the formula_6 parameter and offering performance

neg: Outlier (1982 edition) from "A Manual of Astronomy" 2:558 by Chauvenet.) Other methods flag observations based on measures such as the interquartile range. For example, if formula_3 and formula_4 are the lower and upper quartiles respectively, then one could define an outlier to be any observation outside the range: for some nonnegative constant formula_6. John Tukey proposed this test, where formula_7 indicates an "outlier", and formula_8 indicates data that is "far out". In the data mining task of "anomaly detection", other approaches are distance-based and density-based such as Local Outlier Factor (LOF), and most of them use the distance to the

neg: BIRCH discarding outliers. That is a point which is too far from its closest seed can be treated as an outlier. Given only the formula_14, the same measures can be calculated without the knowledge of the underlying actual values. In multidimensional cases the square root should be replaced with a suitable norm. BIRCH BIRCH (balanced iterative reducing and clustering using hierarchies) is an unsupervised data mining algorithm used to perform hierarchical clustering over particularly large data-sets. An advantage of BIRCH is its ability to incrementally and dynamically cluster incoming, multi-dimensional metric data points in an attempt to produce the best quality

neg: Cochran's C test test assumes a balanced design, i.e. the considered full data set should consist of individual data series that all have equal size. The C test further assumes that each individual data series is normally distributed. Although primarily an outlier test, the C test is also in use as a simple alternative for regular homoscedasticity tests such as Bartlett's test, Levene's test and the Brown–Forsythe test to check a statistical data set for homogeneity of variances. An even simpler way to check homoscedasticity is provided by Hartley's F test, but Hartley's F test has the disadvantage that it only accounts for

neg: Chauvenet's criterion Chauvenet's criterion In statistical theory, Chauvenet's criterion (named for William Chauvenet) is a means of assessing whether one piece of experimental data — an outlier — from a set of observations, is likely to be spurious. The idea behind Chauvenet's criterion is to find a probability band, centered on the mean of a normal distribution, that should reasonably contain all n samples of a data set. By doing this, any data points from the n samples that lie outside this probability band can be considered to be outliers, removed from the data set, and a new mean and standard deviation

neg: DFFITS to the other. Previously when assessing a dataset before running a linear regression, the possibility of outliers would be assessed using histograms and scatterplots. Both methods of assessing data points were subjective and there was little way of knowing how much leverage each potential outlier had on the results data. This led to a variety of quantitative measures, including DFFIT, DFBETA. DFFITS DFFITS is a diagnostic meant to show how influential a point is in a statistical regression. It was proposed in 1980. It is defined as the Studentized DFFIT, where the latter is the change in the predicted value

neg: Robust regression squares fit is that several outliers are masked because the estimate of residual scale is inflated, the scaled residuals are pushed closer to zero than when a more appropriate estimate of scale is used. The plots of the scaled residuals from the two models appear below. The variable on the "x" axis is just the observation number as it appeared in the data set. Rousseeuw and Leroy (1986) contains many such plots. The horizontal reference lines are at 2 and −2, so that any observed scaled residual beyond these boundaries can be considered to be an outlier. Clearly, the least

neg: Robust statistics two outliers prior to proceeding with any further analysis. However, in modern times, data sets often consist of large numbers of variables being measured on large numbers of experimental units. Therefore, manual screening for outliers is often impractical. Outliers can often interact in such a way that they mask each other. As a simple example, consider a small univariate data set containing one modest and one large outlier. The estimated standard deviation will be grossly inflated by the large outlier. The result is that the modest outlier looks relatively normal. As soon as the large outlier is removed, the estimated

neg: Chauvenet's criterion the probability) is 0.05×6 = 0.3. Because 0.3 < 0.5, according to Chauvenet's criterion, the measured value of 50 should be discarded (leaving a new mean of 10, with standard deviation 0.7). Another method for eliminating spurious data is called "Peirce's criterion". It was developed a few years before Chauvenet's criterion was published, and it is a more rigorous approach to the rational deletion of outlier data. Other methods such as Grubbs' test for outliers are mentioned under the listing for "Outlier". Deletion of outlier data is a controversial practice frowned on by many scientists and science instructors; while Chauvenet's

neg: Hodges–Lehmann estimator The Hodges–Lehmann statistic is the median of the "m" × "n" differences. For a population that is symmetric, the Hodges–Lehmann statistic estimates the population's median. It is a robust statistic that has a breakdown point of 0.29, which means that the statistic remains bounded even if nearly 30 percent of the data have been contaminated. This robustness is an important advantage over the sample mean, which has a zero breakdown point, being proportional to any single observation and so liable to being misled by even one outlier. The sample median is even more robust, having a breakdown point of 0.50.

neg: Noisy data collection and data preparation processes, where errors commonly occur. Noise has two main sources: errors introduced by measurement tools and random errors introduced by processing or by experts when the data is gathered. Improper Filtering can add noise, if the filtered signal is treate as if it were a directly measured signal. As an example, Convolution-type digital filters such a moving average can have side effects such as lags or truncation of peaks. Differentiating digital filters "amplify" random noise in the original data. Outlier data is data that appears to not belong in the data set. It can be caused

neg: K-nearest neighbors algorithm some of the data points are needed for accurate classification. Those data are called the "prototypes" and can be found as follows: A training example surrounded by examples of other classes is called a class outlier. Causes of class outliers include: Class outliers with "k"-NN produce noise. They can be detected and separated for future analysis. Given two natural numbers, "k>r>0", a training example is called a "(k,r)"NN class-outlier if its "k" nearest neighbors include more than "r" examples of other classes. Condensed nearest neighbor (CNN, the "Hart algorithm") is an algorithm designed to reduce the data set for "k"-NN

neg: Anomaly detection statistical definition of an outlier as a rare object, and many outlier detection methods (in particular unsupervised methods) will fail on such data, unless it has been aggregated appropriately. Instead, a cluster analysis algorithm may be able to detect the micro clusters formed by these patterns. Three broad categories of anomaly detection techniques exist. Unsupervised anomaly detection techniques detect anomalies in an unlabeled test data set under the assumption that the majority of the instances in the data set are normal by looking for instances that seem to fit least to the remainder of the data set. Supervised anomaly detection

neg: Point set registration to formula_7. In convergence, it is desired for the distance between the two point sets to reach a global minimum. This is difficult without exhausting all possible transformations, so a local minimum suffices. The distance function between a transformed model point set formula_17 and the scene point set formula_7 is given by some function formula_19. A simple approach is to take the square of the Euclidean distance for every pair of points: Minimizing such a function in rigid registration is equivalent to solving a least squares problem. However, this function is sensitive to outlier data and consequently algorithms based on

neg: Edgar Acuna In July 2001, Acuna was considered by the Hispanic Engineer & IT magazine as one of the scientists to watch. In July 2003, he was considered by the same magazine among the Hispanic Power Hitters in Technology and Business. In 2009, was selected as Fulbright Scholar to visit Peruvian Universities. Acuna is the author of "Statistical Analysis using Minitab" (written in Spanish) and published by Wiley and Sons in 2002. He has several publications in data pre-processing for data mining and machine learning. In particular, in handling of missing values, outlier detection, and feature selection. Acuna along with his research

neg: Kinetic data structure re-inserting each point into a "static" (traditional) data structure. However, such an approach is vulnerable to oversampling or undersampling, depending on what interval of time is used, and can also be wasteful of computational resources. The following general approach can be used to construct kinetic data structures: Certificate failures are referred to as "events". An event is considered internal if the property maintained by the kinetic data structure does not change when the event occurs. An event is considered external if the property maintained by the data structure changes when the event occurs. When using the certificates approach, there are

neg: Robust regression assumptions by the underlying data-generating process. In particular, least squares estimates for regression models are highly sensitive to (i.e. not robust against) outliers. While there is no precise definition of an outlier, outliers are observations which do not follow the pattern of the other observations. This is not normally a problem if the outlier is simply an extreme observation drawn from the tail of a normal distribution, but if the outlier results from non-normal measurement error or some other violation of standard ordinary least squares assumptions, then it compromises the validity of the regression results if a non-robust regression technique

neg: Hans-Peter Kriegel management, and access methods for high-dimensional data". He received the 2013 IEEE ICDM Research Contributions Award for his research on data mining algorithm such as DBSCAN, OPTICS, Local Outlier Factor and his work on mining high-dimensional data. He was also awarded the 2015 ACM SIGKDD Innovation Award for his contributions to data mining in clustering, outlier detection and high-dimensional data analysis, in particular for density-based approaches. DBSCAN also received the 2014 ACM SIGKDD test of time award. He is the most cited German researcher in databases and data mining. His current research is focused around correlation clustering, high-dimensional data indexing

neg: Least absolute deviations points. More generally, if there are "k" regressors (including the constant), then at least one optimal regression surface will pass through "k" of the data points. This "latching" of the line to the data points can help to understand the "instability" property: if the line always latches to at least two points, then the line will jump between different sets of points as the data points are altered. The "latching" also helps to understand the "robustness" property: if there exists an outlier, and a least absolute deviations line must latch onto two data points, the outlier will most likely not

neg: Mahalanobis distance the slope or coefficients of the regression equation. Mahalanobis distance is also used to determine multivariate outliers. Regression techniques can be used to determine if a specific case within a sample population is an outlier via the combination of two or more variable scores. Even for normal distributions, a point can be a multivariate outlier even if it is not a univariate outlier for any variable (consider a probability density concentrated along the line formula_26, for example), making Mahalanobis distance a more sensitive measure than checking dimensions individually. Mahalanobis distance The Mahalanobis distance is a measure of the distance between

neg: Outlier observations are far from the center of the data. Outlier points can therefore indicate faulty data, erroneous procedures, or areas where a certain theory might not be valid. However, in large samples, a small number of outliers is to be expected (and not due to any anomalous condition). Outliers, being the most extreme observations, may include the sample maximum or sample minimum, or both, depending on whether they are extremely high or low. However, the sample maximum and minimum are not always outliers because they may not be unusually far from other observations. Naive interpretation of statistics derived from data

neg: R-tree but due to the increased complexity, has not received much attention in practical applications so far. When data is organized in an R-tree, the neighbors within a given distance r and the k nearest neighbors (for any L-Norm) of all points can efficiently be computed using a spatial join. This is beneficial for many algorithms based on such queries, for example the Local Outlier Factor. DeLi-Clu, Density-Link-Clustering is a cluster analysis algorithm that uses the R-tree structure for a similar kind of spatial join to efficiently compute an OPTICS clustering. Data in R-trees is organized in pages, that can have

neg: Nerine or sparse. The first group (the evergreens) can then be considered to have three subgroups corresponding to DNA groups A, B and C but also by other characteristics. "N. marincowitzii" is an outlier being summer growing but narrow-leafed. The other outlier is "N. pusilla" which is narrow-leafed despite being summer growing. "N. duparquetiana" has at times been considered to be a synonym of "N. laticoma" but was restored to species status here. "N. huttoniae" is another species whose status is disputed, but here is treated (as Traub did) as a subspecies of "N. laticoma", a status subsequently confirmed. Two species

neg: St. Mary's Islands of this inference is based on the approach that the Felsic volcanics (rhyolites and Rhyodacites) of the St. Mary's Islands (SMI), Southern India, were originally interpreted as a distant outlier of the 66 Ma Deccan volcanic province of west–central India, comprising dominantly flood basalts. Later studies had dated it at 93 Ma by the K-Ar dating technique. Since the technique used was a simple use of an average of five out of six widely varying dates and arbitrary data selectivity chosen, the results were not considered reliable. A method of Ar–Ar (argon–argon dating) of the SMI volcanic yields is reportedly

neg: Outlier scientifically or methodologically sound, especially in small sets or where a normal distribution cannot be assumed. Rejection of outliers is more acceptable in areas of practice where the underlying model of the process being measured and the usual distribution of measurement error are confidently known. An outlier resulting from an instrument reading error may be excluded but it is desirable that the reading is at least verified. The two common approaches to exclude outliers are truncation (or trimming) and Winsorising. Trimming discards the outliers whereas Winsorising replaces the outliers with the nearest "nonsuspect" data. Exclusion can also be a consequence

neg: Goda Mountains Goda Mountains The Goda Mountains lie northwest of the Gulf of Tadjoura, Tadjoura Region in Djibouti. They rise to above sea level and are the nation's largest heavily vegetated area and is the second highest point in Djibouti. A part of the mountains is protected within the Day Forest National Park, Djibouti's only national park. The ecology of this landform is considered an isolated outlier of the Ethiopian montane forest ecological zone, an important island of forest in a sea of semi-desert. The Goda Mountains and the nearby Mabla Mountains are the last remaining refuge for the Djibouti francolin. It

neg: Polynesian outlier language family, which is sometimes called the Samoan-Outlier language family for this reason. It is a sub-branch of the Nuclear Polynesian languages. In some of these islands, the outlier population may also speak the local Melanesian or Micronesian language. A 1983 study analyzing the DNA of 2400 people in the Solomon Islands and Vanuatu have found markers which clearly distinguish the Polynesian outlier islands of the group. Of the four Polynesian outliers considered, Anuta was the most genetically distinct, followed by Rennell and Bellona. Tikopia showed more influence from the nearby Melanesian population. All indicate traces of inter-island population movements,

neg: Grubbs' test for outliers Grubbs' test for outliers Grubbs' test (named after Frank E. Grubbs, who published the test in 1950), also known as the maximum normalized residual test or extreme studentized deviate test, is a statistical test used to detect outliers in a univariate data set assumed to come from a normally distributed population. Grubbs' test is based on the assumption of normality. That is, one should first verify that the data can be reasonably approximated by a normal distribution before applying the Grubbs' test. Grubbs' test detects one outlier at a time. This outlier is expunged from the dataset and the test

neg: Tourism in the Arab world landform is considered an isolated outlier of the Ethiopian montane forest ecological zone, an important island of forest in a sea of semi-desert. Lake Assal (literally translated to "honey lake") is a crater lake in central-eastern Djibouti. It is a saline lake which lies 155 m (509 ft) below sea level in the Afar Triangle, making it the lowest point on land in Africa and the third lowest land depression on Earth after the Dead Sea and Sea of Galilee. There is no outflow from the lake and, due to high evaporation, the salinity level of its waters is ten

neg: Robust statistics infinity. One of the most important cases is distributional robustness. Classical statistical procedures are typically sensitive to "longtailedness" (e.g., when the distribution of the data has longer tails than the assumed normal distribution). Thus, in the context of robust statistics, "distributionally robust" and "outlier-resistant" are effectively synonymous. For one perspective on research in robust statistics up to 2000, see . A related topic is that of resistant statistics, which are resistant to the effect of extreme scores. Gelman et al. in Bayesian Data Analysis (2004) consider a data set relating to speed-of-light measurements made by Simon Newcomb. The data sets

neg: SKEW SKEW SKEW is the ticker symbol for the CBOE Skew Index, a measure of the perceived tail risk of the distribution of S&P 500 investment returns over a 30-day horizon. The index values are calculated and published by the Chicago Board Options Exchange (CBOE) based on current S&P 500 options market data. SKEW is similar to the VIX index, but instead of measuring implied volatility based on a normal distribution, measures an implied risk of future returns realizing outlier behavior. The index model defines such an outlier as two or more standard deviations below the mean, which would characterize a

neg: Shot grouping process of mechanically readjusting the gunsight’s point of aim to match the MPI is called sighting in or “zeroing” the aim. The size of the grouping is described as the diameter of the smallest circle that can be drawn to contain all the shots. A "flier" is a shot from the same shooting session that is farther from the general grouping, considered to be the outlier of the group. Such shots may be the result of an unexpected gust of wind causing drifting, a muscular flinch of the shooter at the time of the trigger pull, a problem with the

neg: Noisy data by human error such as transposing numerals, mislabeling, programming bugs, etc. If actual outliers are not removed from the data set, they corrupt the results to a small or large degree depending on circumstances. If valid data is identified as an outlier and is mistakenly removed, that also corrupts results. Fraud: Individuals may deliberately skew data to influence the results toward a desired conclusion. Data that looks good with few outliers reflects well on the individual collecting it, and so there may be incentive to remove more data as outliers, or make the data look smoother than it is. Noisy

neg: Melting point Open melting point data are also available from "Nature Precedings". High quality data mined from patents and also models developed with these data were published by Tetko "et al". Melting point The melting point (or, rarely, liquefaction point) of a substance is the temperature at which it changes state from solid to liquid. At the melting point the solid and liquid phase exist in equilibrium. The melting point of a substance depends on pressure and is usually specified at a standard pressure such as 1 atmosphere or 100 kPa. When considered as the temperature of the reverse change from liquid

neg: Local outlier factor estimate the density. By comparing the local density of an object to the local densities of its neighbors, one can identify regions of similar density, and points that have a substantially lower density than their neighbors. These are considered to be outliers. The local density is estimated by the typical distance at which a point can be "reached" from its neighbors. The definition of "reachability distance" used in LOF is an additional measure to produce more stable results within clusters. Let formula_2 be the distance of the object formula_3 to the "k"-th nearest neighbor. Note that the set of the

neg: Isotopic ratio outlier analysis IROA-based data files to identify and quantify all compounds of biological origin and to remove artifacts. Basic IROA is fundamentally an unbiased analysis. This leads to clean, high resolution data sets that clearly define the biological response of a biological system. This workflow utilizes both 95%- and 5%-C13 signatures. Like Stable Isotope Labeling by/with Amino acids in Cell culture (SILAC), cells are differentially labeled by growing them in heavy medium (medium in which all carbon sources are labeled with 95% C13) or light medium (medium in which all carbon sources are labeled with 5% C13). When the cells are growing

neg: Oracle Data Mining for data preparation steps prior to model building such as outlier treatment, discretization, normalization and binning (sorting in general speak) Users can access Oracle Data Mining through Oracle Data Miner, a GUI client application that provides access to the data mining functions and structured templates (called Mining Activities) that automatically prescribe the order of operations, perform required data transformations, and set model parameters. The user interface also allows the automated generation of Java and/or SQL code associated with the data-mining activities. The Java Code Generator is an extension to Oracle JDeveloper. An independent interface also exists: the Spreadsheet Add-In for

neg: Robust statistics mind two challenges. First, an outlier detection method that relies on a non-robust initial fit can suffer from the effect of masking, that is, a group of outliers can mask each other and escape detection. Second, if a high breakdown initial fit is used for outlier detection, the follow-up analysis might inherit some of the inefficiencies of the initial estimator. Robust statistics Robust statistics are statistics with good performance for data drawn from a wide range of probability distributions, especially for distributions that are not normal. Robust statistical methods have been developed for many common problems, such as estimating location,

neg: Josephson voltage standard the null meter can generally be modeled by a Gaussian process with one standard deviation on the order of 20 to 100 nV. There are, however, occasional noise spikes that do not fit this process and generate glitches in the formula_84 data that may lie 1 μV to 10 μV away from the well behaved data. An outlier test is used to detect and eliminate such data. After the collection of the first data set, the polarity of the unknown is reversed (formula_92), the bias is readjusted to select a step that minimizes formula_70, and a second set of data

neg: Corded Ware culture southern Poland and point to the period around 3000 BCE. However, subsequent review has challenged this perspective, instead pointing out that the wide variation in dating of the Corded Ware, especially the dating of the culture's beginning, is based on individual outlier graves, is not particularly in line with other archaeological data and runs afoul of plateaus in the radiocarbon calibration curve; in the one case where the dating can be clarified with dendrochronology, in Switzerland, Corded Ware is found for only a short period from 2750 BCE to 2400 BCE. Furthermore, because the short period in Switzerland seems to

neg: Robust statistics estimators apart from standard deviation (e.g., MAD) and Winsorization. In calculations of a trimmed mean, a fixed percentage of data is dropped from each end of an ordered data, thus eliminating the outliers. The mean is then calculated using the remaining data. Winsorizing involves accommodating an outlier by replacing it with the next highest or next smallest value as appropriate. However, using these types of models to predict missing values or outliers in a long time series is difficult and often unreliable, particularly if the number of values to be in-filled is relatively high in comparison with total record length.

neg: Student's t-test substantially non-normal and the sample size is small, the "t"-test can give misleading results. See Location test for Gaussian scale mixture distributions for some theory related to one particular family of non-normal distributions. When the normality assumption does not hold, a non-parametric alternative to the "t"-test can often have better statistical power. In the presence of an outlier, the t-test is not robust. For example, for two independent samples when the data distributions are asymmetric (that is, the distributions are skewed) or the distributions have large tails, then the Wilcoxon rank-sum test (also known as the Mann–Whitney "U" test) can

neg: Machine learning exist. Unsupervised anomaly detection techniques detect anomalies in an unlabeled test data set under the assumption that the majority of the instances in the data set are normal, by looking for instances that seem to fit least to the remainder of the data set. Supervised anomaly detection techniques require a data set that has been labeled as "normal" and "abnormal" and involves training a classifier (the key difference to many other statistical classification problems is the inherent unbalanced nature of outlier detection). Semi-supervised anomaly detection techniques construct a model representing normal behavior from a given normal training data set, and

neg: Outlier = 3. Outliers can have many anomalous causes. A physical apparatus for taking measurements may have suffered a transient malfunction. There may have been an error in data transmission or transcription. Outliers arise due to changes in system behaviour, fraudulent behaviour, human error, instrument error or simply through natural deviations in populations. A sample may have been contaminated with elements from outside the population being examined. Alternatively, an outlier could be the result of a flaw in the assumed theory, calling for further investigation by the researcher. Additionally, the pathological appearance of outliers of a certain form appears in a

neg: Watchung Outliers Palisades. It was determined later in the 20th century that the hill was composed of extrusive basalt with a minimum thickness of 125 ft (38 m). However, by 2011 it was known that the outlier, once considered to consist only of a single type of basalt, is more complex than originally thought. The outermost part of the outlier is composed of an elongated arc of Orange Mountain Basalt. Lying within this arc and stratigraphically above it is a band of sedimentary rock from the Feltville Formation. Finally, lying within and above the band of sedimentary rock is a central area

neg: Trauma Quality Improvement Program discharge disposition of “death” or hospital discharge disposition of “expired,” as well as the prevalence of inpatient complications. Data reports were created and distributed to each participating trauma center in June 2009. Results were consistent with the previous year's baseline findings. Cohorts of similar verified trauma centers had differences in risk-adjusted mortality rates with a large amount of variance between low-outlier and high-outlier trauma centers. The aggregate group had a relative risk-mortality of 3.3, while the single-system trauma cohort had a mortality 5.9 times higher for high-outlier facilities. The TQIP pilot results gave each trauma center feedback regarding their trauma

neg: Primate city Primate city A primate city (Latin: "prime, first rank") is the largest city in its country or region, disproportionately larger than any others in the urban hierarchy. A "primate city distribution" is a rank-size distribution that has one very large city with many much smaller cities and towns, and no intermediate-sized urban centres: a King effect, visible as an outlier on an otherwise linear graph, when the rest of the data fit a power law or stretched exponential function. The "law of the primate city" was first proposed by the geographer Mark Jefferson in 1939. He defines a primate city

neg: Primate city and the number of functions offered. Higher functions and population will result in higher dominance. Primate city A primate city (Latin: "prime, first rank") is the largest city in its country or region, disproportionately larger than any others in the urban hierarchy. A "primate city distribution" is a rank-size distribution that has one very large city with many much smaller cities and towns, and no intermediate-sized urban centres: a King effect, visible as an outlier on an otherwise linear graph, when the rest of the data fit a power law or stretched exponential function. The "law of the primate city"

neg: Random sample consensus Random sample consensus Random sample consensus (RANSAC) is an iterative method to estimate parameters of a mathematical model from a set of observed data that contains outliers, when outliers are to be accorded no influence on the values of the estimates. Therefore, it also can be interpreted as an outlier detection method. It is a non-deterministic algorithm in the sense that it produces a reasonable result only with a certain probability, with this probability increasing as more iterations are allowed. The algorithm was first published by Fischler and Bolles at SRI International in 1981. They used RANSAC to solve the

neg: Ddrescue from failing drives causing them as little additional damage as possible. GNU ddrescue is considered to have the most sophisticated implementation of a block-size-changing algorithm in free and open source software, and is considered an essential data recovery tool. If one uses the "codice_3" (previously called "logfile") feature of ddrescue, the data is rescued very efficiently (only the needed blocks are read). One can also interrupt the rescue at any time and resume it later at the same point. ddrescue does not write zeros to the output when it finds bad sectors in the input, and does not truncate the

neg: Machine learning or errors in a text. Anomalies are referred to as outliers, novelties, noise, deviations and exceptions. In particular, in the context of abuse and network intrusion detection, the interesting objects are often not rare objects, but unexpected bursts in activity. This pattern does not adhere to the common statistical definition of an outlier as a rare object, and many outlier detection methods (in particular, unsupervised algorithms) will fail on such data, unless it has been aggregated appropriately. Instead, a cluster analysis algorithm may be able to detect the micro-clusters formed by these patterns. Three broad categories of anomaly detection techniques

neg: Data quality of data found. For example, making supply chain data conform to a certain standard has value to an organization by: 1) avoiding overstocking of similar but slightly different stock; 2) avoiding false stock-out; 3) improving the understanding of vendor purchases to negotiate volume discounts; and 4) avoiding logistics costs in stocking and shipping parts across a large organization. For companies with significant research efforts, data quality can include developing protocols for research methods, reducing measurement error, bounds checking of data, cross tabulation, modeling and outlier detection, verifying data integrity, etc. There are a number of theoretical frameworks for understanding data

neg: Robust statistics missing values. The Kohonen self organising map (KSOM) offers a simple and robust multivariate model for data analysis, thus providing good possibilities to estimate missing values, taking into account its relationship or correlation with other pertinent variables in the data record. Standard Kalman filters are not robust to outliers. To this end have recently shown that a modification of Masreliez's theorem can deal with outliers. One common approach to handle outliers in data analysis is to perform outlier detection first, followed by an efficient estimation method (e.g., the least squares). While this approach is often useful, one must keep in

neg: Watchung Outliers vent from which the magma of the Palisades Sill erupted to the surface to form the Watchung flood basalts. Union Hill is located near the village of Suffern in Ramapo, New York, about three and a half miles north of the Watchung Mountains, or about halfway between the Ladentown Outlier and Campgaw Mountain. The hill, primarily composed of trap rock, originally rose 150 feet above the surrounding terrain, but extensive quarrying has since obliterated most of the hill except for the slopes of its perimeter. Like the Ladentown Outlier, Union Hill was once considered to be an extension of the

neg: Structural health monitoring on the extracted features to quantify the damage state of the structure. The algorithms used in statistical model development usually fall into three categories. When data are available from both the undamaged and damaged structure, the statistical pattern recognition algorithms fall into the general classification referred to as supervised learning. Group classification and regression analysis are categories of supervised learning algorithms. Unsupervised learning refers to algorithms that are applied to data not containing examples from the damaged structure. Outlier or novelty detection is the primary class of algorithms applied in unsupervised learning applications. All of the algorithms analyze statistical distributions

neg: Adaptive reuse can rank their existing buildings as per their adaptive reuse potential and this data can be used by government authorities at any point in time. An adaptive reuse score of 50% or above is considered high. A low ARP score is anything below 20%. Anything between that range is considered moderate. Shen and Langston devised this concept of ARP as “rising from zero to its maximum score at the point of its useful life, and then falling back to zero as it approaches physical life”. When the “current building age” is identified to be close to or end of its

neg: Isotopic ratio outlier analysis these groups are labeled using an isotopically defined growth media in which all of the carbon components in the experimental and control samples are replaced with randomly and universally enriched 5% or 95%, respectively. After at least 5 subsequent cell divisions, the experiment group is exposed to a stressor (chemical, genetic, environmental, etc.) When the experiment has concluded, the experimental sample is mixed with the control sample and analyzed using mass spectrometry. The admixing of samples increases data quality as sample-to-sample variance is reduced and the identification of all biological compounds is enhanced. ClusterFinder software fully automates the processing of

neg: Machine learning has already been built, a new training example is associated with the class that is best sparsely represented by the corresponding dictionary. Sparse dictionary learning has also been applied in image de-noising. The key idea is that a clean image patch can be sparsely represented by an image dictionary, but the noise cannot. In data mining, anomaly detection, also known as outlier detection, is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data. Typically, the anomalous items represent an issue such as bank fraud, a structural defect, medical problems

neg: Chauvenet's criterion criterion provides an objective and quantitative method for data rejection, it does not make the practice more scientifically or methodologically sound, especially in small sets or where a normal distribution cannot be assumed. Rejection of outliers is more acceptable in areas of practice where the underlying model of the process being measured and the usual distribution of measurement error are confidently known. Chauvenet's criterion In statistical theory, Chauvenet's criterion (named for William Chauvenet) is a means of assessing whether one piece of experimental data — an outlier — from a set of observations, is likely to be spurious. The idea

neg: Fraser Filter Fraser Filter A Fraser Filter, named after Douglas Fraser, is typically used in geophysics when displaying VLF data. It is effectively the first derivative of the data. If formula_1 represents the collected data then formula_2 is the average of two values. Consider this value to be plotted between point 1 and point 2 and do the same with points 3 and 4: formula_3 If formula_4 represents the space between each station along the line then formula_5 is the Fraser Filter of those four values. Since formula_6 is constant, it can be ignored and the Fraser Filter considered to be formula_7.

neg: Fraser Filter Fraser Filter A Fraser Filter, named after Douglas Fraser, is typically used in geophysics when displaying VLF data. It is effectively the first derivative of the data. If formula_1 represents the collected data then formula_2 is the average of two values. Consider this value to be plotted between point 1 and point 2 and do the same with points 3 and 4: formula_3 If formula_4 represents the space between each station along the line then formula_5 is the Fraser Filter of those four values. Since formula_6 is constant, it can be ignored and the Fraser Filter considered to be formula_7.

neg: Dixon's Q test we cannot be 95% sure. McBane notes: Dixon provided related tests intended to search for more than one outlier, but they are much less frequently used than the r or Q version that is intended to eliminate a single outlier. This table summarizes the limit values of the two-tailed Dixon's Q test. Dixon's Q test In statistics, Dixon's "Q" test, or simply the "Q" test, is used for identification and rejection of outliers. This assumes normal distribution and per Robert Dean and Wilfrid Dixon, and others, this test should be used sparingly and never more than once in a data

neg: Softmax function of the action with the highest expected reward tends to 1. Sigmoidal or Softmax normalization is a way of reducing the influence of extreme values or outliers in the data without removing them from the dataset. It is useful given outlier data, which we wish to include in the dataset while still preserving the significance of data within a standard deviation of the mean. The data are nonlinearly transformed using one of the sigmoidal functions. The logistic sigmoid function: The hyperbolic tangent function, tanh: The sigmoid function limits the range of the normalized data to values between 0 and 1.

neg: Peltarion Synapse of ways. Data is imported through the use of "format" components. The standard release includes format components for reading and writing data from CSV (text) files, SQL databases, images and XML. The imported data can be visualized through visualizer components and filters can be applied to the data. The filter components range from simple data rearrangement to more advanced FFT and outlier removal filters. The visualizers include a variety of plots and grids, which can be interconnected and branched out to perform complex data mining tasks. In design mode components are linked to construct a topology. Linked components enable a

neg: DOME project of degraded pattern quality, outlier detection, object classification and image formation. Moving data from the collectors to the process facilities are traditionally bogged down due to high latency I/O, low bandwidth connections and data is often multiplied along the way due to lack of purposeful design of the communication network. This research project will try to reduce latency to a minimum and design the I/O systems so data will be written directly into the processing engines on an exascale computer design. The first phase will identify system bottlenecks, and investigate Remote direct memory access (RDMA). The second phase will investigate

neg: Data analysis also be identified through a variety of analytical techniques. For example, with financial information, the totals for particular variables may be compared against separately published numbers believed to be reliable. Unusual amounts above or below pre-determined thresholds may also be reviewed. There are several types of data cleaning that depend on the type of data such as phone numbers, email addresses, employers etc. Quantitative data methods for outlier detection can be used to get rid of likely incorrectly entered data. Textual data spell checkers can be used to lessen the amount of mistyped words, but it is harder to tell

neg: Outlier high skewness and that one should be very cautious in using tools or intuitions that assume a normal distribution. A frequent cause of outliers is a mixture of two distributions, which may be two distinct sub-populations, or may indicate 'correct trial' versus 'measurement error'; this is modeled by a mixture model. In most larger samplings of data, some data points will be further away from the sample mean than what is deemed reasonable. This can be due to incidental systematic error or flaws in the theory that generated an assumed family of probability distributions, or it may be that some

neg: Anterior commissure and Parsons criticized this research, noting that 27 of the 33 homosexual males fell within the range of heterosexual males in the study. However, because range is defined only by the two most extreme data points in a group, the existence of a single heterosexual male with an exceptionally large anterior commissure for his group (an outlier) would cause this large range irrespective of the data from the rest of the individuals in the group. This individual's existence would not change the fact that the groups on average were quite different from one another, and that these differences were statistically

neg: WavPack considered in choosing the better of the two available codes (which reduces noise another 1 dB). No floating-point arithmetic is used in WavPack's data path because, according to the author, integer operations are less susceptible to subtle chip-to-chip variations that could corrupt the lossless nature of the compression (the Pentium floating point bug being an example). It is possible that a lossless compressor that used floating-point math could generate different output when running on that faulty Pentium. Even disregarding actual bugs, floating-point math is complicated enough that there could be subtle differences between "correct" implementations that could cause trouble for

neg: Watchung Outliers Outlier is that it is surrounded by a ridge of intrusive diabase on its eastern side. The diabase, which lies stratagraphically below the basalt, is thought to have been intruded into the syncline containing the Jacksonwald Outlier during or after some degree of folding had affected local strata. The diabase is therefore considered to constitute a phacolith. Watchung Outliers The Watchung Outliers include six areas of isolated low hills and rock outcrops of volcanic and sedimentary origin in the U.S. states of New York, New Jersey, and Pennsylvania. These geologic outliers are primarily diminutive and detached remnants of the Triassic/Jurassic

neg: Mesa Verde National Park mud jacal-style buildings that marked the Pueblo I Period toward masonry construction, which had been utilized in the region as early as 700, but was not widespread until the 11th and 12th centuries. The expansion of Chacoan influence in the Mesa Verde area left its most visible mark in the form of Chaco-style masonry great houses that became the focal point of many Mesa Verdean villages after 1075. Far View House, the largest of these, is considered a classic Chaco "outlier," on which construction likely began between 1075 and 1125, although some archaeologists argue that it was begun as early

neg: Metadata repository though it only reflects the structures consumed from various systems. Repository has additional functionalities compared with registry. Metadata repository not only stores metadata like Metadata registry but also adds relationships with related metadata types. Metadata when related in a flow from its point of entry into organization up to the deliverables is considered as the lineage of that data point. Metadata when related across other related metadata types is called linkages. By providing the relationships to all the metadata points across the organization and maintaining its integrity with an architecture to handle the changes, metadata repository provides the basic material

neg: Standard Project Hurricane David Daniel, the chairman of a panel reviewing the corps' investigation, said in an interview, "It was not a terribly sophisticated or detailed analysis by today's standards." The creators of the standard project hurricane, in an attempt to find a representative storm, actually excluded the fiercest storms from the database. "Storms like Hurricane Camille in 1969 were taken out of the data set as lying too far out of the norm; the Berkeley researchers noted that 'excluding outlier data is not appropriate in the context of dealing with extreme hazards.' Also, the calculations of the cost-benefit ratio did not take

neg: Moving average factor on each datum point "p", "p", etc., decreases exponentially: where since formula_21. It can also be calculated recursively without introducing the error when initializing the first estimate (n starts from 1): This is an infinite sum with decreasing terms. The question of how far back to go for an initial value depends, in the worst case, on the data. Large price values in old data will affect the total even if their weighting is very small. If prices have small variations then just the weighting can be considered. The power formula above gives a starting value for a particular

neg: Isotopic ratio outlier analysis The ClusterFinder software achieves a data reduction of complex raw data to concise, high value information by performing these steps: characterization of peaks according to source, artifactual-no label, experimental, or control, removal of artifacts, alignment and pairing of remaining peaks across scans, pair normalization and identification, and determination of the relative / ratios of analytes in each sample. The IROA approach is applicable to measuring an organism's biological response to any stressor, such as disease, environment, drugs or toxins. As an example IROA was used to label biosynthetic pathways, specifically outlining the glutathione pathway, during the fermentation process of "Saccharomyces

neg: Inliers and outliers (geology) Monmouthshire where Silurian age rocks are upfolded amidst Old Red Sandstone rocks of Devonian age. A similar outcrop pattern which results from movement on a thrust fault followed by erosion may be termed a window. Conversely an outlier is an area of younger rock completely surrounded by older rocks. An outlier is typically formed when sufficient erosion of surrounding rocks has taken place to sever the younger rock's original continuity with a larger mass of the same younger rocks nearby. One example from Great Britain is the mass of Triassic sandstone around the Staffordshire town of Leek. This is isolated

neg: Darkover series is an outlier when considered in light of the evidence of the series as a whole. Books describing this era: Short stories describing this era: Bradley's books constantly refer back to the Ages of Chaos, but few books are actually set in this era. In this era, the descendents of the original colonists have organized themselves into a feudal-type society, with "laran" (psionic) abilities as the determiner of which individuals are part of the aristocracy and which are commoners. This period is marked by incredible creativity, the development of "laran"-based technology and weaponry, and the creation of the system of

neg: Dixon's Q test Dixon's Q test In statistics, Dixon's "Q" test, or simply the "Q" test, is used for identification and rejection of outliers. This assumes normal distribution and per Robert Dean and Wilfrid Dixon, and others, this test should be used sparingly and never more than once in a data set. To apply a "Q" test for bad data, arrange the data in order of increasing values and calculate "Q" as defined: Where "gap" is the absolute difference between the outlier in question and the closest number to it. If "Q" > "Q", where "Q" is a reference value corresponding to the

neg: Determining the number of clusters in a data set problem altogether. The correct choice of "k" is often ambiguous, with interpretations depending on the shape and scale of the distribution of points in a data set and the desired clustering resolution of the user. In addition, increasing "k" without penalty will always reduce the amount of error in the resulting clustering, to the extreme case of zero error if each data point is considered its own cluster (i.e., when "k" equals the number of data points, "n"). Intuitively then, "the optimal choice of "k" will strike a balance between maximum compression of the data using a single cluster, and

neg: Data structure alignment is only allowed to contain addresses that are "n"-byte aligned, otherwise it is said to be "unaligned". A memory pointer that refers to a data aggregate (a data structure or array) is "aligned" if (and only if) each primitive datum in the aggregate is aligned. Note that the definitions above assume that each primitive datum is a power of two bytes long. When this is not the case (as with 80-bit floating-point on x86) the context influences the conditions where the datum is considered aligned or not. Data structures can be stored in memory on the stack with a static

neg: Statistical graphics selection, relationship identification, factor effect determination, and outlier detection. In addition, the choice of appropriate statistical graphics can provide a convincing means of communicating the underlying message that is present in the data to others. Graphical statistical methods have four objectives: If one is not using statistical graphics, then one is forfeiting insight into one or more aspects of the underlying structure of the data. Statistical graphics have been central to the development of science and date to the earliest attempts to analyse data. Many familiar forms, including bivariate plots, statistical maps, bar charts, and coordinate paper were used in

neg: Cochran's C test for any number of data series "N", and for any number of individual data points "n" in data series "j". Cochran's C test In statistics, Cochran's C test, named after William G. Cochran, is a one-sided upper limit variance outlier test. The C test is used to decide if a single estimate of a variance (or a standard deviation) is significantly larger than a group of variances (or standard deviations) with which the single estimate is supposed to be comparable. The C test is discussed in many text books and has been recommended by IUPAC and ISO. Cochran's C test