Data preparation in Machine Learning is the initial task, which will widely influence the eventual result. The accuracy of modeling will directly depend on the correctness and completeness of the collected data set. In addition, discrepancies in data can lead to the failure of the whole ML analysis. If you are looking for a professional ML consultation, ask for free expert advice here.
The preparation of the raw data collection often takes place in stages, namely:
1) Assessment of the completeness and quality of the data.
This stage is essential and often the most time-consuming. The main problems at this stage are:
– Discrepancies in the types of source data.
Often different formats are mixed, which cannot be matched with each other without manual correction.
– Differences in the size of the source data array.
Different data sets may include a different number of fields and discrepancies in the names or metric systems. When combining this information into a single array, there will be no consistency required for the completeness of the data analysis. As a result, specific problems will also be traced.
– A mixture of information.
Several sources may contain different fields that generally mean the same thing, for example, man and male. To structurize the data correctly, it is necessary to ensure that the information used as a descriptor is identical (in this example, male gender).
– Single outliers that are included in the data set.
There are exceptions in all rules; emissions are just such exceptions. Sometimes, in rare cases, there are torrential rains in the desert. In machine learning, it is necessary to investigate outliers in order to ultimately understand whether these outliers are an error during the collection of initial data or relate to exceptional phenomena and need to be taken into account when processing the data.
– The absence of certain data that will affect the final array of information.
In collecting information, specific data may be lost or simply missing, which will have a certain impact on the final forecast. This occurs for several reasons at once, the main of which are: the human factor (errors) and software failures. We need eliminated these gaps. Otherwise, it’s impossible to create the ideal base that is required for machine learning.
2) Cleaning up the raw data
The main purpose of this task is to arrange the data that will be used in machine learning. It consists of several stages, namely:
– Collecting missing information.
A fairly common problem that is solved quite simply. With an extensive array of source data, you can simply delete unnecessary (superfluous) fields. In certain cases, you can add information based on logical conclusions (this method is suitable only for inaccurate models).
– Exclusion of noisy data.
Data that is of no use to machine learning is noise. These data mainly include:
– Duplicate data,
– Sets of information that are of no use for research,
– Extra fields that do not affect variables.
When cleaning the raw data, it is necessary to separate emissions from noise because, in cases of deletion, the probability of certain aspects will change, which will ultimately negatively affect machine learning.
3) Data conversion
In most cases, the transformation of the source data takes place directly in the process of smoothing and cleaning. Still, in some cases, the data obtained must be brought to a single format in order for the machine to recognize and learn them.
For example, convert meters, kilometers, and centimeters directly into a single format (can be miles or meters – just you need to unify all the data to the same form).
There are several formats for data conversion, namely:
– Aggregation
In this variant, all the source data are combined into one and presented directly in a single format.
The critical point, in this case, is the amount of high-quality source data because the more of them, the more accurate the machine learning results will be.
– Normalization
This method differs from aggregation, and its success depends directly on the range of source data. It is quite challenging to compare data when their range is too large or too small.
– Function selection
Feature selection is the selection of variables in the data that are working as the best predictors for the variable we want to forecast. Of course, the more features, the more complicated the classifier process will be, and to achieve better results, we need large datasets.
– Discreteness
This process is defined as the process of converting attribute values of continuous data into a final set of intervals with the minor data loss.
– Generation of a hierarchy of concepts
When using the hierarchy of concepts method, it is possible to create a certain scale between existing attributes where this information is missing. For example, in cases where there is information about a certain location indicating the street, city, and other data, but with the absence of a specific hierarchy, this method is perfect for converting the original information.
– Generalization
This method allows you to change low-level functions to higher ones.
4) Compression of the received data
When working with large amounts of information, it is problematic to find exactly what is needed. Reducing the amount of initial information will speed up the process of data analysis. It is simply impossible to completely compress the array in certain cases, but we can reduce it to a representative sample.
There are certain methods of data compression, namely:
– Selection of the attribute function
The transformation of certain information can be used to reduce the quantity of data for analysis. Creating a new function that will combine certain information to increase the efficiency of data mining is called attribute selection.
– Dimension reduction
Data sets that are used to solve real-world problems have a huge number of functions. We can sacrifice certain of them to increase the speed of data processing (it is worth considering only those that will not have a negative impact on quality).
– Decrease in the number of
There are two methods of population reduction: parametric and nonparametric.
The parametric method often uses regression. Nonparametric data is stored thanks to histograms, data sampling, and aggregation.
