Precipitation prediction using recurrent neural networks and long short-term memory

ABSTRACT


INTRODUCTION
Indonesia is between the Indian Ocean and the Pacific Ocean and two continents, especially Asia and Australia, so the climate often changes and is influenced by many factors.Other phenomena, such as El-Nino Southern Oscillation (ENSO), add to the complexity of the environment [1].Precipitation is a part of the climate that has very complicated.Its characteristic in every area is indeed different from one another.It is affected by many factors like geographic, topographic, and monographic.Island structure and orientation have not been calculated.As a result, the precipitation distribution patterns are not prevalent in each area with others in a broad scope.Besides that, ENSO influences almost 70% of the change of precipitation in Indonesia in the Maritime Continent [2].In meanwhile, other parts of the world marked by the displacement of "warm pool" and cloud formation ordinary occurred in Indonesia sea go easterly to the middle of the Pacific Ocean.This phenomenon can degrade the precipitation in several parts in the Pacific area so that it runs into drought [3].With its dynamism, precipitation could not be discovered precisely.
Precipitation is an essential phenomenon in the climate system, which is chaotic and greatly affects all aspects of life, such as the availability of water resources, urban planning, agriculture, and financial security.Over the past few years, research on precipitation prediction is overgrowing.Many models have been could improve the accuracy of precipitation prediction [4].Meanwhile, the life cycle that forms specific patterns  ISSN: 1693-6930 TELKOMNIKA Telecommun Comput El Control, Vol. 18, No. 5, October 2020: 2525 -2532 2526 depends on money and time [5].Indeed, daily rainfall is random.But in the long run, have a particular model that can be identified.It's just also paying attention to many factors that are uncertain, such as climate zones, sunspots, tides, atmospheric circulation, and other human activity factors [6,7].
Climate parameter data provided by the Meteorology, Climatology, and Geophysics Agency (BMKG) daily, except for precipitation produced every hour of each station.The use of time and prediction location area, closely related to the use of usage.Based on the territory of use, some studies use a global area [8] while other studies are predictive of specific regions [9].Meanwhile, climate prediction has a time frame, such as daily [8] of solar forecasting, hours of wind speed [9], hours of temperature [10], multi-step time of wind speed [11], extreme climate every day [5].Other research estimated rainfall in a short time or hours [12], days [13], weeks [14].Some research prefers precipitation prediction in periods monthly with ENSO [15] and yearly [16] to help in proper agricultural planning.This study compared the estimated weekly and monthly precipitation models.
Weather forecasting is a new research problem because its application is extensive in many sectors, such as agriculture to flight navigation.The challenge of climate prediction is to choose the right variables and data sets and to choose representative models to be able to explore hidden structural patterns in a large dataset [15].Unfortunately, it is not a convenient case.Precipitation is a very complicated event because it happens randomly and depends on numerous factors like temperature, humidity, wind speed, and cloud pressure.Besides that, the dependent variables which are possible to affect precipitation are not constant even it cannot be sure how many factors may impact the rainfall.It makes the input parameters to the model may not adequate to predict precipitation precisely [17].Climate forecasting made attention to many researchers of various backgrounds due to its effect on global human life.The support of computer technology and accessibility to obtain big data of weather observation recently made many researchers are encouraged to learn more about the pattern in the large dataset of weather prediction.It can predict weather forecasting using machine learning.The method makes possible learning the pattern of precipitation with other variables in time series before.Regression problems provide some challenging research in the field of machine learning, including weather data.Rainfall is a prime example because it shows unique characteristics with high volatility and chaotic patterns.Therefore the machine learning method can outperform other methods [4].
Prediction variables of climate which may not be clearly understood, traditional linear forecasting techniques are ill-equipped to handle, often producing unsatisfactory results.Previous research using statistical bias correction on the output of the daily climate model in Europe [18] improved to see the relation between ECMWF and the Meteorology, Climatology, and Geophysics Agency (BMKG) [19].The research resulted in the value of the transfer function formed from the bias correction process can be used to improve the distribution of the 2016 rainfall prediction on the island of Bali, to obtain a better prediction.In meanwhile, some research increasingly resorts to techniques that are heuristic and non-linear.Such methods use neural network models [20] with machine-learning, regression, and clustering.Other research used dynamic regional combined short-term rainfall forecasting approach (DRCF) to improve multilayer perceptron and PCA.The study gave accuracy 75-92% but depended on the number of MLP [21].
Weather data prediction has its characteristics, which depend on the variability of these variables.Rainfall is very variable compared to solar radiation, wind speed, and temperature.Of course, this condition has an impact on prediction accuracy, as previous studies predicting weather and temperature provide better accuracy than real ones so that they can be used in real-time [16].Other research developed to predict temperature and humidity [22].Deep learning is new computing in data mining and machine learning [6].A neural network with deep architectures has become a kind of powerful tool to retrieve the high-level abstract features of big data.Some methods that are often used in deep learning are convolutional neural network (CNN), which convolutes with a fixed size kernel.Weather data can be viewed as imagery, so it can be resolved with CNN [23].However, for a limited image segment, it indeed will collide with memory limitations, so other methods are needed.One way that can be used is LSTM for rainfall prediction [24].Using CNN can be modified in one dimension, for example, for sunlight prediction [25], and predict precipitation [26].
Meanwhile, for time series often use RNN.The uniqueness of the RNNs is the feedback connection, which conveys interference information at the previous input that will be accommodated to the following facts.The RNN can study sequential or time-varying patterns so that it is tools in modeling intricate weather data patterns with accurate multi-step estimates [27].This research predicts rainfall in the Bandung area, which is the center of the basin, which has a height of 791m above sea level (ASL).The highest point is in the North with an altitude of 1050 m above sea level, and the lowest point is in the south with an elevation of 675 m above sea level.The area surrounded by mountains forms the city of Bandung into a kind of basin (Bandung Basin).The surrounding mountain climate significantly affects the Bandung city climate.However, in recent years, the temperature has been increasing, and the rainy season becomes more prolonged than usual.In past years, the rainy season is more intensive happening in Bandung.Naturally, Bandung is quite a cool area.During TELKOMNIKA Telecommun Comput El Control  Precipitation prediction using recurrent neural networks and… (Mishka Alditya Priatna) 2527 the year 2012, recorded that the highest temperature in Bandung reached 30.9•C, which occurred in September, and the lowest temperature in Bandung in 2012 was 17.4•C that happened in July.
This paper proposed a precipitation prediction model of the Bandung region using precipitation, humidity, temperature, solar radiation of 36-years before.The model using RNNs with long short-term memory (LSTM) to predict precipitation.The uniqueness of the RNNs is the feedback connection, which conveys interference information at the previous input that will be accommodated to the next data.Machine learning used this model Network based on consecutive time with prior climate data, so produced rainfall prediction.The input variables of machine learning are minimum temperature, maximum temperature, average temperature, relative humidity, duration of sun radiation, average wind speed, maximum wind speed, and precipitation to predict rainfall in a certain period.

RESEARCH METHOD 2.1. Data set
Weather data provided by the Indonesian Agency for Meteorological, Climatological, and (BMKG) from 1981 to 2017.This research using the Bandung city region in the analysis.Datasets provided consists of eight variables (minimum temperature, maximum temperature, average temperature, relative humidity, duration of sun radiation, average wind speed, maximum wind speed, and precipitation) of 36 years .This research used two periods of precipitation prediction, mainly weekly and monthly configurations.The values of climate variables in that period are the daily averages, minimum or maximum.There are four scenarios in the experiment, particularly, variations in the amount of training data (10 years and five years), and the duration of the predictions (monthly and weekly) with details: − 10-years: annual (432 data sets, overlap 11 months) − 10-years: monthly (1872 data sets, overlap 50 weeks) − 5-years: annual (864 data sets, overlap 11 months) − 5-years: monthly (3744 data sets, overlap 50 weeks) All of the models, 75% is used for training data, and 25% is used for non-training or test data.

Precipitation prediction model
The design of precipitation prediction using RNN and LSTM is shown in Figure 1.Prediction chooses one of ten classes with a specific interval.Past research used multivariates for rainfall prediction [28] with RNN and LSTM.Sometimes, climate data has missing or lost observation.Therefore, the data need to prepare automatically before the next process.Some solution is an interpolation of some available data, multivariate [29], or predict the missing using refinement function [10].Meanwhile, BMKG provides daily data.Therefore, weekly and monthly predictions need to convert daily or weekly data.Some studies used the average value in this time frame so that it reflects its projections [30].This research used various variables, i.e., minimum temperature, maximum temperature, average temperature, relative humidity, duration of sun radiation, average wind speed, maximum wind speed, and precipitation, which have different units.Therefore, all datasets before entering the RNN are normalized earlier [18,25] using (1).The normalization takes the maximum and minimum values on the 0-1 scale.
Based on the forecast period, it is divided into two models, i.e., weekly and monthly.The results of the study are then compared and can be utilized for their individual needs.When used for flood disaster management, the selection of the week is more appropriate, taking into account the carrying capacity of soil absorption.While the range of the planting season can use a monthly period, RNN can adapt to flexible classes [29].It can true period or pseudo period.In the meantime, we used rainfall predictions with a certain range of 10 pre-determined classes, i.e. < 60 mm, 60-120, 120-180, 180-240, 240-300, 300-360, 360-420, 420-480, 480-540, and > 540 mm as shown in Figure 1.Then go to the second step with RNN.The dropout layers used to minimize the number of input neurons 0.5 probability that input neuron to the next step is 480.The third step is LSTM layer 2, with the input dropout layer using (2)- (7).The fourth step is the dense layer using the sigmoid function, where the final result from the previous is entered into (1) to produce a new weight.

Recurrent neural networks
Deep learning techniques have been successfully applied to solve many problems in climate and geoscience using massive-scaled observed and modeled data [31].One method of deep learning is RNN.Previous research proposed training three models of deep-learning: RNN, conditional restricted boltzmann machine (CRBM), and CNN using ENSO and Weather dataset [15].The best accuracy was RNN until 84% of  ISSN: 1693-6930 TELKOMNIKA Telecommun Comput El Control, Vol. 18, No. 5, October 2020: 2525 -2532 2528 precipitation forecasting using a deep belief network called DBNPF [6].The research established the characteristics of environmental factors and future precipitation.Other studies proposed to predict the extreme weather using Artificial Neural Network and Support Vector Machines [32].
RNN works resemble the workings of the brain, such as sending and receiving information.Similarly, the work of the RNN brain can also send and receive data from a neuron to another neuron.In general, the human brain is used to make decisions, and in the process of making a rational decision, often takes into account the past.RNN is essentially an artificial neural network that uses recurrence by utilizing past data.RNN can predict a situation in the future [15], and can also classify [33].RNN has an architecture that can be used for data in the form of sequence or list, as shown in Figure 2. RNN is a modification of the Feedforward Neural Network with the characteristic of using feedback from output to input.
In the Architecture of RNNs, there are several connections from one neuron to one of the next neurons.RNN is processed in a sequence of time.So that each information has a relationship with one another.This way makes the RNN has a memory that functions to remember the results of the previous process that will be TELKOMNIKA Telecommun Comput El Control  Precipitation prediction using recurrent neural networks and… (Mishka Alditya Priatna) 2529 used in the next process.However, when the RNN processes quite a lot of data, it has difficulty in maintaining information from the previous steps.The first hidden layer has the weight obtained from the input layer, and at each layer will receive the weight of the prior layer.Then, the calculation of the next hidden layer uses the appropriate entity in the previous input and hidden layer.Meanwhile, the forecast for the output layer uses the last hidden layer.For the process of calculating the hidden layer using functions that can be seen in ( 2) and the calculation of output using softmax function which can be seen in (3).ℎ() = tanh(  () +  ℎ ( − 1) + ) (2) In this study, the data collection that was processed consists of various kinds of data sequences such as meteorology variables, so it is necessary to group data in the input layer to provide the training process.However, RNN has the disadvantage of short-term memory.Therefore, in making predictions using quite a lot of data, RNN has several variations to solve the problem.The gate is the gated recurrent unit (GRU), backpropagation through time (BPTT), and LSTM.This research used long short-term memory (LSTM) to overcome the vanishing gradient [34].
The RNNs training process is similar to neural network training using a backpropagation algorithm but with few cycles.The parameter that shared equally in every time step, so gradient for each output, does not only depend on a calculation from the current time step but also the previous one [35].This research used LSTM gates to overcome the dependence of long-term process, which is often phenomena that occurred in sequential data processing, as shown in Figure 3.
The key from LSTM is cell state with architecture marked by a horizontal line that flows from Ct-1 until Ct.LSTM can delete or add information to the cell state set by the structure, which is called the gate.Gate is a method to pass the information, which consists of biner sigmoid function () and multiplication operation with x.Biner Sigmoid function is as shown in (4).The first step in LSTM is deciding what information will be disposed of from the cell state called forget gate used (5) [34].
where ht-1 and xt values, are in 0-1 interval every cell.Use of 1, which represents this information, is kept, and 0, which expresses this information is deleted.The second is deciding recent data stored in the cell.This step is divided into two parts.First, Input gate will determine values that will be updated using ( 6) and calculation for recent cell candidate ( ̂) which; ̂ = ℎ(  [ℎ −1 ,   ] +   ) There will be added to the old cell state (Ct) with a cell ( ̂) candidate using (8).The former cell state multiplicated by forgetting state.And cell candidate multiplicated with the input gate.It updated the cell state.The last step is the output gate used to determine which output will be produced based on cell state from the results of ( 6) calculated with the biner sigmoid function as shown in (9) Furthermore, multiplicated with activating function from updated cell state using (10).Some previous research used sigmoid and tanh [35].

RESULTS AND ANALYSIS
Experiments from precipitation prediction models are carried out with variations in the interval of training data (10 years and five years), and the prediction time (monthly and weekly), with the RNN configuration in Table 1.In getting the optimal prediction, variations are performed on the dataset time and prediction duration.The experiment also tested the accuracy of the optimization model, the adaptive moment estimation (Adam) model, and the stochastic gradient descent (SGD) model.Accuracy is calculated from the accuracy of the output class against the actual class label.This research used two predictive models.First, it used weather data for ten years and five years.Both models are tested weekly and monthly.The accuracy is obtained as in Table 2, Figure 4 of 10-years dataset, and Figure 5 of the 5-years dataset.This simulation developed using two optimizer model (Adam and SGD) to correct weight, in 200 epoch.2, visible 5-year data sets provide an accuracy of 83.33% for monthly and 85.71% for weekly.While in Figure 5 gave an accuracy of 73.25% for monthly and 80.95% for weekly.This result strengthens the hypothesis that the amount of training data is more dominant than the number of variables that are processed.Refer to the prediction periods in Table 2.It appears that weekly period predictions have better accuracy (85.71%), given the variation of meteorological data every day of the week is not too large.Meanwhile, there is more variety of daily meteorology in a month so that when combined in the month makes a higher deviation.However, the model used to predict rain for a month is more robust when available weather data is incomplete.
Precipitation prediction depends on the training data sequence, configuration, method, and period of forecast.The results of the study, Adam model, provided better accuracy than SGD, considering Adam model used aggregate data in training.In contrast, SGD used only one or several parts randomly selected, so the possibility of occurrence is minimum local, as a result of not representing all data in each class.The weekly forecast offers higher efficiency than monthly finding that in that period, meteorological data is more homogeneous than monthly data.Training data used for ten years provides better correctness, considering it provides more variation than the accuracy of training data from the last five years.The best accuracy was 85.71% with weekly using RNN and LSTM of training data for ten years.This research can compare with MLP with 75-92% [21] and RNN with heuristic optimized of 59-84.6%[15].

CONCLUSION
In this work, we studied how to use multiple variables of weather can rainfall prediction monthly.proposed the advantage of recurrent neural networks to automatically gave accuracy 85.71 of test data.The research showed that using five years of weather data can predict precipitation weekly and monthly.Weather data in the last five years can predict rainfall for a month of the following year.However, weekly predictions have higher accuracy.The experimental results also show that a large number of data sets can improve accuracy.In the future, compared the proposed methods approach with a public weather forecast center results and demonstrated the effectiveness of the model.So that output prediction in value, improving the output class of this study.Current uncertain rainfall predictions do not only pay attention to past data patterns but also need to consider extreme phenomena such as El Nino as additional features.

Table 2 .
Accuracy of Precipitation toward dataset and duration