Nov 2021 to Apr 2022 project summary
Project Name: eStatement of Malaysia **Duration: **Dec 2021 to Apr 2022 Project size: 6 people (1 Squad) + TDM Keywords: tough project management, and Spring template 400 error resolving,Spring RestTemplate usage Issue #20220514
**Background: **
- This project came to me which had 2-3 months behinded the schedule already. Project manager resigned 2 weeks later. The requirment/time line/detail design/engineer resource/other teams' support were the major challenges.
- With tough work for couples of months, project pilot launched on 25 March withinoriginal project timeline. 3.At the end of pilot phase, when team planned to open this new feature of Personal Internet Banking and Mobile Application to public, suddenly found that Personal Internet Banking will intermittently hit 400 error.
Issue checking and solutions:
-
The error was happening on both of enquiry and download 2 API calls randomly and intermittently. When issue happened, client side (Personal Internet Banking) connected to target EST server, and disconnected after 20 seconds.
-
After discussion with network team, network can be excluded tentatively.
-
Turn on target EST apache webserver log to debug level, there were below error: [Mon Apr 25 19:00:34.855994 2022] [reqtimeout:info] [pid 3292:tid 8] [client 172.29.20.20:50399] AH01382: Request body read timeout [Mon Apr 25 19:00:34.856172 2022] [weblogic:error] [pid 3292:tid 8] [client 172.29.20.20:50399] <3292165088441412> Error reading POST data from client \n
4.Found that the waiting 20 seconds was from the apache 2.4 setting. The connection is controlled by module reqtimeout_module. It has default 20 seconds on connection timeout, includes handshake + header + body.
5.Suspected it was due to apache server capability, thus did parameter tunning: 1) increase server worker thread number 2)increase above timeout parameter. All didnt resolve the problem.
6.In parallel, checked client side (Personal Internet Banking) potential reason. Found that the Rest Api call is using default method. After looking into RestTemplate library source code, found below suspect reason: 1)default usage as shown in below, it will only use SimpleClientHttpRequestFactory, and the connection may not designed for production use case. It has no connection/read time out. It seems that use some buffer for request posting. default usage:
- RestTemplate restTemplate = new RestTemplate(); ResponseEntity responseWS =restTemplate.postForEntity(webServiceURL, entity, String.class);*
solution: changed it to use HttpComponentsClientHttpRequestFactory to have powerful httpclient.
2)the SSL handshake has no special handling. solution: Since it is app to app internal within same company, we can implement host name trusted which can save handshake time.
3)the request message got google json converted twice. Suspected it is not thread safe and may caused thread dead lock solution: Use google json to convert java PO once and save into String class since it is immutable
4)defint the request connection to be short connection type. and set request to be UTF_8 explicty code: headers.add(HTTP.CONN_DIRECTIVE, HTTP.CONN_CLOSE);restTemplate.getMessageConverters().add(0, new StringHttpMessageConverter(StandardCharsets.UTF_8));;
After above change, issue was resolved and project released to public successsfully.
