Ubuntu: Portal Server manual install

Ubuntu: Portal Server Install Instructions

Prerequisites

Supported Hardware

  • Amazon EC2 Cloud instances
  • Own hosted Servers with any Intel or AMD CPU

Minimum Hardware Requirements

  • Minimum required CPU Cores of Processor: 4
  • Minimum required RAM: 16 GB
  • Minimum required Disk: 512 GB
  • Minimum required Network Speed: 1000 Mbps

Email Server

The Portal Server sends its emails via SMTP. You need an email server which receive and forward these SMTP messages.

Twilio SMS Gateway

If the Portal Server will be operated/configured in such a way that any person can “sign up” (= self registration) you need a customer account for the Twilio SMS Gateway www.twilio.com/docs/sms

Network & System Tuning

In /etc/sysctl.conf add:

# TCP/IP Tuning
# =============
fs.file-max = 524288
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_fin_timeout = 30
net.ipv4.ip_local_port_range = 16384 60999
net.core.somaxconn = 256
net.core.rmem_max = 1048576
net.core.wmem_max = 1048576

In /etc/security/limits.conf add:

# TCP/IP Tuning
# =============
* soft     nproc          262140
* hard     nproc          262140
* soft     nofile         262140
* hard     nofile         262140
root soft     nproc          262140
root hard     nproc          262140
root soft     nofile         262140
root hard     nofile         262140

Enter: systemctl show -p TasksMax user-0

output: TasksMax=8966

if you get a value less than 262140 then add in /etc/systemd/system.conf

# Ubuntu Tuning
# =============
DefaultTasksMax=262140

Reboot the system and verify the settings. Enter: ulimit -n

output: 262140

Enter: systemctl show -p TasksMax user-0

output: TasksMax=262140

Forward the external TCP/IP server port 80 (HTTP) to port 8000, and forward external port 443 (HTTPS) to port 8001

Install ifupdown

sudo apt-get update
sudo apt-get install ifupdown

Create/edit the file DKFQSiptables in /etc/network/if-pre-up.d/ and add:

#!/bin/sh
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8000
iptables -t nat -I OUTPUT -p tcp -d 127.0.0.1 --dport 80 -j REDIRECT --to-ports 8000
iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 8001
iptables -t nat -I OUTPUT -p tcp -d 127.0.0.1 --dport 443 -j REDIRECT --to-ports 8001
exit 0

Then give execute permission to /etc/network/if-pre-up.d/DKFQSiptables : sudo chmod 755 /etc/network/if-pre-up.d/DKFQSiptables

Reboot the machine and check with:

sudo iptables -L -t nat

> Chain PREROUTING (policy ACCEPT)
> target     prot opt source               destination
> REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:http redir ports 8000
> REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:https redir ports 8001
> REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:http redir ports 8000
> REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:https redir ports 8001
>
> Chain INPUT (policy ACCEPT)
> target     prot opt source               destination
>
> Chain OUTPUT (policy ACCEPT)
> target     prot opt source               destination
> REDIRECT   tcp  --  anywhere             localhost            tcp dpt:https redir ports 8001
> REDIRECT   tcp  --  anywhere             localhost            tcp dpt:http redir ports 8000
> REDIRECT   tcp  --  anywhere             localhost            tcp dpt:https redir ports 8001
> REDIRECT   tcp  --  anywhere             localhost            tcp dpt:http redir ports 8000
>
> Chain POSTROUTING (policy ACCEPT)
> target     prot opt source               destination

Install Dependencies

Install fontconfig

sudo apt-get update
sudo apt-get install fontconfig

Install haveged

sudo apt-get update
sudo apt-get install haveged

Install SQLite

sudo apt-get update
sudo apt install sqlite

Install OpenJDK Java 8 and 11

Get the Java Installation Kits

wget https://download.java.net/openjdk/jdk8u41/ri/openjdk-8u41-b04-linux-x64-14_jan_2020.tar.gz
wget https://download.java.net/java/GA/jdk11/13/GPL/openjdk-11.0.1_linux-x64_bin.tar.gz

Install OpenJDK Java 8

gunzip openjdk-8u41-b04-linux-x64-14_jan_2020.tar.gz
tar -xvf  openjdk-8u41-b04-linux-x64-14_jan_2020.tar
rm openjdk-8u41-b04-linux-x64-14_jan_2020.tar
sudo bash *******
mkdir /opt/OpenJDK
mv java-se-8u41-ri /opt/OpenJDK
cd /opt/OpenJDK
ls -al
chown root -R java-se-8u41-ri
chgrp root -R java-se-8u41-ri
exit # end sudo bash

Verify the Java 8 installation.

/opt/OpenJDK/java-se-8u41-ri/bin/java -version

openjdk version "1.8.0_41"
OpenJDK Runtime Environment (build 1.8.0_41-b04)
OpenJDK 64-Bit Server VM (build 25.40-b25, mixed mode)

Install OpenJDK Java 11

gunzip openjdk-11.0.1_linux-x64_bin.tar.gz
tar -xvf openjdk-11.0.1_linux-x64_bin.tar
rm openjdk-11.0.1_linux-x64_bin.tar
sudo bash
mv jdk-11.0.1 /opt/OpenJDK
cd /opt/OpenJDK
ls -al
chown root -R jdk-11.0.1
chgrp root -R jdk-11.0.1

Execute the following commands (still as sudo bash):

update-alternatives --install "/usr/bin/java" "java" "/opt/OpenJDK/jdk-11.0.1/bin/java" 1
update-alternatives --install "/usr/bin/javac" "javac" "/opt/OpenJDK/jdk-11.0.1/bin/javac" 1
update-alternatives --install "/usr/bin/keytool" "keytool" "/opt/OpenJDK/jdk-11.0.1/bin/keytool" 1
update-alternatives --install "/usr/bin/jar" "jar" "/opt/OpenJDK/jdk-11.0.1/bin/jar" 1
update-alternatives --set "java" "/opt/OpenJDK/jdk-11.0.1/bin/java"
update-alternatives --set "javac" "/opt/OpenJDK/jdk-11.0.1/bin/javac"
update-alternatives --set "keytool" "/opt/OpenJDK/jdk-11.0.1/bin/keytool"
update-alternatives --set "jar" "/opt/OpenJDK/jdk-11.0.1/bin/jar"
exit # end sudo bash

Verify the Java 11 installation.

java -version

openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)

Install the Portal Server

Create the DKFQS account which is running the Portal Server

sudo adduser dkfqs    # follow the questions, remember or write down the password

Install the Portal Server

Login with the dkfqs account (SSH) - or - Enter: sudo -u dkfqs bash | OR: Install Samba to get convenient access to /home/dkfqs as Samba dkfqs user

Create the directory /home/dkfqs/portal (as dkfqs user):

cd /home/dkfqs
mkdir portal

Create the following sub-directories at /home/dkfqs/agent (as dkfqs user):

  • backup
  • bin
  • config
  • db
  • developerExamples
  • htdocs
  • javaUsersLib
  • jks
  • log
  • reportTemplate
  • scripts
  • temp
  • usersLib
  • usersData
cd /home/dkfqs/portal
mkdir backup bin config db developerExamples htdocs javaUsersLib jks log reportTemplate scripts temp usersLib usersData

Copy the following files to the bin directory /home/dkfqs/portal/bin

  • bcpkix-jdk15on-160.jar
  • bcprov-jdk15on-160.jar
  • bctls-jdk15on-160.jar
  • DKFQS.jar
  • jasperreports-fonts-6.20.5.jar

Copy the following files to the db directory /home/dkfqs/portal/db

  • CreateNewAdminDB.sql
  • CreateNewMonitoringDB.sql
  • CreateNewOperationsDB.sql
  • CreateNewUsersDB.sql

Edit the file CreateNewUsersDB.sql and modify the following line to set the nickname, the email, the phone number and the temporary password of the Admin account. Note: The nickname must always start with “Admin-”

insert into AdminAccountsTable (adminUserId, nickname, adminPrimaryEmail, adminPrimaryPhone, initialPassword) values (1, "Admin-One", "falarasorn@yahoo.com", "+43123456789", "ginkao1234");

Navigate to /home/dkfqs/portal/db and create the Admin, Operations and the Users DB (as dkfqs user):

sqlite3 AdminAccounts.db < CreateNewAdminDB.sql
sqlite3 Monitoring.db < CreateNewMonitoringDB.sql
sqlite3 Operations.db < CreateNewOperationsDB.sql
sqlite3 Users.db < CreateNewUsersDB.sql

Copy the following file to the htdocs directory /home/dkfqs/portal/htdocs

  • htdocs.jar

Navigate to /home/dkfqs/portal/htdocs and execute (as dkfqs user):

jar -xvf htdocs.jar
rm htdocs.jar   # delete the jar)
rm -R META-INF  # delete the META-INF directory)

Copy the following files to the javaUsersLib directory /home/dkfqs/portal/javaUsersLib

  • com.dkfqs.selenium.jar
  • commons-io-2.13.0.jar
  • hamcrest-core-1.3.jar
  • junit-4.13.2.jar
  • selenium_bundle-4.10.0.jar
  • selenium_bundle-4.12.1.jar

Copy the following file to the jks directory /home/dkfqs/portal/jks

  • dkfqscom.jks

Copy the following files to the usersLib directory /home/dkfqs/portal/usersLib

  • com.dkfqs.tools.jar
  • DKFQSLibrary2.psm1 (optional)

Copy the following files to the bin directory /home/dkfqs/portal/reportTemplate

  • AnnotationEvent.jasper
  • Annotations.jasper
  • ErrorCharts.jasper
  • ErrorTypeDistribution.jrxml
  • ErrorTypes.jasper
  • JobProperties.jasper
  • MonitoringGroup.jasper
  • MonitoringGroupAvailability.jasper
  • MonitoringJob.jasper
  • MonitoringJobAvailability.jasper
  • MonitoringJobPerformance.jasper
  • OverviewStatistics.jasper
  • SamplesThroughputPerSecond.jasper
  • SummaryStatistic.jasper
  • TestResults.jasper
  • TestSpecificData.jasper

Copy the following files to the bin directory /home/dkfqs/portal/config

  • dkfqs.properties
  • rewrite.properties
  • twilio.properties

Modify the dkfqs.properties file. Set the following properties:

  • ServerName
  • ServerDNSName
  • DNSJavaDefaultDNSServers
  • UsersMailServerHost
  • UsersMailFrom
  • UsersMailServerAuthUser
  • UsersMailServerAuthPassword
  • ServerStatusPageEnabledIPList
  • AlertMailServerHost
  • AlertMailFrom
  • AlertMailToList
  • AlertMailBounceAddress
  • AlertMailServerAuthUser
  • AlertMailServerAuthPassword

Example: dkfqs.properties

IsProduction=true
ServerName=192.168.0.50
ServerDNSName=192.168.0.50
DiskDocumentRootDirectory=/home/dkfqs/portal/htdocs
SQLiteDBDirectory=/home/dkfqs/portal/db
ReportTemplateDirectory=/home/dkfqs/portal/reportTemplate/
JasperReportsTemporaryDirectory=/home/dkfqs/portal/temp/
UsersDataRootDirectory=/home/dkfqs/portal/usersData
OSProcessLogFile=/home/dkfqs/portal/log/DKFQS.log
LogLevel=info
StaticContentMaxAgeTime=7200
MaxHTTPRequestSize=20240000
MaxInvalidAnonymousSessionsPerIPLimit=32
AnonymousSessionTimeout=1200
MaxAnonymousSessionTime=21600
MaxWebSocketConnectTimeSeconds=14400
MaxInboundWebSocketTrafficPerConnection=67108864
MaxInboundWebSocketPayloadPerFrame=1048576
MaxInboundWebSocketFramesPerIPTimeFrame=10
MaxInboundWebSocketFramesPerIPLimit=1000
HTTPExternalServerPort=80
HTTPInternalServerPort=8000
HTTPSExternalServerPort=443
HTTPSInternalServerPort=8001
HTTPSKeyStoreFile=/home/dkfqs/portal/jks/dkfqscom.jks
HTTPSKeyStorePassword=topsecret
#
FileTreeApiMaxRequestSizeMB=256
FileTreeApiWorkerThreadBusyTimeoutSeconds=330
FileTreeApiWorkerThreadExecutionTimeoutSeconds=300
TestjobsApiMaxRequestSizeMB=256
TestjobsApiWorkerThreadBusyTimeoutSeconds=330
TestjobsApiWorkerThreadExecutionTimeoutSeconds=300
#
DNSJavaDefaultDNSServers=8.8.8.8,8.8.4.4
#
JavaSDK8BinaryPath=/opt/OpenJDK/java-se-8u41-ri/bin
JavaSDK11BinaryPath=/opt/OpenJDK/jdk-11.0.1/bin
HTTPTestWizardJavaCodeLibraries=/home/dkfqs/portal/usersLib/com.dkfqs.tools.jar
#
UserSignInURL=/SignIn
UsersMailServerHost=192.168.1.4
UsersMailFrom=xxxxxxxxx@xxxxxxx.com
UsersMailServerAuthUser=xxxxxxxxx@xxxxxxx.com
UsersMailServerAuthPassword=*********
UsersMailTransmitterThreads=2
UsersMailDebugSMTP=false
#
smsGatewaysClassNames=com.dkfqs.server.sms.twilio.TwilioSMSGateway
#
# ServerStatusPageEnabledIPList=127.0.0.1,192.168.0.99
ServerStatusPageEnabledIPList=*.*.*.*
AdminSignInURL=/AdminSignIn
AlertMailEnabled=true
AlertMailServerHost=192.168.1.4
AlertMailFrom=xxxxxxxxx@xxxxxxx.com
AlertMailToList=yyyyyyyyy@xxxxxxx.com,zzzzzzzzz@xxxxxxx.com
AlertMailBounceAddress=bbbbbbb@xxxxxxx.com
AlertMailServerAuthUser=xxxxxxxxx@xxxxxxx.com
AlertMailServerAuthPassword=*******
AlertMailDebugSMTP=false
AlertMailNotifyStartup=false
#
SecurityIpBlacklistBlockTime=900
SecurityMaxRequestsPerIpLimit=200
SecurityMaxRequestsPerIpTimeFrame=10
SecurityMaxInvalidRequestsPerIpLimit=12
SecurityMaxInvalidRequestsPerIpTimeFrame=60
SecurityMaxAnonymousFormSubmitPerIpLimit=8
SecurityMaxAnonymousFormSubmitPerIpTimeFrame=60
SecurityMaxAuthenticationFailuresPerIpLimit=5
SecurityMaxAuthenticationFailuresPerIpTimeFrame=60
#
MeasuringAgentConnectTimeout=10

# Support of license provider 'Real Load Pty Ltd / nopCommerce', if enabled then the LicenseProviderRealLoadPtyLtdNopCommerceCARootFileName is located in the config directory of the portal server
LicenseProviderRealLoadPtyLtdNopCommerceEnabled=true
LicenseProviderRealLoadPtyLtdNopCommerceCARootFileName=nopCommerceCARoot.pem

# Synthetic Monitoring settings
MonitoringJobSchedulerMainThreadIntervalSeconds=10
MonitoringJobSchedulerInitialDBLogLevel=warn
MonitoringJobSchedulerInitialStdoutLogLevel=info
MonitoringJobSchedulerInitialUserFileLogLevel=info
MonitoringGroupThreadsStartDelayMillis=1000

# Support for compiling Junit and Selenium Tests
JavaUsersLibDirectory=/home/dkfqs/portal/javaUsersLib
JavaJUnit4CompilerLibraries=hamcrest-core-1.3.jar,junit-4.13.2.jar
JavaJUnit4TestExecutionLibraries=com.dkfqs.selenium.jar,com.dkfqs.tools.jar,commons-io-2.13.0.jar,hamcrest-core-1.3.jar,junit-4.13.2.jar
JavaSelenium4CompilerLibraries=hamcrest-core-1.3.jar,junit-4.13.2.jar,selenium_bundle-4.12.1.jar,com.dkfqs.selenium.jar
JavaSelenium4TestExecutionLibraries=selenium_bundle-4.12.1.jar,com.dkfqs.selenium.jar,com.dkfqs.tools.jar,commons-io-2.13.0.jar,hamcrest-core-1.3.jar,junit-4.13.2.jar

Modify the twilio.properties file. Set the following properties:

  • sid
  • authToken
  • fromTwilioPhoneNumber

Example: twilio.properties

apiURLMainPath=https://api.twilio.com/2010-04-01/Accounts/
sid=********************************
authToken=********************************
fromTwilioPhoneNumber=+1123456789
fromTwilioAlphanumericSenderID=RealLoad
tcpConnectTimoutMillis=10000
sslHandshakeTimeoutMillis=5000
httpProcessingTimeoutMillis=10000
debugAPI=false

First Test - Start the Portal Server manually (as dkfqs user)

cd /home/dkfqs/portal/bin
export CLASSPATH=bcpkix-jdk15on-160.jar:bcprov-jdk15on-160.jar:bctls-jdk15on-160.jar:DKFQS.jar
java -Xmx2048m -DdkfqsProperties=../config/dkfqs.properties -DrewriteProperties=../config/rewrite.properties -Dnashorn.args="--no-deprecation-warning" com.dkfqs.server.internal.StartDKFQSserver

Internal RSA 2048 bit keypair generated in 1220 ms
2021-03-10 22:27:25.040 | QAHTTPd | INFO | SQL connection pool for DB "UsersDB" initialized
2021-03-10 22:27:25.062 | QAHTTPd | INFO | SQL connection pool for DB "AdminAccountsDB" initialized
2021-03-10 22:27:25.068 | QAHTTPd | INFO | Alarm adapter "IP Blacklist Alarm Adapter" started
2021-03-10 22:27:25.069 | QAHTTPd | WARN | QAHTTPd V1.3-U started
2021-03-10 22:27:25.071 | QAHTTPd | INFO | Execute PreUpStartupLoadIPRangeBlacklist
2021-03-10 22:27:25.082 | QAHTTPd | INFO | HTTP server starting at port 8000
2021-03-10 22:27:25.109 | QAHTTPd | INFO | HTTP server ready at port 8000
2021-03-10 22:27:25.110 | QAHTTPd | INFO | HTTPS server starting at port 8001
2021-03-10 22:27:25.124 | QAHTTPd | INFO | HTTPS server ready at port 8001
2021-03-10 22:27:25.821 | EMAIL-1 | INFO | Email transmitter thread started
2021-03-10 22:27:25.822 | EMAIL-2 | INFO | Email transmitter thread started
2021-03-10 22:27:25.828 | main | INFO | Twilio SMS Gateway registered
2021-03-10 22:27:25.853 | main | INFO | Twilio SMS Gateway initialized
2021-03-10 22:27:25.857 | SMS-Dispatcher | INFO | Thread started

Create the Portal Server Startup Script (as root)

sudo bash # become root
cd /etc/init.d
vi DKFQS

Edit - create /etc/init.d/DKFQS

#!/bin/sh
# /etc/init.d/DKFQS
# install with: update-rc.d DKFQS defaults

### BEGIN INIT INFO
# Provides:          DKFQS
# Required-Start:    $local_fs $network $time $syslog
# Required-Stop:     $local_fs $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start DKFQS daemon at boot time
# Description:       DKFQS daemon
### END INIT INFO

case "$1" in
  start)
    if [ -f /home/dkfqs/portal/log/DKFQS.log ]; then
       mv /home/dkfqs/portal/log/DKFQS.log /home/dkfqs/portal/log/DKFQS.log_$(date +"%Y_%m_%d_%H_%M")
    fi
    sudo -H -u dkfqs bash -c 'CLASSPATH=/home/dkfqs/portal/bin/bcpkix-jdk15on-160.jar:/home/dkfqs/portal/bin/bcprov-jdk15on-160.jar:/home/dkfqs/portal/bin/bctls-jdk15on-160.jar:/home/dkfqs/portal/bin/jasperreports-fonts-6.20.5.jar:/home/dkfqs/portal/bin/DKFQS.jar;export CLASSPATH;cd /home/dkfqs/portal/temp;nohup java -Xmx3072m -DdkfqsProperties=/home/dkfqs/portal/config/dkfqs.properties -DrewriteProperties=/home/dkfqs/portal/config/rewrite.properties -Dnashorn.args="--no-deprecation-warning" com.dkfqs.server.internal.StartDKFQSserver 1>/home/dkfqs/portal/log/DKFQS.log 2>&1 &'
    ;;
  stop)
       PID=`ps -o pid,args -e | grep "StartDKFQSserver" | egrep -v grep | awk '{print $1}'`
       if [ ! -z "$PID" ] ; then
          echo "DKFQS stopped with pid(s) : $PID"
          kill -9 ${PID} 1> /dev/null 2>&1
       fi
    ;;
  status)
       PID=`ps -o pid,args -e | grep "StartDKFQSserver" | egrep -v grep | awk '{print $1}'`
       if [ ! -z "$PID" ] ; then
          echo "DKFQS running with pid(s) : $PID"
       else
          echo "No DKFQS running"
       fi
    ;;
  *)
    echo "Usage: /etc/init.d/DKFQS {start|stop|status}"
    exit 1
    ;;
esac

exit 0

Change owner and file protection of /etc/init.d/DKFQS (root at /etc/init.d):

chown root DKFQS
chgrp root DKFQS
chmod 755 DKFQS

Register /etc/init.d/DKFQS to be started at system boot (root at /etc/init.d):

update-rc.d DKFQS defaults

Reboot the system. Then check /home/dkfqs/portal/log/DKFQS.log

Administrator Sign In

Enter in your browser https://admin-portal-host/admin .

You will get a browser warning because the SSL server certificate is expired. Ignore the warning and enter in the Sign In the email address and the password as you have set in CreateNewAdminDB.sql.

You will now asked to set a new password. Then you are signed in.

“alt attribute”

Disable Sign Up if you don’t have an SMS gateway:

“alt attribute”

User accounts can be added directly:

“alt attribute”

Replace the SSL Server Certificate

If you or your company can already issue SSL server certificates you can skip the next sub-chapter. Continue in such a case with “Convert and Install the SSL Server Certificate”.

Get a Let's Encrypt SSL Server Certificate | Ubuntu 20

Make sure that your portal server has a public, valid DNS name.

Install certbot:

sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot

Stop the Portal Server:

sudo /etc/init.d/DKFQS stop

To get the SSL server certificate enter:

sudo certbot certonly --standalone   # enter your email address and the DNS name ou your portal server, follow the instructions

On success certbot generates the following two files:

  • fullchain.pem
  • privkey.pem

Start the Portal Server:

sudo /etc/init.d/DKFQS start

Convert and Install the SSL Server Certificate

Become root and navigate to the directory where the fullchain.pem and privkey.pem files are located. Enter:

sudo bash
openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out your-certificate-name.p12  # convert cert to PKCS12 file
keytool -importkeystore -srckeystore your-certificate-name.p12 -srcstoretype PKCS12 -destkeystore your-certificate-name.jks -deststoretype JKS  # convert PKCS12 file to Java keystore file

Copy the Java keystore file to /home/dkfqs/portal/jks

cp your-certificate-name.jks /home/dkfqs/portal/jks

Edit /home/dkfqs/portal/config/dkfqs.properties and replace:

HTTPSKeyStoreFile=/home/dkfqs/portal/jks/your-certificate-name.jks
HTTPSKeyStorePassword=*********

Restart the Portal Server

sudo /etc/init.d/DKFQS stop
sudo /etc/init.d/DKFQS start

Create a Cron Job to Renew the Let's Encrypt SSL Certificate

Create at your home directory the sub-directory system_cronjobs_scripts and add/edit the file “DKFQS_certbot_renew” in this directory. Replace the ********* placeholders with your real values.

#!/bin/sh
#
# renew the letsencrypt DKFQS certificate
# =======================================
certbot renew
#
# set the default working directory
cd /home/*********/system_cronjobs_scripts
#
# cleanup in any case
rm -f *.jks
rm -f *.p12
#
# convert the letsencrypt certificate to PKCS12 and place it in the default directory
openssl pkcs12 -export -in /etc/letsencrypt/live/*********/fullchain.pem -inkey /etc/letsencrypt/live/*********/privkey.pem -out ./*********.p12 -passin pass:******** -passout pass:********
#
# convert the PKCS12 certificate to a Java keystore
echo ******** | keytool -importkeystore -srckeystore *********.p12 -srcstoretype PKCS12 -destkeystore *********.jks -deststoretype JKS -storepass ********
#
# update DKFQS keystore file
cp *********.jks /home/dkfqs/portal/jks
chown dkfqs /home/dkfqs/portal/jks/*********.jks
chgrp dkfqs /home/dkfqs/portal/jks/*********.jks
chmod 600 /home/dkfqs/portal/jks/*********.jks
#
# restart DKFQS server
/etc/init.d/DKFQS stop
sleep 5
sudo /etc/init.d/DKFQS start
#
# cleanup again in any case
rm -f *.jks
rm -f *.p12
#
# display DKFQS log file
sleep 5
cat /home/dkfqs/portal/log/DKFQS.log
#
# all done
exit 0
sudo bash # become root
chmod 700 DKFQS_certbot_renew # change file protection and set execute bit
./DKFQS_certbot_renew  # try out manually

Add the file to crontab. Important: The last line in crontab must be an empty line!

sudo crontab -e
1 1 1 * * /home/*********/system_cronjobs_scripts/DKFQS_certbot_renew > /home/*********/system_cronjobs_scripts/DKFQS_certbot_renew.log 2>&1