Published: 2022-02-07 Last modified: 2022-02-07
What problem this solution solves? |
|
|
This setup could be quite easily adopted for backuping other apps data. |
This post is quite long, as it is intended for people, who, like me, while have some experience with command line and Bash in Linux, are less acquainted with the Android internals and Termux abilities, thus each step is well (I hope) explained and documented. |
- 1. The main idea and tools
- 2. Preparations
- 3. Applying the automatic backup scripts
1. The main idea and tools
-
Since I already have a good (3-2-1 and encrypted) backup solution for my computer, and the phone is rooted (so calendar and contacts databases and other data files are accessible), I realized that all I need is a daily automatic backup of the relevant files and folders to a folder on my comp, where it would be properly backup-ed as all the other comp’s data.
-
The backup is for an accident case only (phone lost or damaged), so the restore process not needed to be automatic.[2] For planned migration to a new phone I can simply export Calendar and Contacts data via their apps and import on a new phone.
-
As I consider my computer to be more secure than my phone, the backup solution should rely on computer→phone communication and not vice-versa (if your situation is inverted, you need to apply some minor changes).
-
Required apps/tools:
-
Calendar and Contacts apps allowing export-import, and the creation of an offline calendar. I chose the Simple Mobile apps (available on
F-Droid
). -
Termux
as the phone’s terminal app (available onF-Droid
).[3] -
ssh
andrsync
for the comp→phone communication (Termux
packages on the phone; your distro packages on the comp). -
crond
app[4] for the phone, andcrontab
for the comp, to initiate the backup process. -
Two simple
Bash
scripts (one for the phone, one for the comp) that do the actual backup. -
On the phone,
notify-send
tool from your distro (e.g.,sudo apt install libnotify-bin
)
-
2. Preparations
2.1. Install the apps and give them permissions
-
Install the apps mentioned above.
-
Give root permission to
Termux
(installtsu
package to use it). -
If you have a firewall on the phone, whitelist
Termux
. -
Disable battery optimization for
crond
(should not cause any visible increase in battery usage) and give it root permission.
2.2. Set static IP for the phone and the computer in the home network
-
On the phone, disable randomized MAC address for your home network (look in "network details" or similar for the given network).
-
Setup static DHCP lease for the phone and the computer (based on their MAC addresses) in your router so that they will not change their IP each time they connect to the home network (read your router’s manual, or, better, use OpenWRT).
2.3. Configure and test ssh
and rsync
-
In
Termux
:~ pkg install openssh rsync
-
Follow the instructions in Termux wiki regarding OpenSSH server, including Setting up public key authentication.
-
Edit the
sshd
configuration file as follow:
PrintMotd yes PasswordAuthentication no (1) Subsystem sftp /data/data/com.termux/files/usr/libexec/sftp-server (2) ListenAddress 192.168.1.100 (3) PermitRootLogin yes (4) StrictModes no (5)
1 | because you have key-based authentication. |
2 | default setting, not needed for the proposed here setup, could be removed. |
3 | change this to the actual phone’s static IP. |
4 | since Termux anyway has root permission, and password authentication is disabled, and sshd will listen for root only on home network (specified by the static IP), and just for a few minutes, I don’t see a problem to allow root login (instead of login as a user, then sudo ing). |
5 | needed to allow both user- (for setup and tests) and root- (for the actual backup mechanism) connections to the phone, since without this settings, sshd requires that the ~/.ssh/authorized_keys file and the containing folder will be owned by the user who runs the sshd , however they cannot be owned both by the regular user and by root. |
-
The last point probably needs some clarification. Since
Termux
is single-user, there is no need to mention a user when connecting withssh
to theTermux
-runsshd
. Upon successful connection,ssh
session login will automatically be setup as the user who runs thesshd
. So, for example, if you runsudo sshd
inTermux
, and then connect from outside with something likessh -i id_rsa <ip_address>
, you will be root[5], but if you runsshd
as the regular user, the exactly samessh
command will result in the login as the user. -
In the computer, configure the phone details to simplify the connecting commands (
ssh
andrsync
): add the following lines to~/.ssh/config
file (create if doesn’t exist yet).
Host my-phone Hostname 192.168.1.100 (1) port 8022 (2) IdentityFile /home/user/.ssh/my-phone_rsa (3)
1 | change this to the actual static IP of the phone. |
2 | default sshd port in Termux. |
3 | change to the actual filename of the key-file you created for the public key authentication. |
-
Verify the configuration by connecting to the phone with
ssh my-phone
(be sure thatsshd
is running on the phone). -
Verify
rsync
connection from the computer:~$ cd ~$ echo '123' > testfile.txt ~$ rsync -av testfile.txt my-phone:/data/data/com.termux/files/home/
then on the phone:
~$ cat ~/testfile.txt 123
2.4. Firewall (optional)
-
Skip this section if you don’t use any firewall on the phone and can
ssh
to the phone as root (whensshd
on the phone was started withsudo
ortsu
). -
If you never heard of
iptables
, you may want to read some short intro to it. -
If you have firewall other than
AFWall+
, and/or added some custom firewall rules, you may need to change a bit the instructions. -
For the rest of this section, suppose you have
AFWall+
and allowed (whitelisted)Termux
app in it as suggested here. -
Since
Termux
whitelisted inAFWall+
, the latter enablesssh
connection to the phone asTermux
user. However, when thesshd
is started by root,AFWall+
doesn’t consider the connection as aTermux
process, and blocks it. AFAIU,AFWall+
works on theOUTPUT
chain, so it lets packets in, but blocks packets out, effectively terminating connection. Therefore, one additionaliptables
rule is needed to allow such connection. The rule is already in the script, but to be sure it works as expected, follow the rest of this section. -
Create an alias in
~/.bashrc
:[6]
alias ipto='sudo /system/bin/iptables -L OUTPUT --line-numbers'
-
Don’t forget to activate the alias:
~$ . ~/.bashrc
-
See the default
AFWall+
rules in theOUTPUT
chain by runningipto
. -
Create and
chmod u+x
the following files. They are almost identical, but the second deletes (-D
) the rule which the first inserts (-I
):
#!/data/data/com.termux/files/usr/bin/bash bin_iptables='/system/bin/iptables' interface='wlan0' (1) allowed_ip='192.168.1.200' (2) local_ssh_port='8022' (3) sudo $bin_iptables -I OUTPUT -o $interface -p tcp -d $allowed_ip --sport $local_ssh_port -m conntrack --ctstate ESTABLISHED -j ACCEPT
#!/data/data/com.termux/files/usr/bin/bash bin_iptables='/system/bin/iptables' interface='wlan0' (1) allowed_ip='192.168.1.200' (2) local_ssh_port='8022' (3) sudo $bin_iptables -D OUTPUT -o $interface -p tcp -d $allowed_ip --sport $local_ssh_port -m conntrack --ctstate ESTABLISHED -j ACCEPT
1 | change to the actual interface name with which the phone connects to the home network (could be seen with sudo ifconfig ). |
2 | change to the LAN IP of the computer (needs to be static). |
3 | default sshd port (check with sudo netstat -pltn while sshd is running). |
-
Run
./allow-ssh-root.sh
, thenipto
, you should see a new first line inserted in theOUTPUT
chain.ssh
as root should work now, test it. -
Run
./revert-rules.sh
, thenipto
, you should see the rules reverted to the default state, andssh
as root should be blocked.
2.5. Calendar migrating (optional)
-
If you have any synchronizing Calendar app/account (e.g.
EteSync
orDecSync CC
or similar), and want to migrate to the local 'offline' calendar, inSimple Calendar
app (installed as one of the required tools):-
check
Settings→CalDAV→CalDAV sync
, choose the account, -
export events to
.ics
file, -
uncheck
CalDAV sync
, -
import from
.ics
file, in import settings use "ignore events type, always use regular". -
now you should have all the events in the off-line
Simple Calendar
's own storage.
-
2.6. Contacts migrating (optional)
-
If you have any synchronizing Contacts app/account (e.g.
EteSync
orDecSync CC
or similar), and want to migrate to the local 'offline' contacts, read this section.
2.6.1. Export contacts to file
-
In
Simple Contacts
app (installed as one of the required tools) export contacts to.vcf
file (e.g.,contacts.vcf
).[7] -
It is recommended to verify that the number of exported contacts is equal to the number of the contacts in Contacts Provider. For fast and dirty check (without analysing the
sqlite
contacts database):-
To show number of contacts in the exported file:
~ $ grep "BEGIN:VCARD" contacts.vcf | wc -l
(i.e., count lines having
BEGIN:VCARD
) -
To show number of contacts in the Provider, install Termux API (pay attention that you need to install both the app and the package) and run
~ $ termux-contact-list | grep '{' | wc -l
(i.e., count lines having
{
symbol, since each contact enclosed in curly brackets). -
If the two commands give equal numbers, you may skip the troubleshooting sub-section.
-
2.6.2. Troubleshooting
Toggle visibility
-
There are several reasons for the discrepancy between the number of contacts in the database (as reported by
termux-contact-list
) and in the exported.vcf
file. For example, it seems that Termux command doesn’t report contacts with no telephone number. To better understand the situation, analyzing the database of theContacts Provider
is required. -
Transfer to the computer the database file
/data/data/com.android.providers.contacts/databases/contacts2.db
: either pull withadb
, orrsync
. For the latter, on the phone runsshd
as root (and, if you have firewall, applyallow-ssh-root.sh
as explained in firewall section); then on the computer:~$ rsync -av my-phone:/data/data/com.android.providers.contacts/databases/contacts2.db .
-
Install
sqlitebrowser
[8] from your distro, then open with it the database file:~$ sqlitebrowser contacts2.db
-
In
Browse Data
tab you can manually inspect your contacts, especially useful are the tables (views, actually)view_contacts
andview_raw_contacts
. -
Run following queries in
Execute SQL
tab to better understand the situation:-
total number of contacts as shown in Contacts app:
SELECT count(*) FROM contacts;
-
number of contacts that have phone number (this should be equal to the number acquired with the
termux-contact-list
as explained above):SELECT count(*) FROM contacts WHERE has_phone_number=1;
-
if the numbers are different, see the contacts without phone number (may want to edit them in the Contacts app, if phone numbers are missing by mistake):
SELECT display_name FROM view_contacts WHERE has_phone_number=0;
-
to discover 'linked contacts' (two or more contacts entries with identical names which could be created as a result of improper import/migrating, or just by erroneously creating twice the same contact), compare the numbers above with the result of:
SELECT count(*) from raw_contacts WHERE deleted=0;
-
if the latter number is bigger, you probably have linked contacts. Additional verification could be done with (the number should be equal to the number of the first query in this list):
SELECT count(DISTINCT contact_id) from raw_contacts WHERE deleted=0;
-
finally, to show all the linked contacts, use:
SELECT display_name, count(contact_id) AS c FROM view_raw_contacts GROUP BY display_name HAVING c>1;
-
you can unlink linked contacts (and, preferably, delete one of the each pair of duplicate entries, after verifying that the other fields are also equal) using the Calendar app.
-
if you have contacts from several sources (for example, both local off-line contacts and a syncing CardDAV account), list account types with:
SELECT DISTINCT account_type FROM view_raw_contacts;
-
then you can modify all the queries above to count or show only the contacts from a specific account, e.g.
SELECT count(*) FROM view_raw_contacts WHERE deleted=0 AND account_type='com.android.contacts';
-
finally, if you edited the contacts in the app, you can export the contacts again from the app, re-transfer the updated database to the computer, and run the checks again, until all the numbers make sense.
-
2.6.3. Import contacts from file
-
Simple Contacts
can maintain its own contacts database (called 'phone storage not visible by other apps') or useContacts Provider
(referred asDEVICE
in the app), I suggest usingDEVICE
for contacts to be visible to Phone and Messaging apps. -
Import contacts from the
.vcf
file toDEVICE
, check functionality of the Phone and Messaging apps. -
If you have doubts regarding the completeness of the contacts data, you may want to rerun the checks listed in the Troubleshooting sub-section.
2.7. Removing syncing accounts (optional)
-
If you followed the migrating to local storage of the contacts and the calendar, on this stage you can remove the syncing app/account, and to turn off 'automatically sync app data' (in Phone Settings→Accounts).
3. Applying the automatic backup scripts
3.1. Creating, configuring, and testing the scripts
3.1.1. Overview of the functionality
-
The backup functionality is provided by two
Bash
scripts (one on the phone, one on the computer), which run simultaneously (bycron
) on the two devices. -
The phone’s script (
allow-sync.sh
), run by root (actually, run bycrond
app with root permission), does the following:-
Starts
sshd
as root, thus enabling computer to connect withssh
andrsync
as root; -
If needed, adds a firewall rule to enable such a connection;
-
Waits for creation of a flag file by the computer, signalizing that the syncing finished successfully, or until configured timeout is reached;
-
Terminates the
sshd
daemon, and reverts the firewall state (if it was altered).
-
-
The computer’s script (
my-phone-sync.sh
) does the following:-
Tries several times (configurable) to establish
ssh
connection to the phone, checks that the login user is root; -
Upon successful connection,
rsync
a list of folders of the phone (configurable) to the chosen local folder; -
Finally, creates in the phone a flag file, signalizing that the syncing finished successfully.
-
Sends desktop notifications during the process.
-
3.1.2. Phone script (allow-sync.sh)
-
On the phone, create and make executable (
chmod +x
) the following file inTermux
home (/data/data/com.termux/files/home/
):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
#!/data/data/com.termux/files/usr/bin/bash
# exit in case of unbounded variable(s)
set -u
### Settings, default should work, change if needed
# abort waiting for the computer after 15 min (900 sec)
readonly max_wait_for_sync=900
# default value for termux
readonly termux_bin_path='/data/data/com.termux/files/usr/bin'
# the computer will create this (empty) file to signal that syncing finished;
# should match the corresponding setting in my-phone-sync.sh
readonly sync_done_flag='/data/data/com.termux/files/home/.syncdone'
### END of regular settings, the rest is related to firewall settings
### Firewall Settings, tested on AFWall+
# 'true' -> the phone has firewall, root ssh connection blocked by default, need to allow (temporarily)
# 'false -> doesn't have firewall, root ssh connection allowed
# if the setting is 'false', all the following settings are irrelevant (not used)
readonly have_firewall='false'
# default location
readonly bin_iptables='/system/bin/iptables'
# check with 'sudo ipconfig' when connected to the home network
readonly interface='wlan0'
# IP address of the computer
readonly allowed_ip='192.168.1.200'
# default Termux sshd port
readonly local_ssh_port='8022'
# since AFWall+ works on OUTPUT chain only, all we need is to allow established connection from the comp to sshd on specific interface
readonly iptables_rule="-o $interface -p tcp -d $allowed_ip --sport $local_ssh_port -m conntrack --ctstate ESTABLISHED -j ACCEPT"
### END of Firewall Settings
# write the date-time for the log
date
# don't let CPU to sleep during the sync process
${termux_bin_path}/termux-wake-lock
# flag file should not exist at this stage
[[ -f $sync_done_flag ]] && rm $sync_done_flag
# if needed, open firewall (insert rule into OUTPUT chain)
$have_firewall && $bin_iptables -I OUTPUT $iptables_rule
# start sshd as root (as this script is run as root by crond)
${termux_bin_path}/sshd
# get current time in seconds
start_time_epoch=$(date +%s)
# wait until the computer creates the flag file, but no more than $max_wait_for_sync seconds
while [[ ! -f $sync_done_flag && $(date +%s) -lt $((start_time_epoch + max_wait_for_sync)) ]]; do
sleep 10;
done
# check the reason why while loop terminated, write it out
if [[ -f $sync_done_flag ]]; then
echo "flag file detected, finishing"
rm $sync_done_flag
else
echo "timeout reached, sync probably failed"
fi
# terminate sshd
${termux_bin_path}/pkill sshd
# restore firewall state (delete inserted rule from OUTPUT chain)
$have_firewall && $bin_iptables -D OUTPUT $iptables_rule
# let CPU sleep again
${termux_bin_path}/termux-wake-unlock
# finish termux session
exit
-
Check the
Settings
part in the beginning:-
if you don’t have firewall, up until the
END of regular settings
line -
if you do have firewall, additionally configure the settings in
Firewall Settings
part (according to firewall section), up toEND of Firewall Settings
line.
-
3.1.3. Computer script (my-phone-sync.sh)
-
On the computer, create and make executable (
chmod +x
) the following file in, for example,~/bin
folder:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
#!/bin/bash
# exit if there are unbound variables
set -u
### Settings, change according to your needs
# where on this machine to store synced data (including trailing slash)
readonly local_sync_dir='/home/user/.my-phone-sync/'
# empty file that will be created on the phone after successful syncing,
# to indicate that it can revert the steps it took to allow connection;
# should match the corresponding setting in allow-sync.sh
readonly sync_done_flag='/data/data/com.termux/files/home/.syncdone'
# how many times to try to establish ssh connection to the phone
readonly num_retry_attempts=10
# ssh default timeout is very big, so
readonly ssh_timeout=15
# time to sleep between ssh connection attempts
readonly retry_wait=60
# how the phone is called in ~/.ssh/config
readonly android_host=my-phone
# folders in phone to sync, without trailing slash (important!)
readonly dirs_to_sync=( \
'/data/data/com.android.providers.contacts' \
'/data/data/com.android.providers.calendar' \
'/data/data/com.simplemobiletools.calendar.pro' \
'/data/data/com.simplemobiletools.contacts.pro' \
'/data/data/com.termux/files/home' \
)
# function to send desktop notification, verify that you have 'notify-send' command available
notify_from_script () {
DISPLAY=:0 /usr/bin/notify-send "Phone Syncing" "$@"
}
### END of Settings
# function to send error notification and abort
fail () {
notify_from_script -i error "failed: $@"
exit
}
# if local_sync_dir doesn't exist, create it, abort with message, if failed
[[ -d $local_sync_dir ]] || err_msg=$(mkdir -p $local_sync_dir 2>&1) || fail $err_msg
notify_from_script -i info "checking ssh connection..."
# try ssh connection to phone until success or timeout; abort if ssh login is not root
for i in $(seq $num_retry_attempts); do
# get the ssh session's username (we need root) or ssh error message if connection failed
# note: stderr redirected to stdout
if ssh_output=$(ssh -o ConnectTimeout=$ssh_timeout $android_host whoami 2>&1); then
# if we are here, ssh connection was successful. if we were root, proceed to rsync, otherwise abort
[[ $ssh_output == 'root' ]] && break || fail "ssh user is $ssh_output, not root"
else
# ssh connection failed, tell why and try again
notify_from_script -i info "$ssh_output, retrying in $retry_wait seconds"
fi
# abort if max amount of attempts have been reached
[[ $i -eq $num_retry_attempts ]] && fail "ssh connection failed after $i attempts"
sleep $retry_wait
done
# rsync each dir from the array defined in settings; abort with error message in case of an error
for dir in "${dirs_to_sync[@]}"; do
notify_from_script -i info "syncing $dir"
# $error_msg will receive stderr, while stdout discarded; if rsync fails, send $error_msg and abort
error_msg=$(rsync -a --delete ${android_host}:${dir} $local_sync_dir 2>&1 >/dev/null) || fail $error_msg
done
# let the phone know that sync finished and it should close sshd etc.
ssh $android_host touch $sync_done_flag || fail "failed to create $sync_done_flag"
notify_from_script -i trophy-gold "success"
-
Check the
Settings
part in the beginning up until theEND of Settings
line -
Regarding
folders in phone to sync
setting, see the next sub-section:
3.1.4. What exactly to sync
-
If you use, as suggested above,
Simple Calendar
app using 'regular' (off-line) events, the calendar data is in/data/data/com.simplemobiletools.calendar.pro/
folder. -
As for contacts, the approach recommended above uses
Contact Provider
to keep the data, so the folder to sync is/data/data/com.android.providers.contacts/
. -
To be on the safe side, I recommend to also sync
/data/data/com.simplemobiletools.contacts.pro/
and/data/data/com.android.providers.calendar/
folders. -
A good idea is also to sync
Termux
home folder/data/data/com.termux/files/home/
. -
You may add additional directories to sync, such as notes, messaging app data etc. (generally,
/data/data/<package_name>
).
3.1.5. Test the scripts
-
In the phone, run
sudo ./allow-sync.sh
. Check thatsshd
is listening (e.g., withsudo netstat -pltn
). If you have firewall, check that the needed rule was added (runipto
). -
In the same time, or shortly afterwards, in the computer, run
my-phone-sync.sh
. Pay attention to desktop notifications. -
Upon success, check the folder on the computer to which the phone’s folders should have been synced.
3.2. Setting up cron
for automation
3.2.1. Testing
-
For testing, create some file in
Termux
home (if you chose to sync it as recommended, if not, make any other change in one of the to-be-synced folders). -
Choose a time to test
cron
functionality, say half an hour from the current time (to allow phone to go to sleep). Let’s say you decided to testcron
at 12:00 (for other time, change the setup below accordingly). -
On the computer, run
crontab -e
and add the following line (change the path to the actual location of the phone script):00 12 * * * /home/user/bin/my-phone-sync.sh
-
On the phone:
-
Edit as root the file
/data/crontab
(whichcrond
app uses), and add the following line:00 12 * * * /data/data/com.termux/files/home/allow-sync.sh &> /data/data/com.termux/files/home/cron.log
-
Open
crond
app, and:-
Check "Use wake lock…" and "Create notification…" options;
-
Verify that the line you just added to
/data/crontab
appears in the upper pane; -
Verify the message "Scheduled line…" in the bottom (log) pane.
-
Close
crond
app.
-
-
Verify that
sshd
is not running, and that the firewall (if you have it) is in its default state (by runningipto
). -
Exit
Termux
, let phone sleep.
-
-
When the configured time arrives, pay attention to the desktop notifications on the computer.
-
Upon completion, a notification should appear in the phone (by
crond
). -
Verify that the sync was successful by checking the presence of the file, which you created in the beginning of this section, in the sync folder in the computer.
-
Check the log file on the phone (
/data/data/com.termux/files/home/cron.log
) which should be created bycrond
.
3.2.2. Final setting
-
Chose time when you want the backup process to happen daily. Think of the time when a phone is usually connected to the home network and you are at the computer.
-
Change the time setting accordingly, both in
cron
settings in the computer (bycrontab -e
), and in the phone (by editing/data/crontab
, then re-openingcrond
app). -
After the scheduled time, re-run checks mentioned in the previous sub-section.