Replicant

h1. RestoreApplicationInternalData

{{toc}}

h2. /!\ Warning: Draft

This article is in draft form and is being written:

* Everybody is welcome to contribute

* Some things might not be accurate yet, so beware before using the information contained in it.

h2. TODO

* Explain how to handle a corrupted /data/system/packages.xml and /data/system/appops.xml

* Explain how to mount a full backup, and why not to restore full backup completely

* Explain how and why create a tarball of the application data

h2. Rationale

In some case, it is useful to be able to restore internal applications data:

For instance you might need to move the data of an application from a device to another if you want to switch device.

Another use case is if /data/system/packages.xml or /data/system/appops.xml get corrupted, applications can loose access to their data. This can make the launcher and other applications crash. 

So while it is possible to recover from that by wiping the data partition in the recovery, sometimes it's very impractical to do that because you might have important data like silence encryption keys and established sessions that you don't want to loose.

h2. Goals

This howto will explain how to move silence data from a device to another.

For instance you could want to move from a Galaxy SIII (GT-I9300) to a Galaxy SII (GT-I9100) or vice versa, and you might not want to recreate keys, sessions, etc when moving device.

This could also be adapted to restore application data from a block device level backup of the internal eMMC or the internal data partition.

h2. Silence

Silence has been chosen as an example for this tutorial because:

* It's an application commonly used

* Loosing its data (key, sessions) can be painful

Silence stores its data in the internal application storage. As far as I know it's not supposed to store any data on the microSD or user storage beside potential backups.

It might be interesting to make additional tutorial for other cases. For instance for:

* Applications that also require data to be on the microSD or user storage.

* System applications that have their data in a database.

h2. Requirements 

This howto assumes that the data partition is unencrypted. If you know how to open encrypted data partition in the recovery, or in a GNU/Linux distribution, it would be great to either modify this tutorial to add information on how to do it, or contact us on the mailing list or through the bugreporting system about it.

h2. Setup ADB

Follow the instructions for [[ADB|setting up ADB on your computer]] so that you can access a root shell on your device.

*NOTE*: when prompted on your Replicant device, make sure that you check the box that says *Always allow from this computer* when you grant your computer USB debugging permissions. Otherwise, you will be unable to obtain root shell access on your Replicant device when you reboot it into the recovery OS to actually perform the backup.

*NOTE*: for security reasons, you may want to [[ADB#Revoking-all-computers-USB-debugging-permissions|revoke these non-expiring permissions]] once the backup is complete.

h2. Backuping Silence's data from the old device

To correctly backup the application data, you need to make sure that the application isn't writing to its data while you are doing the backup and that the data is consistent.

To make sure of that the easiest way to do that is to go in the recovery: this way the application will not be started automatically and the data is consistent. You will also need a root shell there, so make sure to setup that beforehand.

If everything is setup correctly, running @adb shell@ from your computer should result in a shell that now looks like this:

<pre>

root@m0:/ #

</pre>

Once that you have a shell in the recovery, you need to mount the data partition in /data from within that shell.

For the Galaxy SIII (GT-I9300), this can be done from your computer with this command:

<pre>

mount /dev/block/platform/*/by-name/USERDATA /data

</pre>

We will then create an archive of the silence data. 

This makes things much easier because with an archive:

* We can easily store and move that data.

* The archive preserves all its permissions, and it's best to keep them as applications might not work correctly otherwise. For instance in GNU/Linux, the 'sshd' program will refuse to work if some of its configuration files have the wrong permissions.

The silence data is in @/data/data/org.smssecure.smssecure/@. To make sure that the archive is created correctly and that once decompressed it will create the @org.smssecure.smssecure@ directory (with all the silence data in it) in the current directory, we need to go to @/data/data@ first.

To do that, we can go in /data/data with the following command:

<pre>

cd /data/data

</pre>

We can then create the archive in the recovery with that command:

<pre>

root@m0:/data/data # tar cvpf /org.smssecure.smssecure.tar org.smssecure.smssecure/

</pre>

This will create the @/org.smssecure.smssecure.tar@ file which is our archive.

We will then exit the recovery shell as we will then need to type commands on your computer.

You can exit the recovery shell with the following command:

<pre>

root@m0:/ # exit

</pre>

We're now back on your computer.

As for now, we've created the archive but it's still in the recovery, so we need to copy it to your computer. 

This can be done with the following command:

<pre>

$ adb pull /org.smssecure.smssecure.tar ./

</pre>

It will create the @org.smssecure.smssecure.tar@ file in the current directory.

We will then need to verify that the archive was created correctly. 

More precisely we need to check that it will be extracted in the @org.smssecure.smssecure/@ directory, in the current directory, otherwise it could mess up the data partition when it's being uncompressed.

To do that we can list its content with the following command:

<pre>

tar tvf org.smssecure.smssecure.tar

</pre>

It should output something that will look more or less like that:

<pre>

drwxr-x--x u0_a61/u0_a61     0 2020-10-28 18:58 org.smssecure.smssecure/

lrwxrwxrwx root/root         0 2020-10-28 18:58 org.smssecure.smssecure/lib -> /data/app/org.smssecure.smssecure-1/lib/arm

[...]

</pre>

What is important to check is that the files and directories starts with @org/smssecure.smssecure/@ or @./org.smssecure.smssecure/@. If not something went wrong during the creation of the backup (for instance you might have forgetten to go in the @/data/data/@ recovery, or this tutorial could contain mistakes or be out of date).

we all is done, we can finally umount the data partition and reboot the device.

To do that, we need to go back in the device recovery with the following command:

<pre>

adb shell

</pre>

It should output something that looks more or less like that:

<pre>

root@m0:/ #

</pre>

To do that first we need to go outside of data, else the mount will fail. This can be done with this command:

<pre>

cd /

</pre>

Then we can simply unmount /data/ with this command:

<pre>

umount  /data/

</pre>

Then it's a good practice to make sure that everything is written to the data partition before rebooting.

We can do that with this command:

<pre>

sync

</pre>

And we can finally reboot or shutdown the device.

Rebooting can be done by selecting @Reboot system now@ in the recovery menu.

Shutting down the device can be done by selecting @Power off@ in @Advanced@  in the recovery menu.

h2. Restoring Silence's data to the new device

In the previous section, we did a backup of silence data that is now contained in the org.smssecure.smssecure.tar archive that you have on your computer.

In this section, we will restore that archive to a new device.

First you need to install silence, and if possible, give it the same permissions it had before. If not, we can still fix that later as silence is not required to boot the device, so we can still access the Android settings and fix that afterward. 

Silence is installed before restoring its data for several reasons:

* We need to give it the necessary permissions. If you don't do that it might crash at startup once the data is restored. However it's still possible to fix it when it happens.

* We need a new @/data/data/org.smssecure.smssecure@ directory to be created to be able to easily find the new username in which silence is running, as right after restoring the backup will need to fix the permissions of the new data to use this new username.

Once Silence is installed, you will also need to enable root shells in the recovery like you did when doing a backup of Silence data.

Here too, it's done this way to make sure that the application isn't writing to its data while you are restoring the data from the older device.

To make sure of that, here too, the easiest way to do that is to go in the recovery: this way the application will not be started automatically and the data will be restored while the application isn't started.

You will also need a root shell in the recovery of this new device, so make sure to setup that beforehand.

If everything is setup correctly, running @adb shell@ from your computer should result in a shell that now looks like this:

<pre>

root@m0:/ #

</pre>

Once that you have a shell in the recovery, you need to mount the data partition in /data from within that shell.

For the Galaxy SIII (GT-I9300), this can be done from your computer with this command:

<pre>

mount /dev/block/platform/*/by-name/USERDATA /data

</pre>

If we extract the archive we made eariler, it will create the @org.smssecure.smssecure@ directory and extract everything in it. 

Since the data of Silence is in @/data/data/org.smssecure.smssecure/@ we then need to go in @/data/data@ before extracting the archive. This way when extracting it, everything will be at the place it's supposed to be.

To do that, we can go in /data/data with the following command:

<pre>

cd /data/data

</pre>

However we can't restore Silence data yet as:

* It could mix both data sets.

* We need some information (silence username) contained in the new silence data.

As applications are sandboxed, and that as part of that sandboxing, they have their own usernames, we need to retrieve this username, as we'll reuse it to fix the restored silence data permissions.

To get that username we can just use @ls -ld@ on the directory holding the internal data of the newly installed silence (which is in @/data/data/org.smssecure.ssmsecure@).

So with @ls -ld@ we can find the application username in this way:

<pre>

root@m0:/data/data # ls -ld org.smssecure.smssecure

__bionic_open_tzdata: couldn't find any tzdata when looking for localtime!

__bionic_open_tzdata: couldn't find any tzdata when looking for GMT!

__bionic_open_tzdata: couldn't find any tzdata when looking for posixrules!

drwxr-x--x 2 u0_a61 u0_a61 4096 2012-01-01 00:01 org.smssecure.smssecure

</pre>

Here the users and groups are @u0_a61@.

We will then need use this information later on to restore the silence data from the other device: If we restore Silence's data as-is it will most likely have wrong permissions: when the the Silence application was installed on the older device, it was assigned an username. As this username depends on the number of applications that were installed before it, we cannot expect it to always be the same between the two devices.

We will also move the data of the silence we just installed. Moving it can be done with this command:

<pre>

mv org.smssecure.smssecure org.smssecure.smssecure.delme

</pre>

Moving it has several advantages over just deleting it:

* We can still verify the username later on to see if it matches with the backup we restored.

* We can interrupt this tutorial more easily if something goes wrong.

To be sure that the data is restored in the right directory, we will need to verify that the archive will extract its files in the @org.smssecure.smssecure@ directory and not directly in the current directory (which should be @/data/data@ as we moved into it before):

The archive can be verified with the following command:

<pre>

tar tf /org.smssecure.smssecure.tar

</pre>

It will output something that looks a bit like that:

<pre>

./org.smssecure.smssecure/

./org.smssecure.smssecure/lib -> /data/app/org.smssecure.smssecure-1/lib/arm

[...]

</pre>

We will need to verify that everything starts with @./org.smssecure.smssecure/@ or @org.smssecure.smssecure/@. Here it is the case, so the archive is good.

If we had something like that instead:

<pre>

root@m0:/data/data # tar tf /org.smssecure.smssecure.tar

./lib -> /data/app/org.smssecure.smssecure-1/lib/arm

[...]

</pre>

Then it's best to recreate the archive. In that case, if you want to abort to restart later on, you could also move back org.smssecure.smsecure.delme to org.smssecure.smssecure if needed.

If the archive was good, we can then proceed to extract the application data.

This can be done with the following command:

<pre>

tar xpf /org.smssecure.smssecure.tar --numeric-owner

</pre>

Here we can see that the username differs from the one we need with the following command:

<pre>

ls -ld org.smssecure.smssecure 

</pre>

It should then print something like that ( the lines starting with @__bionic_open_tzdata:@ can be ignored):

<pre>

__bionic_open_tzdata: couldn't find any tzdata when looking for localtime!

__bionic_open_tzdata: couldn't find any tzdata when looking for GMT!

__bionic_open_tzdata: couldn't find any tzdata when looking for posixrules!

drwxr-x--x 9 u0_a63 u0_a63 4096 2012-01-01 00:21 org.smssecure.smssecure

</pre>

Here we have @u0_a63@, while the new silence data used @u0_a61@, so we need to fix it.

This can be done with the @chown@ command, like that:

<pre>

root@m0:/data/data # chown u0_a61:u0_a61 -R org.smssecure.smssecure            

root@m0:/data/data # 

</pre>

You need to replace @u0_a61@ by the username you found earlier.

When this is done, we don't need the @org.smssecure.smssecure.delme@ directory anymore, and it's best to remove it not to create any issues later on.

This can be done with the following command:

<pre>

rm -rf org.smssecure.smssecure.delme

</pre>

If everything went fine, it shouldn't output anything.

We are not done yet at restoring the permissions as in addition to the standard unix permissions which we just fixed, Android also uses selinux, which also has its own permissions.

The restorecon command can be used to fixup selinux permissions.

Here's its help (it can be shown just by typing @restorecon@):

<pre>

usage: restorecon [-D] [-F] [-R] [-n] [-v] FILE...

Restores the default security contexts for the given files.

-D	apply to /data/data too

-F	force reset

-R	recurse into directories

-n	don't make any changes; useful with -v to see what would change

-v	verbose: show any changes

restorecon: Needs 1 argument

</pre>

So to use it to fixup the selinux permissions, we can use the following command:

<pre>

restorecon -D -F -R -v /data/

</pre>

The order of the arguments (-D, -F, etc) seem to be important here as the wrong order might result in nothing being done.

Without the @-v@ argument and with the wrong order of argument, it might make you think that it did its job while it did nothing.

It will then print something that looks like that:

<pre>

SELinux: Loaded file_contexts contexts from /file_contexts.

[...]

SELinux:  Relabeling /data/data/org.smssecure.smssecure from u:object_r:system_data_file:s0 to u:object_r:app_data_file:s0:c512,c768.

SELinux:  Relabeling /data/data/org.smssecure.smssecure/lib from u:object_r:system_data_file:s0 to u:object_r:app_data_file:s0:c512,c768.

[...]

</pre>

Now the permissions fixing is finally done.

we can then umount the data partition and reboot.

To do that first we need to go outside of data, else the mount will fail. This can be done with this command:

<pre>

cd /

</pre>

Then we can simply unmount /data/ with this command:

<pre>

umount  /data/

</pre>

Then it's a good practice to make sure that everything is written to the data partition before rebooting.

We can do that with this command:

<pre>

sync

</pre>

And we can finally reboot or shutdown the device.

Rebooting can be done by selecting @Reboot system now@ in the recovery menu.

Shutting down the device can be done by selecting @Power off@ in @Advanced@  in the recovery menu.

h2. How to fix Silence if it still refuses to start with the restored data.

After rebooting, silence may still refuse to start if it doesn't have the right permissions.

To identify if it doesn't start due to missing permissions, you can use the following command from your computer:

<pre>

adb logcat -b main

</pre>

Then you need to wait until no more new logs were printed, otherwise it'd be harder to find the crash we're looking for in this huge amount of logs.

To better identify where the part where Silence crash starts and stops, you could for instance create a separation in the logs being printed by adding

many new lines once no more logs are printed, right before launching silence. This can be done by pressing enter multiple times or by pressing many time a character like @=@ to create a visible line.

You can then launch silence, and as soon as the crash is done, do the same to mark the end, not to have too much logs to read.

When Silence crashed, the following appeard in the logs:

<pre>

01-01 01:27:48.260  4126  4126 D AndroidRuntime: Shutting down VM

01-01 01:27:48.265  4126  4126 E AndroidRuntime: FATAL EXCEPTION: main

01-01 01:27:48.265  4126  4126 E AndroidRuntime: Process: org.smssecure.smssecure, PID: 4126

01-01 01:27:48.265  4126  4126 E AndroidRuntime: Theme: themes:{}

01-01 01:27:48.265  4126  4126 E AndroidRuntime: java.lang.RuntimeException: Unable to create application org.smssecure.smssecure.ApplicationContext: java.lang.SecurityException: getActiveSubscriptionInfoList: Neither user 10061 nor current process has android.permission.READ_PHONE_STATE.

01-01 01:27:48.265  4126  4126 E AndroidRuntime: 	at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4754)

[...]

</pre>

Here we can clearly see that it's a permission issue as it says that @Neither user 10061 nor current process has android.permission.READ_PHONE_STATE@.

And we know it's from silence as it has @Process: org.smssecure.smssecure@.

To fix that issue, I went in @Settings->Apps->Silence->Permissions@ and gave it all the permissions it needed.

I had this issue because I didn't even launch silence after installing it, so it couldn't ask me for the permissions it needed.

And the silence of the former device probably wrote in its data that it already asked the permissions not to re-ask for it each time.

h2. How to find which directory holds the internal data of an application.

The directories holding the data are in @/data/data/@ and have the internal name of the application.

Here are some well known name correspondances:

| Internal name           | Application                                 |

| org.smssecure.smssecure | Silence                                     |

| com.android.dialer      | Dialer (Android's stock dialer application) |

| fil.libre.repwifiapp    | RepWiFi                                     |

So silence RepWiFi will be @/data/data/fil.libre.repwifiapp@

For pakcages comming from f-droid, the f-droid website can find the correspondance.

For instance the "Silence page":https://f-droid.org/en/packages/org.smssecure.smssecure/ has @org.smssecure.smssecure@ in its URL and inside the page.

h2. Revoke USB debugging permissions

If you don't need @USB debugging permissions@ anymore, it might be a good idea to remove them. The [[ADB#Revoking-all-computers-USB-debugging-permissions|Revoking all computer's USB debugging permissions]] section in the ADB wiki page explains how to do that.

That's it! Your device's EFS partition is now backed up. Your device should be running Replicant normally again.