RestoreApplicationInternalData » History » Revision 21
Revision 20 (Denis 'GNUtoo' Carikli, 10/30/2020 02:45 PM) → Revision 21/49 (Denis 'GNUtoo' Carikli, 10/30/2020 03:33 PM)
h1. RestoreApplicationInternalData
{{toc}}
h2. /!\ Warning: Draft
This article is in draft form and is being written:
* Everybody is welcome to contribute
* Some things might not be accurate yet, so beware before using the information contained in it.
h2. Rationale
In some case, it is useful to be able to restore internal applications data:
For instance you might need to move the data of an application from a device to another if you want to switch device.
Another use case is if /data/system/packages.xml or /data/system/appops.xml get corrupted, applications can loose access to their data. This can make the launcher and other applications crash.
So while it is possible to recover from that by wiping the data partition in the recovery, sometimes it's very impractical to do that because you might have important data like silence encryption keys and established sessions that you don't want to loose.
h2. Goals
This howto will explain how to move silence data from a device to another.
For instance you could want to move from a Galaxy SIII (GT-I9300) to a Galaxy SII (GT-I9100) or vice versa, and you might not want to recreate keys, sessions, etc when moving device.
This could also be adapted to restore application data from a block device level backup of the internal eMMC or the internal data partition.
h2. Silence
Silence has been chosen as an example for this tutorial because:
* It's an application commonly used
* Loosing its data (key, sessions) can be painful
Silence stores its data in the internal application storage. As far as I know it's not supposed to store any data on the microSD or user storage beside potential backups.
It might be interesting to make additional tutorial for other cases. For instance for:
* Applications that also require data to be on the microSD or user storage.
* System applications that have their data in a database.
h2. Requirements
This howto assumes that the data partition is unencrypted. If you know how to open encrypted data partition in the recovery, or in a GNU/Linux distribution, it would be great to either modify this tutorial to add information on how to do it, or contact us on the mailing list or through the bugreporting system about it.
h2. TODO
* Explain why we need the uid/gid
* Explain why the ls -ld gives the application uid/gid
* Point to how to get root
* Explain how to handle a corrupted /data/system/packages.xml and /data/system/appops.xml
* Explain how to mount a full backup, and why not to restore full backup completely
* Explain how and why create a tarball of the application data
Once we have a tarball backup of the application data we need to reboot in the recovery to avoid any writes to the data filesystem.
h2. Backuping Silence's data from the old device
To correctly backup the application data, you need to make sure that the application isn't writing to its data while you are doing the backup and that the data is consistent.
To make sure of that the easiest way to do that is to go in the recovery: this way the application will not be started automatically and the data is consistent. You will also need a root shell there, so make sure to setup that beforehand.
If everything is setup correctly, running @adb shell@ from your computer should result in a shell that now looks like this:
<pre>
root@m0:/ #
</pre>
Once that you have a shell in the recovery, you need to mount the data partition in /data from within that shell.
For the Galaxy SIII (GT-I9300), this can be done from your computer with this command:
<pre>
mount /dev/block/platform/*/by-name/USERDATA /data
</pre>
We will then create an archive of the silence data.
This makes things much easier because with an archive:
* We can easily store and move that data.
* The archive preserves all its permissions, and it's best to keep them as applications might not work correctly otherwise. For instance in GNU/Linux, the 'sshd' program will refuse to work if some of its configuration files have the wrong permissions.
The silence data is in @/data/data/org.smssecure.smssecure/@. To make sure that the archive is created correctly and that once decompressed it will create the @org.smssecure.smssecure@ directory (with all the silence data in it) in the current directory, we need to go to @/data/data@ first.
To do that, we can go in /data/data with the following command:
<pre>
cd /data/data
</pre>
We can then create the archive in the recovery with that command:
<pre>
root@m0:/data/data # tar cvpf /org.smssecure.smssecure.tar org.smssecure.smssecure/
</pre>
This will create the @/org.smssecure.smssecure.tar@ file which is our archive.
We will then exit the recovery shell as we will then need to type commands on your computer.
You can exit the recovery shell with the following command:
<pre>
root@m0:/ # exit
</pre>
We're now back on your computer.
As for now, we've created the archive but it's still in the recovery, so we need to copy it to your computer.
This can be done with the following command:
<pre>
$ adb pull /org.smssecure.smssecure.tar ./
</pre>
It will create the @org.smssecure.smssecure.tar@ file in the current directory.
We will then need to verify that the archive was created correctly.
More precisely we need to check that it will be extracted in the @org.smssecure.smssecure/@ directory, in the current directory, otherwise it could mess up the data partition when it's being uncompressed.
To do that we can list its content with the following command:
<pre>
$ tar tvf org.smssecure.smssecure.tar
</pre>
It should output something that will look more or less like that:
<pre>
drwxr-x--x u0_a61/u0_a61 0 2020-10-28 18:58 org.smssecure.smssecure/
lrwxrwxrwx root/root 0 2020-10-28 18:58 org.smssecure.smssecure/lib -> /data/app/org.smssecure.smssecure-1/lib/arm
[...]
</pre>
What is important to check is that the files and directories starts with @org/smssecure.smssecure/@ or @./org.smssecure.smssecure/@. If not something went wrong during the creation of the backup (for instance you might have forgetten to go in the @/data/data/@ recovery, or this tutorial could contain mistakes or be out of date).
When all that is done, you can power off the device by going going in @Advanced@ in the recovery menu and then selecting @Power off@. If you want to reboot to Replicant instead you could select @Reboot system now@ in the recovery menu instead.
h2. Restoring Silence's data to the new device
In Once the previous section, we did a backup of silence data that is now contained in the org.smssecure.smssecure.tar archive that you have on your computer.
In this section, we will restore that archive to a new device.
First you need to install silence, and if possible, give it the same permissions it had before. If not, we can still fix that later as silence is not required to boot the device, so we can still access the Android settings and fix that afterward.
Silence is installed before restoring its data for several reasons:
* We need to give it the necessary permissions. If you don't do that it might crash at startup once the data is restored. However it's still possible to fix it when it happens.
* We need a new @/data/data/org.smssecure.smssecure@ directory to be created to be able to easily find the new username in which silence is running, as right after restoring the backup will need to fix the permissions of the new data to use this new username.
Once Silence is installed, you will also need to enable root shells in the recovery like you did when doing a backup of Silence data.
Here too, it's done this way to make sure that the application isn't writing to its data while you are restoring the data from the older device.
To make sure of that, here too, the easiest way to do that is to go in the recovery: this way the application will not be started automatically and the data will be restored while the application isn't started.
You will also need a root shell in the recovery of this new device, so make sure to setup that beforehand.
If everything is setup correctly, running @adb shell@ from your computer should result in a shell that now looks like this:
<pre>
root@m0:/ #
</pre>
Once that you have a shell in the recovery, you need to mount the data partition in /data from within that shell.
partition.
For the Galaxy SIII (GT-I9300), this can be done from your computer with this command:
<pre>
mount /dev/block/platform/*/by-name/USERDATA /data $ adb shell "mount /dev/block/platform/dw_mmc/by-name/USERDATA /data"
</pre>
If Then we extract the archive we made eariler, it will create the @org.smssecure.smssecure@ directory and extract everything in it.
Since the data of Silence is in @/data/data/org.smssecure.smssecure/@ we then need to go in @/data/data@ before extracting get a root shell inside the archive. recovery. This way when extracting it, everything will can be at the place it's supposed to be.
To do that, we can go in /data/data done with the following @adb shell@ command:
<pre>
cd /data/data $ adb shell
</pre>
However Then we can't restore Silence data yet as: assume that you are in /data/data to simplify this tutorial.
* It could mix both data sets.
* We You will need some information (silence username) contained to remember adjust all other commands if you are not in this directory.
To go in /data/data, you can use the new silence data. following command:
<pre>
root@m0:/ # cd /data/data/
root@m0:/data/data #
</pre>
As applications are sandboxed, and that as part of that sandboxing, they have their own usernames, usersname, we need to retrieve this username, as we'll reuse it to fix the restored silence data permissions.
username.
To get do that username we can just use @ls -ld@ on the directory holding the application internal data data.
The directory has the internal name of the newly installed silence (which is application.
Here are some well known name correspondances:
| Internal name | Application |
| org.smssecure.smssecure | Silence |
| com.android.dialer | Dialer (Android's stock dialer application) |
| fil.libre.repwifiapp | RepWiFi |
For pakcages comming from f-droid, the f-droid website can find the correspondance.
For instance the "Silence page":https://f-droid.org/en/packages/org.smssecure.smssecure/ has @org.smssecure.smssecure@ in @/data/data/org.smssecure.ssmsecure@). its URL and inside the page.
So with @ls -ld@ we can find the application username in this way:
<pre>
root@m0:/data/data # ls -ld org.smssecure.smssecure
__bionic_open_tzdata: couldn't find any tzdata when looking for localtime!
__bionic_open_tzdata: couldn't find any tzdata when looking for GMT!
__bionic_open_tzdata: couldn't find any tzdata when looking for posixrules!
drwxr-x--x 2 u0_a61 u0_a61 4096 2012-01-01 00:01 org.smssecure.smssecure
</pre>
Here the users and groups are @u0_a61@.
We will then need use this information later on to restore the silence data from the other device: If we restore Silence's that data as-is it will most likely have wrong permissions: when the the Silence silence application was installed on the older device, it was assigned an username. As this username depends on the number of applications that were installed before it, we cannot expect it to always be the same between the two devices.
We will It's also best to move or delete the data of the silence we just installed. Moving it can be done with this command: don't want:
<pre>
root@m0:/data/data # mv org.smssecure.smssecure org.smssecure.smssecure.delme
</pre>
Moving it has several advantages over just deleting it: advantages:
* We can still verify the username later on to see if it matches with the backup we restored.
* We can interrupt this tutorial more easily if something goes wrong.
To be sure that the data is restored in the right directory, Here we will need to verify that the archive will extract its files in the @org.smssecure.smssecure@ directory and not directly in the current directory (which should be @/data/data@ as we moved into it before):
The archive can be verified with the following command: which is @/data/data@:
<pre>
root@m0:/data/data # tar tf /org.smssecure.smssecure.tar
</pre>
It will output something that looks a bit like that:
<pre>
./org.smssecure.smssecure/
./org.smssecure.smssecure/lib -> /data/app/org.smssecure.smssecure-1/lib/arm
[...]
</pre>
We will need to verify Here we see that everything starts with @./org.smssecure.smssecure/@ or @org.smssecure.smssecure/@. Here it is the case, (or @org.smssecure.smssecure/@) so the archive is it's good.
TODO: move this part earlier
If we had something like that instead:
<pre>
root@m0:/data/data # tar tf /org.smssecure.smssecure.tar
./lib -> /data/app/org.smssecure.smssecure-1/lib/arm
[...]
</pre>
Then it's best to recreate the archive. In that case, if
If you want to abort to restart later on, need more time you could also move back org.smssecure.smsecure.delme to org.smssecure.smssecure if needed.
If the archive was good, we We can then proceed to extract the application data.
This can be done with data (with the following command: username from the old device):
<pre>
root@m0:/data/data # tar xpf /org.smssecure.smssecure.tar --numeric-owner
</pre>
Here we can see that the username differs from the one we need with the following command: need:
<pre>
root@m0:/data/data # ls -ld org.smssecure.smssecure
</pre>
It should then print something like that ( the lines starting with @__bionic_open_tzdata:@ can be ignored):
<pre>
__bionic_open_tzdata: couldn't find any tzdata when looking for localtime!
__bionic_open_tzdata: couldn't find any tzdata when looking for GMT!
__bionic_open_tzdata: couldn't find any tzdata when looking for posixrules!
drwxr-x--x 9 u0_a63 u0_a63 4096 2012-01-01 00:21 org.smssecure.smssecure
</pre>
Here we We have @u0_a63@, while the new silence data used @u0_a61@, so @u0_a63@ instead of @u0_a61@.
So we need to fix it.
This can be done with the @chown@ command, like that:
<pre>
root@m0:/data/data # chown u0_a61:u0_a61 -R org.smssecure.smssecure
root@m0:/data/data #
</pre>
You need to replace @u0_a61@ by the username you found earlier.
When At this is done, point, we don't need the @org.smssecure.smssecure.delme@ directory anymore, and it's best to remove it not to create any issues later on.
This can be done with the following command:
<pre>
root@m0:/data/data # rm -rf org.smssecure.smssecure.delme
root@m0:/data/data #
</pre>
If everything went fine, it shouldn't output anything.
We are not done yet at restoring the permissions as in In addition to the standard unix permissions which we just fixed, permissions, Android also uses selinux, which so we also has its own need to fixup the selinux permissions.
The restorecon command can be used to fixup selinux permissions.
for that. Here's its help (it can be shown just by typing @restorecon@):
help:
<pre>
root@m0:/data/data # restorecon
usage: restorecon [-D] [-F] [-R] [-n] [-v] FILE...
Restores the default security contexts for the given files.
-D apply to /data/data too
-F force reset
-R recurse into directories
-n don't make any changes; useful with -v to see what would change
-v verbose: show any changes
restorecon: Needs 1 argument
</pre>
So to use it to fixup the selinux permissions, we can use the following command:
<pre>
root@m0:/data/data # restorecon -D -F -R -v /data/
</pre>
The order of the arguments (-D, -F, etc) seem to be important here as the wrong order might result in nothing being done.
Without the @-v@ argument and with the wrong order of argument, it might make you think that it did its job while it did nothing.
It will then print something that looks like that:
<pre>
SELinux: Loaded file_contexts contexts from /file_contexts.
[...]
SELinux: Relabeling /data/data/org.smssecure.smssecure from u:object_r:system_data_file:s0 to u:object_r:app_data_file:s0:c512,c768.
SELinux: Relabeling /data/data/org.smssecure.smssecure/lib from u:object_r:system_data_file:s0 to u:object_r:app_data_file:s0:c512,c768.
[...]
</pre>
Now the permissions The premissions fixing is finally now done.
So we can then umount the data partition and reboot.
To do that first we need to go outside of data, else the mount will fail. This can be done with this command: fail:
<pre>
root@m0:/data/data # cd / /For instance you could want to move from a Galaxy SIII (GT-I9300) to a Galaxy SII (GT-I9100) or vice versa, and you might not want to recreate keys, sessions, etc when moving device.
root@m0:/ #
</pre>
Then we can simply unmount /data/ it with this command:
<pre>
root@m0:/ # umount /data/
root@m0:/ #
</pre>
Then it's a good practice to make sure that everything is written to the data partition before rebooting.
We can do that with this the @sync@ command:
<pre>
root@m0:/ # sync
</pre>
And we can finally reboot. It can be done with this command: reboot:
<pre>
root@m0:/ # reboot
</pre>
h2. How to fix Silence if it still refuses to start with the restored data.
After rebooting, silence may still refuse refused to start if it doesn't have the right permissions. start.
To identify if it doesn't start due to missing permissions, you can use So I looked at the following command logs from your computer: my laptop with this command:
<pre>
$ adb logcat -b main
</pre>
Then you need to wait I waited until no more new logs were printed, otherwise it'd be harder printed.
I then press enter multiple times to find the crash we're looking for in this huge amount of logs.
To better identify where the part where Silence crash starts and stops, you could for instance create a separation in the logs being printed by adding
with many new lines once no more to be able to get back to the begining of the new logs are printed, right before launching silence. This can be done by easily.
Then I launched silence and started pressing enter multiple times or by pressing many time a character like @=@ again to create a visible line.
You can then launch silence, and as soon as mark the crash is done, do end of the same to mark the end, not to have too much logs to read. silence related logs.
When Silence crashed, the following appeard I then had that in the these new logs:
<pre> <pre>For instance you could want to move from a Galaxy SIII (GT-I9300) to a Galaxy SII (GT-I9100) or vice versa, and you might not want to recreate keys, sessions, etc when moving device.
01-01 01:27:48.260 4126 4126 D AndroidRuntime: Shutting down VM
01-01 01:27:48.265 4126 4126 E AndroidRuntime: FATAL EXCEPTION: main
01-01 01:27:48.265 4126 4126 E AndroidRuntime: Process: org.smssecure.smssecure, PID: 4126
01-01 01:27:48.265 4126 4126 E AndroidRuntime: Theme: themes:{}
01-01 01:27:48.265 4126 4126 E AndroidRuntime: java.lang.RuntimeException: Unable to create application org.smssecure.smssecure.ApplicationContext: java.lang.SecurityException: getActiveSubscriptionInfoList: Neither user 10061 nor current process has android.permission.READ_PHONE_STATE.
01-01 01:27:48.265 4126 4126 E AndroidRuntime: at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4754)
[...]
</pre>
Here we can clearly see that it's a permission issue as So to fix it says that @Neither user 10061 nor current process has android.permission.READ_PHONE_STATE@.
And we know it's from silence as it has @Process: org.smssecure.smssecure@.
To fix that issue, I went in @Settings->Apps->Silence->Permissions@ and gave it all the permissions it needed.
I had this issue because I didn't even launch silence after installing it, so it couldn't cound't ask me for the permissions it needed.
And the silence of the former device probably wrote in its data that it already asked the permissions not to re-ask for it each time.
h2. How to find which directory holds the internal data of an application.
The directories holding the data are in @/data/data/@ and have the internal name of the application.
Here are some well known name correspondances:
| Internal name | Application |
| org.smssecure.smssecure | Silence |
| com.android.dialer | Dialer (Android's stock dialer application) |
| fil.libre.repwifiapp | RepWiFi |
So silence RepWiFi will be @/data/data/fil.libre.repwifiapp@
For pakcages comming from f-droid, the f-droid website can find the correspondance.
For instance the "Silence page":https://f-droid.org/en/packages/org.smssecure.smssecure/ has @org.smssecure.smssecure@ in its URL and inside the page. permissions.