How to get low level cellular network information on Android phone.

At the beginning of this year me and my friend made an (weak) attempt to create an Android app which would gather and display low level information from cellular network.

This attempt had been ended about half a year ago and I hadn’t gotten very far – just explored the possibility to make such an app so I do not remember or even know the proper terminology, unfortunately. I can’t say which information precisely I intended to pull out of my phone – the goal had been formulated as “as much as possible”.
I was absolutely noob in the field of cellular network communications when I started this project and I do not feel any changes in myself after this project has ended 🙂
The project has died on the stage of estimation of its difficulty. It had been admitted that difficulty is too high for a team of 2 people.

Though I investigated the existence of similar software at the beginning of the project – only months later I accidentally found the Chinese app that apparently does the things we wanted to do. Except call flow. So, If you want just an inexpensive app to use you probably should use this app or maybe there is another somewhere. It seems that this program also uses approach, described in current article.

After a lot of futile reading of different forums on the internet I concluded that our goal couldn’t be achieved using only hard work, because there is no API for functions that we need though they are exist in Radio Interface Layer (RIL) libraries.

Eventually I saw two possible ways:
1) To fight with RIL and try to get information from it, using RIL_REQUEST_OEM_HOOK_RAW function from hidden API. There are few hints that it is possible – patched RIL for some HTC phones, EVA’s project, Doc containing an example of usage of RIL_REQUEST_OEM_HOOK_RAW.
But it is unknown how much we can get out there and the usage of the low level RIL functions requires modifications much heavier than even root access (you either have to make your app a system app to access hidden API or build your own version of Android to make this API open).
2) To use Qualcomm’s DIAG interface. (mobileinsight and snoopsnitch projects use this approach)

Eventually I began to work with DIAG interface because there are open source projects which could be used as an example, though there are no docs available on the interface itself and root privileges are required to access it.
Actually, my app requires root privileges only to access /dev/diag and to enable permissive mode for seLinux. You could do it yourself by executing from terminal (local app or adb) as root:

chmod 666 /dev/diag
echo 0 > /sys/fs/selinux/enforce

Unfortunately this interface couldn’t be used properly without docs and I could’t find them. I gathered hundreds of megabytes of old Qualcomm’s leaked docs and gigabytes of leaked sources but there were no DIAG docs.
Apparently wanted docs (Interface Control Documents ICDs) should have names:
80-B5525-2 QTI Tools Serial Interface Control Document for GSM, GPRS, and EGPRS
80-B5704-2 QTI Tools Serial Interface Control Document for WCDMA/HSPA+
80-VP457-2 QTI Tools Serial Interface Control Document for LTE
80-B1188-2 QTI Tools Serial Interface Control Document for CDMA 1X and 1xEV
80-N9184-3 QTI Tools Serial Interface Control Document for TDSCDMA
80-NR215-3 QTI Tools Serial Interface Control Document for EMBMS Middleware

The final decision in which way the app should be created had been made when I came across this paper. It’s a brief description of the open source project mobileinsight.net
From this paper I concluded that significant part of needed functionality had already been implemented by this project and I just need to create a relatively small part on top of that. After reading this document I also had an impression that the clear understanding of DIAG functionality could be inferred from open source drivers.
I myself peered about a week into those driver files and couldn’t find enough clues to understand DIAG initialization sequence and the structure of its packets.
But that was me. My knowledge and coding ability is a fraction of percent of those that guys from mobileinsight have.

To make your own opinion on the architecture of their rather complex project you should go and visit their github.
For my limited point of view their project consisted of:
1) diag_revealer – a separate daemon process written in C language which always exists in memory. It reads standard Qualcomm’s Diag.cfg file upon start, sends it to DIAG and uses UNIX FIFO to send data from DIAG interface to other parts of the program. Basic principles of diag_revealer are well described on project’s site.
Interestingly enough, Snoopsnitch also has a separate process similar to diag_revealer, but with two FIFO pipes, so it apparently can accept some commands (one pipe IN and one OUT).

2) Some chunk of code (C/C++) responsible for creating Diag.cfg file. This file should initialize DIAG interface and it contains commands, describing which messages we are interested in.
I don’t know where mobileinsight took all proper binary sequences. Perhaps they had been creating many different diag.cfg files using QXDM software and had been analyzing them, because their diag.cfg is a standard one which could be used (as they claim somewhere on their site) by similar software like QXDM or diag_mdlog

3) Some chunk of code (C++/Python) that converts binary packets, received from DIAG, into human-readable format. This piece of software is able to decode only those parts of DIAG packet that are described by (unknown) Qualcomm’s proprietary format. Some DIAG packets contain fields filled with actual network packets in binary form and these fields couldn’t be deciphered by mobileinsight’s routines.
Authors claim that they used Qualcomms’s QCAT software to infer the internal structure of DIAG packets.

4) Mobileinsight also has Wireshark library. It seems that the only HOWTO on the Internet for cross compiling Wireshark for Android has been created by one of the mobilinsight authors…
As far as I understand they use Wireshark as a separate process which is accessed through stdin/stdout. Maybe it had been done for licensing purposes (because if you link your project against wireshark library – your project becomes GPL) – maybe for some other reason.

5) Mobilinsight also has some GUI written on Python which I wasn’t interested in…

Now, How I used this project.
My project actually is an extremely simplified version of mobileinsight.

1) First of all I got rid of multi-process architecture and Python. The former because such architecture, in my opinion, is not good for Android and the latter because I don’t know Python at all. My project is written on Java/C++.

2) diag_revealer has been rewritten as a native library for Java using JNI. It gets commands from and sends data to the main program using JNI .
I also run the cycle, in which data is read from DIAG, as a separate thread so it could be killed without crashing the whole app. The reason why the process of reading from DIAG sometimes needs to be killed lies in the fact that it could accept external command only when there is a data flow from DIAG interface and DIAG works only in blocking mode no matter what. I tried all I could to make it non-blocking but failed.
So, no DIAG packets – no response to commands and no possibility to load new diag.cfg.
The application crash during the DIAG reading often led to a state when DIAG couldn’t be used anymore until complete device reboot.
The killing only a thread, which is working with DIAG, didn’t cause such a problem on my equipment.
After the initialization of DIAG interface my phone always lost the ability to enter a deep sleep state until the next reboot. Maybe DIAG should be switched off somehow, but I couldn’t find such a command though I tried some combinations.
Despite the fact that mobileinsight contains many well-structured code, working with DIAG, the interface itself remains a blackbox which is sad.
I’m afraid that my approach (especially the one with separate thread for reading cycle which continuously detaches from and attaches to JVM) had significant impact on performance, unlike guys from mobileinsight I didn’t do performance tests. Speed and power consumption wasn’t my primary goal on this stage.

2) I started to move piece by piece to my project the part of the mobileinsight’s code which parses DIAG binary packets into human readable format. Mobileinsight uses a mixture of Python/C++ code for this purpose and uses XML to store processed packet. I have gotten rid of Python and XML using Json C++ library. Later, this proved to be a controversial idea, because although json seems to be an appropriate container for Qualcomm’s data in DIAG packets, these packets also contain binary strings which should be further parsed by Wireshark and it produces huge… no, !!!HUGE!!! amount of XML data.
For test purposes I moved to my project only three packet types (mobileinsight handles about 80):
LTE_PHY_Serving_Cell_Measurement_Result
GSM_RR_Cell_Reselection_Meas
WCDMA_CELL_ID
WCDMA_Signaling_Messages

Here is the old demo-video where LTE_PHY_Serving_Cell_Measurement_Result packets in json form are being logged in Textarea.

While adopting this part of mobileinsight’s code I made two sad discoveries…
– Some packets have a number of versions. Different phones (chipsets, firmwares) produce packets of different (incompatible?) versions.
For instance I have Xiaomi Redmi Note 3 Pro, which produces version 19 of DIAG LTE_PHY_Serving_Cell_Measurement_Result subpacket. I also had old Sony Xperia which sent version 22 of these subpackets. And these versions have different structure. Mobileinsight parses both versions, but only these versions. Are there any more? Are there phones which use packet versions 5 or 20? I don’t know. If I had to make and test 22 parsers only for one packet type it would be already a large amount of work but I must find out their structure first. Mobileinsight proposes to use Qualcomm’s QCAT to find out the internal structure and “ground truth” about DIAG packets, but it’s not easy.
Apparently, you should be able to capture these packets in certain format to feed them to QCAT and, apparently, you can examine only the structure of packets that you have captured, not all possible packets.
QCAT, by the way, has many unencrypted database *.db files which, it seems to me, contain information about the structure of various packets. Unfortunately I couldn’t find any tools to see these db files in human-readable form. If you know how to read them – please tell me, it would be very interesting.

– Some DIAG packets contain data which couldn’t be parsed “manually” and require extremely huge and heavy tools for that purpose. Namely – Wireshark.

3) I took a deep breath and managed to compile Wireshark library for android using instructions, provided (as far as I understand) by one of the creators of mobileinsight.
I really admire these guys. I don’t think I could made something similar myself for reasonable time. Not only they managed to compile Wireshark library for Android which is not supported officially – they also used it in a such way that I couldn’t find in docs. The code they had written requires very deep understanding of packet processing by wireshark.
As far as I understand they made this part of their project as a separate thing. Separate process which uses STDIN/STDOUT to get binary packet and return XML result. Maybe this is because of licensing issues maybe not. I wasn’t concerned about licensing so I compiled my project against wireshark library (easy to say, but it took a week or so of headers shuffling).
I slightly changed the source code to use memory buffers instead of STDIN/STDOUT, and included a couple of external source files (perhaps from DragonFlyBSD?) to use fmemopen function which is absent in NDK.

Mobileinsight contains some predefined binary strings to test wireshark functionality. I just did nine test buttons on GUI to run these test cases. I didn’t use wireshark to parse actual data from network.
This is where I stopped.

You can download zip archive containing my project. It’s a Android Studio project so, I suppose, you can just install Android Studio and open this project there. Android Studio will ask you to install additional components.
You also can download a prebuilt APK.

witnettest.zip (120MB)
witnettest.apk (18MB)