Mobile Forensics: Examining DuckDuckGo on Android

Overview

  Image 1: DuckDuckGo Application on Android (Nox)

When people use DuckDuckGo, they generally associate it with enhanced privacy while browsing the Internet. With a flame icon (Image 1) which allows users to "Clear All Tabs and Data", users may inherently believe their browsing data is wiped the moment they see the flame animation clear their currently open tabs and browser history. From a forensics perspective, we should be asking: How well does DuckDuckGo clear user data which could be useful in a mobile forensics investigation?

By examining the DuckDuckGo application and its underlying data on an emulated Android device, we can determine that DuckDuckGo can provide a variety of useful forensic artifacts to an examiner. In balancing user experience with privacy, forensic examiners can recover artifacts such as bookmarks, application summary data, and residual user data even after being "cleared".
To reach this determination, we can use the Nox Player Android emulator and a variety of free tools to explore how DuckDuckGo stores, persists, and clears user data. Special thanks to Jessica Hyde (@B1N2H3X) for her guidance and advice over the course of this project and to Alexis Brignoni (@AlexisBrignoni) for his article on Android emulation[1] which served as my foundation for testing DuckDuckGo on an emulated Android device.

Tools Utilized

- Nox Player (Version 6.2.8.0015)
- Android (Version 5.1.1 - SM-G955N)
- Android Debug Bridge (Version 1.0.36)
- DuckDuckGo (Version 5.21.2 mk - 52102)
- Notepad++ (Version 7.6.6)
- HxD Hex Editor (Version 2.2.1.0)
- DB Browser for SQLite (Version 3.11.2)
- Windows PowerShell (Version 5.1.17134.590)

Testing Methodology

In order to provide a more holistic picture of what data can be recovered from DuckDuckGo, we will periodically retrieve the underlying storage under different conditions to determine what data is stored and if/when the data is cleared. These conditions include:
- Application installed but not used[2]
- Application used but not cleared
- Application used but cleared
- Application used, cleared, used after being cleared

In order to ensure a majority of the caches and databases are populated, you can browse to sites such as 'jigsaw.w3.org/HTTP/' which provides sample HTTP basic authentication and 'rsolomakhin.github.io/autofill/' which provides autofill forms. Other sites such as 'www.apple.com', 'www.google.com', 'www.amazon.com', or the website for your restaurant can be visited as they provide various web browsing artifacts such as cookies and images which are persisted in the caches and additional databases.

Analysis of DuckDuckGo's Underlying Storage

To retrieve DuckDuckGo data from the Nox Player, we can utilize the Android Debug Bridge to connect to the emulated device. To connect to the device we can run `adb.exe connect 127.0.0.1:62001` followed by `adb.exe devices` to ensure the connection was successful. Once we have established a connection to the device[3], we can open a command shell on the device `adb.exe -s 127.0.0.1:62001 shell` and run use the find command to search for files with duck in their file path `find / -type f | grep -i duck`.

In my testing, the majority of the user data for DuckDuckGo was stored in:
'/data/data/com.duckduckgo.mobile.android'

Within the above directory, DuckDuckGo data is broken up into 5 subdirectories, each with a specific purpose: 
- databases: DuckDuckGo application databases (app.db. http_auth.db, androidx.work.workdb)
- cache: duckduckgo.com cache 
- app_webview: web browsing cache and databases 
- shared_prefs: DuckDuckGo application configuration files 
- files: DuckDuckGo application support files (trackers whitelist, helper scripts, etc.)

To retrieve the data from the device, we can exit the shell and pull back each of the individual subdirectories:
- `adb.exe -s 127.0.0.1:62001 pull /data/data/com.duckduckgo.mobile.android/databases` 
- `adb.exe -s 127.0.0.1:62001 pull /data/data/com.duckduckgo.mobile.android/cache` 
- `adb.exe -s 127.0.0.1:62001 pull /data/data/com.duckduckgo.mobile.android/app_webview` 
- `adb.exe -s 127.0.0.1:62001 pull /data/data/com.duckduckgo.mobile.android/shared_prefs` 
- `adb.exe -s 127.0.0.1:62001 pull /data/data/com.duckduckgo.mobile.android/files`

Artifacts of Interest

Having examined the data pulled back from the device, I found the following artifacts to be of potential forensic interest:

1. 'databases/app.db' - SQLite3 database containing tables for artifacts such as bookmarks (Image 2), tabs (Image 3 and Image 4), sites visited (Image 5), and app usage (Image 6). To determine the currently open tab, you match the tabId from the tabs table to the tab_selection table.
 
Image 2: 'bookmarks' table in app.db 

 
Image 3: 'tabs' table in app.db


Image 4: 'tab_selection' table in app.db

 
Image 5: 'site_visited' table in app.db 


Image 6: 'app_days_used' table in app.db

2. 'databases/http_auth.db' - SQLite3 database containing a table for authentication information for successful HTTP authentication attempts (Image 7).


Image 7: 'httpauth' table in http_auth.db

3. 'cache/*' - The primary cache for duckduckgo.com interaction. Each entry in the duckduckgo.com cache is named in the following format '{MD5 Hash of URL}.{File Type}' where the file type refers to whether the file is the HTTP request ('0') or the HTTP content ('1') returned (Example 1). When examining this cache, in addition to DuckDuckGo application files like the one below, there are also cached autocomplete recommendations for queries being typed in the search bar by the user (Example 2).
 
Example 1: File name structure for cache/*
    File Name: 0ef3124400f18b4ac08607674469f816.0
    File Extension: 0 (HTTP Request)
    URL: https://staticcdn.duckduckgo.com/https/https-mobile-bloom-spec.json?cache_version=1
    ----
    File Name: 0ef3124400f18b4ac08607674469f816.1
    File Extension: 1 (HTTP Content)
    URL: https://staticcdn.duckduckgo.com/https/https-mobile-bloom-spec.json?cache_version=1
 
Example 2: HTTP request and corresponding content
   File: 73cdcf52a64bae199d0a01d76a5e46d9.0
    GET: https://duckduckgo.com/ac/?q=the%20good%20compan
    Version: HTTP/1.1
    Status Code: 200
    Content Type: content-type: application/javascript; charset=UTF-8
    Content Encoding: content-encoding: gzip
    ----
    File: 73cdcf52a64bae199d0a01d76a5e46d9.1
    Content: [{"phrase":"the good company"},
    {"phrase":"the good companions"},
    {"phrase":"the good company austin tx"},
    {"phrase":"the good company clothing store"},
    {"phrase":"the good company ton wi"},
    {"phrase":"the good company clothing"},
    {"phrase":"the good company restaurant houston"},
    {"phrase":"the good company store"},
    {"phrase":"the good company nyc"},
    {"phrase":"the good company restaurant"}]

4. 'app_webview/Cache/*' - The cache for all other sites interacted with through the DuckDuckGo app. This cache does not use the same structure as the cache described previously. Instead, each file in the cache contains a combination of strings and raw data which can be parsed. By examining the overall structure of these files, it is possible to define the format of the data which can be parsed (Image 8). 

For cached files in the app_webview cache, starting at offset 12, the following 4 bytes refer to the length of the URL which in Image 8 is 88 ('\x58\x00\x00\x00'). The URL always starts at offset 20 and has a length which was previously determined. To determine the size of the content, you can search for the following in the cached file '\x00\x00\x48\x54\x54\x50' which is the equivalent of 2 NULL characters followed by HTTP in ASCII and serves as the start of the HTTP Header. By shifting the offset back 32 bytes from the start of the HTTP header ('\x48\x54\x54\x50'), the following 4 bytes refer to the size of the HTTP content which in Image 8 is 119 ('\x77\x00\x00\x00'). Starting at the first byte immediately after the URL and by reading the length of the HTTP content previously determined, we can extract the HTTP content to its own file for analysis. By reading the HTTP header located previously until you hit the next 2 '\x00' (NULL) characters and breaking the bytes read at each NULL character, we can create a list of the items in the HTTP header. In the case of Image 8, the first 4 bytes of content match the header of a PNG file ('\x89\x50\x4E\x47) and the last 4 bytes match the CRC32 of a PNG file ('\xAE\x42\x60\x82'). This is expected as both the URL and the HTTP header lead us to believe the HTTP content is a PNG file.


Image 8: app_webview/Cache/{cache_file} in HxD

5. 'app_webview/Cookies' - SQLite3 database containing cookies from web browsing (Image 9).
Image 9: 'cookies' table in Cookies

6. 'app_webview/Web Data' - SQLite3 database containing autofill data from web browsing (Image 10)

Image 10: 'autofill' table in Web Data

7. 'shared_prefs/*.xml' - XML formatted files which contain summary data about the DuckDuckGo application to include the installation time (Image 11) and the last cleared time (Image 12). As most timestamps are stored in seconds since epoch, it is worth noting the timestamps below are stored in milliseconds since epoch.


Image 11: com.duckduckgo.app.install.settings.xml


Image 12: com.duckduckgo.app.fire.unsentpixels.settings.xml

Data Available to Forensic Examiners

Using the previously discussed testing methodology on the artifacts identified, we can determine what data survives DuckDuckGo's "Clear All Tabs and Data" feature. Based on my analysis, DuckDuckGo does a good job of clearing the cached data as well as most of the user data stored in SQLite databases when a users clears the application data. That being said, tables such as 'site_visited', 'app_days_used', and 'bookmarks' which were shown previously survive the clearing process.

If data has not been cleared, all of the artifacts discussed can be extracted with varying levels of difficulty. For example, all SQLite3 databases and XML files are easily parsed with an assortment of free tools (DB Browser for SQLite, Notepad++, etc.) whereas the primary web cache 'app_webview/Cache/*' requires carving multiple items of interest from each cache file (as previously demonstrated). 

Python Parser

To aid in the analysis of DuckDuckGo data, I wrote a parser in Python 3 which provides summary data about the application and parses items of potential forensic interest. This has been manually tested and verified with HxD and DB Browser for SQLite. Currently, the parser does not process all tables and files (only the ones described above[4]) which means additional investigation of DuckDuckGo data may be required to obtain a full picture of application usage. The parser can be found on GitHub at: https://github.com/ItWasDNS/DDG-Parser

To run the DuckDuckGo Python Parser, run the below command and follow the prompt:
`python3 process_duckduck.py`

Note: User will be prompted for DuckDuckGo application data directory. This directory does not need to be com.duckduckgo.mobile.android but should contain DuckDuckGo application data.

Footnotes

[1] Viewing extracted Android app data using an emulator
[2] The DuckDuckGo data directory was not populated until after the app was opened for the first time
[3] If multiple devices are present, we can use the -s option to specify the connection to the emulated device previously established
[4] All artifacts are parsed but there may be other items within the artifact which can be parsed to a greater extent