Share

engineering

13min read

A Curious Relationship: Android BLE and Location

A Curious Relationship: Android BLE and Location

Introduction

While developing mobile applications I sometimes hear complaints such as: “Why this app asks for location permissions when I just want to connect my BLE device?”

Instinctively, it is just not right—you probably feel the same.

I wanted to understand the origin of this requirement and wondered if anything can be done about it. In the end, it proved to be an interesting journey. In this blog post, I will describe why access to the location is mandatory on Android to work with BLE. Then, I will explain what we have tried to do in our attempts to reassure all privacy-concerned users. By the end, you will know how to work around the titular restriction under typical situations.

Why on Android Location is used for BLE?!

I wish the application would not need to ask for location access. Unfortunately, it is necessary since BLE functionality may be used to locate the user. How the app may locate the user? Let’s examine what most of the apps that connect to BLE peripheral do.

First, the user installs a new application that is dedicated to work with their shiny new peripheral. Unfortunately, the app cannot connect to it right off the bat because it does not know the unique identifier that is needed to establish a connection. To get the information about this identifier, the application needs to first issue a scan (in Classic Bluetooth nomenclature the process was named discovery but it may now be confused with services discovery in BLE) and the system should callback the app whenever it will scan a new BLE advertisement of each nearby device.

ScanCallback onScanResult will be called with a ScanResult which in turn contains a reference to a BluetoothDevice representing the peripheral and a ScanRecord reference that contains advertisement data. This information is enough to connect to the peripheral the user is interested in. Unfortunately, with this information, there are at least two ways in which one may try to locate the user:

  1. There is a number of BLE beacon protocols that may be used to locate the user via information transmitted in the ScanRecord’s data.
  2. Each BluetoothDevice has its own unique identifier called the MAC address. It is possible to locate the user given the knowledge of BLE peripherals locations and MAC addresses.

Alright. So this is why the Android BLE scanning API usage needs access to the location from the app.

What I find confusing and what makes the experience from the user’s standpoint worse—the Android OS has its location access divided between two switches:

  1. Location Permissions—needed for the application to access anything that is location related.
  2. Location Services—a more temporary location tracking switch which is often referenced as the “GPS switch”.

image5

I wondered—what exactly is needed for the application to successfully scan a peripheral? Let’s look into the docs.

The Android BluetoothLeScanner’s API documentation of startScan(List<ScanFilters>, ScanSettings, ScanCallback) method states:

An app must hold ACCESS_COARSE_LOCATION or ACCESS_FINE_LOCATION permission in order to get results.

It turns out to be not entirely true. I have tried to call the method without permissions but it failed with an error code that says: application failed to register the scan. The above description would better match the case when Location Services are turned off.

The role-model phones for Android OS (e.g. Pixels, Motorola) tend not to call the listener with scan results while other vendors/models seem not to care at all and return the results happily even when the Location Services are off.

As usual, it is all about finding the right balance between usability and privacy. At the moment, different Android OS implementations are highly inconsistent.

Why iOS is different?

I work mostly on Android applications but I look on iOS with envy because it is designed with privacy in mind. Apple decided to strip unique device information from the API—things like MAC address, some characteristics and data types that may identify a peripheral are inaccessible. Additionally, each iOS device generates its own unique identifier (UUID) for every Bluetooth peripheral it scans—these generated UUIDs will be different on another iOS device and therefore cannot be trackable. Beacon advertisements are also unavailable among BLE scan results—there is a separate API for them which needs access to the location as well.

All this sums up to a system that gives a good user experience without sacrificing its privacy.

image7

Next connections

I understand that the scanning procedure may be used to locate the user so the location access is justified. But then I thought: “Isn’t it possible, given the peripheral’s MAC address, to connect without scanning? Android API has a possibility to retrieve an instance of BluetoothDevice given its MAC address after all”.

That is true. Unfortunately, Android API also has a horrible flaw—it is exposing a 48 bit MAC address without information about what type of address is that (more info about addresses may be found in Core Bluetooth Spec. v4.2 [Vol 6, Part B] 1.3).

You read it right. The system made by mighty Google has some substantial shortcomings which are totally undocumented.

To successfully connect to a BLE peripheral both the MAC address and its type are required. But how Android OS solves this problem? It connects after all.

In case the peripheral was scanned its address type seems to be cached by the lower layers of BLE stack. However, if cache has no information about a particular MAC address, the system uses internal heuristics to determine the kind of address it is working with—if the guess is correct then a successful connection will be established without scanning—if not one may expect the obscure status=133 (GATT_ERROR) being returned even if the peripheral is advertising right beside the device.

I made a test by connecting to a device with a known MAC address. From my experience, Nordic Semiconductor nRF5 chips in default configuration are usually guessed badly. Those chips are using random addresses out of the box. Most probably it is because public address space must be bought from Bluetooth Signature Group. As nRF chips are meant to be programmed and used by third parties, it is not in their business to pay for the public space.

Therefore, Android caches the address type internally after the first scan. Unfortunately (again) caches may get cleared at some point which leaves us with the nasty bug I have just described.

I asked myself: “Is it possible to monitor the cache validity?”. New tests began. Knowing that nRF devices don’t connect without their address type cached, I went with checking when the connections will start to fail.

It quickly turned out that subsequent connections on most devices will not be successful if the Bluetooth will be power cycled. Curiously enough, on some, it would still work until a complete switch off was made.

I wondered if it would be possible to track the moment when a new scan has to be made to make the connection possible. Two doubts came to my mind:

  1. Newer versions of the OS introduce background process limitations

    1. A process cannot run without a foreground notification
    2. Bluetooth adapter power state broadcast cannot be registered in the AndroidManifest and therefore wake the app
  2. It is not possible to determine when the cache gets cleared without experimentation

As the application process may get killed at random moments in the background, it cannot reliably track whether a scan is needed before the connection—the only safe way to reliably connect is to always scan after the application start. Yet another inconvenience.

Challenging status quo

Are we doomed to always ask for the user access to their location to work with BLE peripherals? Is there no other way?

Some time ago, I was asked this question by one of our clients and was about to struggle against the system. I already knew that the autoConnect flag, that is passed when establishing the connection, changes something more than just the amount of time needed to connect so I decided to play with it a bit.

I found out that when using autoConnect=true we were able to connect to our target device reliably, though it needed a longer time. The first suspicion was that, in this situation, the MAC address type resolution may be deferred to the time of the first sight of the peripheral.

Bluetooth Low Energy is all about preserving energy. The devices are trying to sleep as long as possible and stay awake only for a short period of time after advertising—this is the time a phone may connect to them. So every connection to a peripheral always starts with a Bluetooth scan. When using autoConnect=true, a duty cycle of scanning is quite relaxed. I ran some tests when doing my first steps in BLE field and found that it could take up to several minutes for a connection to be established in this case. It should be possible to force a high duty scan by other means though.

image6

How to scan without access to Location?

No one can use BLE scan API without having Location Permissions, and, on most devices, it is useless without active Location Services. However, Bluetooth Low Energy is an extension to so-called Bluetooth Classic and interestingly enough Google decided this API does not need location permissions to be called.

Different Bluetooth versions are mostly backward compatible. There is a good chance that both Classic and LE scans are actually totally the same under the hood if a handset supports both. The only difference is that the caller will get informed in different ways. However, the Classic version of the scan will not return any BLE advertisement packet data. There will be no possibility to detect if your BLE device is advertising unless you know its MAC address. This approach only works if applied for subsequent connections to an already known peripheral.

It quickly turned out that the BluetoothLeScanner javadoc annotation about permissions is a better fit for the classic API. It actually will not return any results until the application will have location permissions but the scan seems to be working correctly (checked by subscribing to BluetoothAdapter.ACTION_DISCOVERY_STARTED and BluetoothAdapter.ACTION_DISCOVERY_FINISHED).

image8

Testing BLE without Location Permissions

Is it possible to connect reliably to a peripheral knowing its MAC address only? I wanted an answer to a range of different handsets. Knowing information from previous paragraphs, I wrote an algorithm shown below in pseudocode (runnable code is available here):

race (
  const connected = autoconnect(deviceId)
  const discovered = classicBluetoothDiscovery().filter(deviceId)
  const timeout = timeoutAfter(30 seconds)
)

if (connected) => return

cancelIndirectConnection(deviceId)
cancelClassicBluetoothDiscovery()
connect(deviceId)

Before each test, the Bluetooth cache was cleared. At first, three things happen:

  1. An indirect connection is started (autoConnect=true)
  2. A Bluetooth classic scan is started
  3. A timeout procedure is started

If the connection will happen first, the tests end. On some mobiles, the Classic scan will return results even if location access is not granted. If that is the case and the target peripheral is found, the race terminates and the app can proceed to direct connection. Timeout also finishes the race after which a direct connection attempt is made (which should be quicker). The test measures the time needed to get a valid connection.

The results:

phone version

OS

address type cache clear moment

average workaround connection time (s)

Nexus 5X

8.0.0

adapter cycle

21

Huawei P8 Lite

6.0

adapter cycle

5

OnePlus 3T

8.0.0

power cycle

12

SGS8

8.0.0

power cycle

1

Asus Zenfone AR

7.0

power cycle

13

HTC U11

8.0.0

power cycle

1

LG G6

7.0

adapter cycle

7

Moto X 2nd Gen

6.0

adapter cycle

(~90% success rate) 35

Nexus 6

7.1.1

adapter cycle

2

The workaround behaved quite nicely and I was able to get connections without location access. There is, however, one device (Moto X 2nd Gen) which always timed out and only the second, direct connection succeeds.

I had confirmed that the Motorola phone does not return any Classic Bluetooth scan results in the current setup. Additionally, none of the successes happened before the timeout. The conclusion was that moment of address resolution takes place at the time of the call, at least for this handset.

Anyway, the amount of time needed for connecting was unacceptable given that the test peripheral was constantly advertising right beside the phone.

Even though our approach was not an ultimate success, I found it to be promising and decided to dig deeper.

New findings

Next step I took was to investigate how the address type resolution is really made. To do so, a look into the implementation of Android BLE stack was needed. I confirmed that the system does keep a cache of resolved MAC address types. Additionally, if the address is not cached, a “guess” is made; on all Android versions in direct connection mode (autoConnect=false), the type is assumed to be public. In indirect mode the situation is a bit different—due to another bug (which by the way is hilarious due to an apparent misunderstanding of Core Bluetooth Spec. v4.2 [Vol 6, Part B] 1.3.2); on Android Nougat and Oreo the type would default to random. On other OS versions, it would still be treated as public.

Getting back to the previous table it now had a lot more sense. The address type resolution is always made at the time of the call.

The workaround did work for me due to a few reasons:

  1. Undocumented difference related to autoConnect flag and the type resolution.
  2. Some phones being not consequent with returning results when there is no access to location.
  3. Bluetooth Classic scan also resolving BLE peripherals address type.

With this knowledge, it should be possible to always connect to any peripheral if its address type is known to the developer at least on Android 7 and 8. Devices which are returning scan results without location access would work as well.

image3

The missing piece

Recently, I have learned that there indeed is a way to check whether a particular peripheral is already in the phone’s address type cache. BluetoothDevice has a function getType() which returns if the peripheral is a ‘BLE’, ‘classic’ or ‘dual-mode’ device. It may return a fourth option though—unknown. It turns out that unknown is returned either when Bluetooth is currently switched off or the device was not yet put into the cache. Bingo!

I need to mention that the above behavior was determined by reading Android AOSP implementation. It is not described in the documentation, therefore not guaranteed and may change in the future.

Conclusion

With this new information, it is possible to eliminate the address type resolution bug! The connection, no matter if indirect or direct mode, has to be requested only after the peripheral is put into the cache. Combining this with the ability to run a scan without the need of location access we have an open path to write Android BLE apps that need the location access only for the initial connection as long as advertisement data is not relevant afterwards. How cool is that?! Especially while still keeping user’s location private!

To do that efficiently, especially in the background, is a totally different topic. Do you have questions? Want to point out a mistake or discuss? Feel free to contact us — I will try to answer as soon as possible.

Share

Darek

Staff Software Engineer

Did you enjoy the read?

If you have any questions, don’t hesitate to ask!

Did you enjoy the read?

If you have any questions, don’t hesitate to ask!