Rewriting Privacy Manager

First version of the Privacy Manager was created mid 2012 and since then the codebase didn't have much of drastic updates. In 7 years web technologies has been advanced quite a lot and maintaining an old monolithic code was quite a challenge:

• No modularization.
• No tests running in the browser.
• Callbacks hell.
• and more..

Even though it took 24 hours to create Privacy Manager for a Hackathon and 1 week, to modify it with the help of Google Chrome team in Munich, the complete rewrite took me around 4-5 months. So let's have a look into those changes.

Web Components

Initial version of Privacy Manager had a monolith implementation and as it was designed as part of a Hackathon project, my goal was to just make it work, as a result I have got a quite messy interconnected widgets, which I was trying to refactor from time to time. The GIF below describes quite well Privacy Manager's development experience:

In the mid 2017 decision was made to get rid of Jquery and Jquery UI dependencies from the Privacy Manager, because most of the things I needed Jquery for, was already simpler doing in vanila JavaScript. That change saved me from the need of keeping Jquery up to date and made me realize again how advanced has JavaScript become, but yet each time there was a request from a community to add a new feature I kept realizing that in order to keep project maintainable I need to find another solution to the old monolith architecture. After getting rid of Jquery UI plan was to decide on a framework that would help me develop components in isolation, but I always was occupied with other projects.

Early 2019 I switched to working part-time and decided to provide rest of my time to my personal projects. Mid 2019 I have started closely looking into Web Components which seem to be a great fit for Privacy Manager and a good enough reason for experimenting with the technology, especially after successfull adaptation of Web Components in the company where I'm working.

It actually took me longer than I have expected to migrate my old-fashioned widgets into Web Component, but now they seem to work pretty well and most importantly be in complete isolzation. They definatelly has a lot of room for improvement, but in comparison to the previous version of Privacy Manager, the code looks finally maintainable :) Below you can find Web Components, used in Privacy Manager.

pm-toggle

pm-toggle is a basic toggle switch which comes with text, description and actual toggle.

You can find pm-toggle here and source code for pm-toggle here.

pm-table

Lazy loaded sortable table which is used in both Cookies and Network tabs of the Privacy Manager.

You can find pm-table here and source code for pm-table here.

pm-dialog

Modal dialog which is used as informational, confirmation and cookie editing dialog.

You can find pm-dialog here and source code for pm-dialog here.

pm-tab-panel

Tab panel which is used for switching between various sections in the Privacy Manager.

You can find pm-tab-panel here and source code for pm-tab-panel here.

pm-button

Various buttons used in the Privacy Manager.

You can find pm-button here and source code for pm-button here.

Automated tests and puppeteer

Until lately only test that have been implemented in Privacy Manager was some linter checks, to ensure consistency of the code and styles. Lack of test always used to strike back in the form of slow development speed and stress when addressing changes.

When the development of the Web Components for Privacy Manager has started I was hopping that it would be possible to use jsdom or a similar alternative to implement tests, but unfortunately jsdom has problem with Web Components support. After spending way too much time looking into alternatives, the simplest solution for my use case appears to be runnig actual Chrome browser in order to minimize all issues related to Web Components support. Even though decision of running actual browser, in order to test simple components felt in the begining as an overkill, but it has saved a lot of headache and worked apparently pretty well.

puppeteer for web components

Puppeteer is a powerful Node.js library that allows running and controling Chromium browser by writing Node.js code. Puppeteer can run both headless and headfull mode.

Running test in the browser has notable benefit over using other implementation of web standards like jsdom, simply because tests are being run in much closer to the production environment, in case of the Privacy Manager production environment actually is the Chrome browser. Also Browser vendors and especially Chrome and Firefox team are one of the first to have the standarts implementation ready in the browsers, so if something works in the puppeteer than it's much more likely to work in the Chrome browser as well, but surelly it's a beast in comparison to jsdom because you are running actual browser.

Runnig browser using puppeteer is as simple as running the code below:

const browser = await puppeteer.launch({headless: true}); // Launching browser
const page = await browser.newPage();  // Opening a new tab
// Specifying User-agent.
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3419.0 Safari/537.36');
// Loading a page into the open tab
await page.goto(`https://google.com`);

The code above creates a headless browser - a browser without a graphical user interface, if one needs to run the browser with the graphical interface, it's can be done by using await puppeteer.launch({headless: false});.

Puppeteer has a quite extensive API, but most importantly it allows executing script in the page context and also return a value or a representation of the DOM object(JSHandle) that can be used in execution method which eventually will reffer to a DOM object.

const pmToggle = await page.evaluateHandle(() => document.querySelector("pm-toggle"));
const result = await page.evaluate((pmToggleElement) =>
{
  return pmToggleElement.isEnabled();
}, pmToggle);
console.log(result); // true | false

This makes puppeteer quite flexible and basically that's all I needed for testing web components, connecting that with assertion API and travis CI was last piece in my Continues Integration workflow.

Puppeteer tests implementation for the pm-components can be found here, implementation would need a refactoring, but currently it serve the purpose pretty well and it takes in average 3s to run all the tests, which is quite great considering that they are being run in actual browser.

puppeteer for web extensions

Even though puppeteer served it's purpose for web compoents, unfortunatelly it wasn't as smooth when it came to the actual integration tests for the web extension(Privacy Manager).

There are some limmitations when it comes to testing actual browser extension, some of the limmitations have workaround, but for others it doesn't seem to be much of workarounds available.

First limmitation is that it's not possible to load extensions in the headless mode, in order to load web extension headfull mode should be used instead. To load the browser extension, we are going to use --load-extension and --disable-extensions-except arguments, first we need to specify path of the extension we would like to use and then disable all other extensions.

const extensionPath = "dist";
const browser = await puppeteer.launch({
    headless: false, // extension are allowed only in the head-full mode
    args: [
        `--disable-extensions-except=${extensionPath}`,
        `--load-extension=${extensionPath}`
    ]
});

After loading the extension, we need to find that extension's ID. That can be done by looping through all available targets, where you can find actual background page of the extension and hence also extension ID.

const targets = await browser.targets();
const backgroundPage = targets.find(({ _targetInfo }) =>
{
    return _targetInfo.title === "Privacy Manager";
});
const [,, extensionID] = backgroundPage._targetInfo.url.split('/');

Main UI of the Privacy Manager is located in the extension's popup window, which in this case is located in popup.html.

await page.goto(`chrome-extension://${extensionID}/popup.html`);

Now we can run the code and test the functionality of the popup page.

Privacy Manager is using optional permissions, in order to use cookies tab and/or monitor the network traffic the extesnion needs to ask the user's consent to access that data, but the problem is that there seem to be no way to access browser's native permission consent dialog using puppeteer to confirm or deny the request, which is quite sad and it prevents the extension from being fully tested. An issue was created about requesting and removing optional permissions, but unfortunatelly no answer there for 2 months already and I couldn't find a better solution than modifying the manifest.json in the test environment to bypass the permission wall, which make it possible to test most of the functionality, but leaves some of the edge cases and permission related tests untouched for now.

As a result, now most of the important and tidious integration tests are ready and running automatically, with the exception of several edge cases that wasn't possible to test because of the limmitations and now it feels so annoying to realize that I used to test all that manually. Test that used to take sometimes even 20-30minutes of my time, now are being run in just 9s and automatically.

Puppeteer tests implementation for the privacy-manager are located here and builds hopefully are passing :)

.

webextension-polyfill

Privacy Manager's code was a real callbacks hell and finally I've got my hands on fixing that. Plan was to wrap Chrome's callback's based API to one that will use promises instead and eventually make the code compatible with Firefox's WebExtensions API, for the future plan of making Privacy Manager being available in Firefox as well. That looked to be a lot of work, but doing a little reseach I have stumbled upon mozilla/webextension-polyfill repository, which were basically doing what I was looking for. The problem was that the project were not supporting all Chrome Privacy APIs, but only those which were supported by Firefox, thanksfully mozilla development team was very supportive after the issue about missing chrome privacy APIs was reported to them and they helped me fixing that issue in a spearate PR.

Currently Privacy Manager is using promises and it's now more closer into being ported into Firefox, than ever before. There are still some work to do to make that possible, but it's now more of a documentation work and tests automation.

Update: Unfortunatelly there is still a bug in Firefox which prevents requesting Additional permissions from the extension popup and hence blocking Privacy Manager port to Firefox.

Privacy Managers organization

After moving Web Components into a separate repository, it already made sense to move Privacy Manager related repositories into a new Github organization. Now the new organization is ready and both, Privacy Manager Components and Privacy Manager are moved into the Privacy Managers organization. The reason why the organization name is plural, is that the plan is to host Privacy Manager's landing page at privacymanagers.org(website is in the designing phase) and hopefully also host there various resources about Privacy and Security.

Get Privacy Manager here.