Setup

For the purposes of these manual tests, it is currently assumed you are running in a GNOME environment - these tests might be updated in the future to include other desktop environments.

$ sudo apt install flatpak gnome-software-plugin-flatpak
$ flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo
$ flatpak update --appstream

Also check that the version of flatpak is the expected version to be testing.

$ flatpak --version

Reboot your machine, so that GNOME Shell picks up the .desktop files etc (https://github.com/flatpak/flatpak/issues/1256). Also it ensures that GNOME Software service picks up the new plugin and appstream info.

Note that each section of tests are designed to be run in-order.

CLI Tests

This section of tests are mainly performed using a terminal and are testing the basics of managing apps and repositories in flatpak.

1.1 Search for an app

Skip for versions of flatpak below 0.10.1

$ flatpak search recipes

1.2 Install an app

Install the org.gnome.Recipes, if prompted to install a runtime select yes and no errors should be reported.

$ flatpak install flathub org.gnome.Recipes

Check that the app appears in the list of installed items, there should also be at least one runtime auto installed.

$ flatpak list --app
$ flatpak list --runtime

1.3 Run an app

Run the application and visually check that it runs.

$ flatpak run org.gnome.Recipes

1.4 Updating an app

Skip for versions of flatpak below 0.10.1

First we need to downgrade an app.

$ flatpak remote-info --log flathub org.gnome.Recipes

Pick a previous commit from the history (change the commit hash to the one you see in the output of the previous command).

$ sudo flatpak update --commit=460af46218c492ee4a0266222d2191d60f150d1b791234c3789dde6fc1ebbd64 org.gnome.Recipes

Now when update is run we should see the app update.

$ flatpak update

1.5 Remove an app

Remove the app

$ flatpak uninstall org.gnome.Recipes

1.6 Add a repo

Add the KDE repo and check it appears in the remotes list

$ flatpak remote-add --if-not-exists kdeapps --from https://distribute.kde.org/kdeapps.flatpakrepo
$ flatpak remotes

For versions of flatpak below 0.9.1, instead of 'remotes' use $ flatpak remote-list

Update the appstream, check you can search and list apps

$ flatpak update --appstream
$ flatpak search kate
$ flatpak remote-ls kdeapps --app

For versions of flatpak below 0.10.1, skip the search.

1.7 Install an app from second repo

Install the Kate, if prompted to install a runtime select yes and no errors should be reported

$ flatpak install kdeapps org.kde.kate

Check that the app appears in the list of installed items and there should be a kde runtime as well now.

$ flatpak list --app
$ flatpak list --runtime

1.8 Remove app and repo

Remove the app and check it is not in the list of installed apps.

$ flatpak uninstall org.kde.kate
$ flatpak list --app
$ flatpak list --runtime

Remove the KDE repo

$ flatpak remote-delete kdeapps
$ flatpak remotes

For versions of flatpak below 0.9.1, instead of 'remotes' use $ flatpak remote-list

1.9 Test an extra data app

Test installing an app that uses extra data (eg Spotify), ensure that it downloads and launches correctly.

$ flatpak install spotify

1.10 Remove a runtime

Remove the runtime(s) and check it has been removed.

$ flatpak uninstall org.kde.Platform
$ flatpak uninstall org.gnome.Platform
$ flatpak list --app
$ flatpak list --runtime

UI Tests (GNOME)

This section of tests are mainly performed using GNOME Software and GNOME Shell and are testing the basics of a client using libflatpak.

Before running these tests please ensure all previous flatpak packages have been removed.

$ flatpak list --app
$ flatpak list --runtime
$ flatpak uninstall $APPID

This set of tests also assume you don't have GNOME Recipes already installed.

2.1 Search for an app in GNOME Software

Tap the search field
Search for Recipes
Click on the one with "flathub.org" stated as the source
- (If you are using < 3.28 you might get a warning popup https://gitlab.gnome.org/GNOME/gnome-software/issues/242)
Check the the resultant page shows screenshots, description, relevant version info etc

2.2 Install an app in GNOME Software

Click install
The progress bar should animate
After some time the installation should complete successfully

2.3 Run an app from GNOME Shell

Open the activities overview
Search for Recipes
Click on it and visually check that it runs.

2.4 Update an app in GNOME Software

Skip for versions of flatpak below 0.10.1

For this test we need to first downgrade an app.

$ flatpak remote-info --log flathub org.gnome.Recipes

Pick a previous commit from the history (change the commit hash to the one you see in the output of the previous command).

$ sudo flatpak update --commit=460af46218c492ee4a0266222d2191d60f150d1b791234c3789dde6fc1ebbd64 org.gnome.Recipes

Open the "Updates" section of GNOME Software
- If no updates are displayed, click on the refresh at the top left
An update should be shown for <app>
Click update and ensure that it is performed correctly

2.5 Remove an app from GNOME Software

Open the "Installed" section of GNOME Software
Find Recipes in the list with "flathub.org" as the source
Click remove and ensure that it is performed correctly

2.6 Remove a runtime from GNOME Software

Open the "Installed" section of GNOME Software
Find GNOME Platform in the list (under Add-ons at the bottom), if it isn't shown search for GNOME Platform
Click remove and ensure that it is performed correctly

Integration (GNOME)

The section of the tests are mainly performed using GNOME Shell and are testing the integration points to the shell and portals. Note that it is assumed that xdg-desktop-portal and xdg-desktop-portal-gtk/xdg-desktop-portal-kde have been auto installed due to recommends dependency.

3.1 Set an app as default

Install Celluloid flatpak with your preferred method
Open GNOME Settings -> Details -> Default Applications
For Videos, check that you can change the default to "Celluloid"
Open Nautilus, right click on a video file and check that Celluloid is the default option
- This works for at least mp4 and avi

3.2 Open an external file from Nautilus into an app (tests file forwarding)

This test assumes that 3.1 has been performed.

Open Nautilus, right click a video file select "Celluloid"
Check that Celluloid plays the video

For versions of flatpak lower than 0.9.4, file forwarding does not exist so you need to use an app which isn't as confined as Celluloid - eg GIMP.

Install GIMP flatpak
Open Nautilus, right click on a photo select "GIMP"
Check that GIMP opens with the image

3.3 Open an external file from inside an app (tests the file picker)

This test assumes that you have installed GNOME MPV from 3.1.

Open Celluloid
Select the '+' and open a video file
Check that the video is played

3.4 Open an internal file in an external app (tests the app picker)

Ensure you have multiple targets for a PDF installed, eg installed GIMP, Inkscape, Evince flatpak's. Also ensure you have installed org.gnome.Epiphany from flathub.

Open Web (org.gnome.Epiphany)
Search for a test pdf (eg https://github.com/apple/cups/raw/master/examples/testfile.pdf )
Download the PDF
Once this is complete click on the item to open it
Notice the app picker appears (sometimes it appears behind the existing window)
Select an app, such as Evince, check that the PDF is opened

3.5 Desktop icon in shell

This test assumes that you have installed GNOME MPV from 3.1.

Open the activities overview
Search for Celluloid
Check that Celluloid appears with the current icon
Add it to your favourites and check it appears in the sidebar and is launch able

3.6 Install flatpakref file

Skip this test for if you are using less than Ubuntu 18.04 or less than GNOME Software 3.28, as it was broken https://bugs.launchpad.net/ubuntu/+source/gnome-software/+bug/1716409

Skip this test if running GNOME Software 3.30 series less than 3.30.3 (eg Ubuntu 18.10) or try the workaround. Workaround try adding a user remote flatpak remote-add --user --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo . Upstream commit https://gitlab.gnome.org/GNOME/gnome-software/commit/f55a8822acc3cbf26492256d0d4558e319386757

In your browser navigate to https://flathub.org/
Find an app you don't currently have installed, eg Spotify
Click the install button in the browser
Your browser should prompt to open the flatpakref in software installer, select OK
GNOME Software should now open and allow you to install the app, ensure this works successfully

3.7 Install flatpakrepo file

This test assumes that the KDE apps repo has not been configured, if it has then remove the remote first.

In your browser navigate to https://distribute.kde.org/
Click on "kdeapps.flatpakrepo" (or navigate straight to https://distribute.kde.org/kdeapps.flatpakrepo
Your browser should prompt to open the flatpakrepo in software install, select OK (you might need to save target as in some browsers)
GNOME Software should now open and allow you to install the repository.
Switch to the updates pane
Force an update (to ensure that the appstream is up to date)
Now search for a KDE app (eg Kate) and check that there is one which has a source of distribute.kde.org

For versions of GNOME Software below 3.28 or Ubuntu below 18.04, force an appstream update with the CLI (flatpak update --appstream)

(Also one can use the steps in 1.6 to check the remote repo was added via the CLI)