Debugging
Revision 2 as of 2015-06-05 17:34:03
Clear message
From a private email thread from jdstrand:
...
Start of ideas for tutorial on developing a snap:
What we don't have is a tutorial to tie everything together for the first time
developer so they don't have to read through all of the above to do meaningful
work. We have something along these lines on Touch:
https://developer.ubuntu.com/en/publish/security-policy-groups/
https://developer.ubuntu.com/en/start/platform/guides/app-confinement/
I wrote 'https://developer.ubuntu.com/en/start/platform/guides/app-confinement/'
for Touch and 'https://developer.ubuntu.com/en/snappy/guides/security-policy/'
(ie, docs/security.md) is roughly equivalent to that, but the Community team has
a tech writer that did the other tutorial and AIUI, quite a few others.
A tutorial that went step by step through each of python, Go, nodejs and
whatever else would be great. If I were writing this myself, I would:
1. write a small cli shell script that tries to write to the install directory,
hardcoding paths
2. package the simple cli binary using 'caps: []'
3. install it and show how to run it
4. observe that the script errored out because it couldn't write to the install
area due to apparmor (FHS violation), and how to check for security policy
issues
5. show how to update the script to use the SNAP_* environment variables
(referencing hello-world.env) and fix it to use SNAP_APP_DATA_PATH
6. remove then install the snap
7. run it and show it works now
8. add a simple httpd python service to the existing snap using 'caps: []'
9. install it and show how the systemd service is created in
/etc/systemd/system
10. debug why the httpd service didn't start since it is missing the
'network-service' cap
11. update the snap to use 'network-service' cap for the httpd service
12. remove then install the new snap
13. show various systemctl commands to show it is running
14. update the the httpd service to display what the small cli shell script
writes out, but with a bug in it (eg, an import error)
15. remove and then install the snap
16. debug the problem with systemctl, lots, etc and fix the bug in the snap
17. remove then install the snap
18. point a browser at the http service
19. run the cli command
20. refresh the browser to show that the cli command updated the page
21. show how to upload to the store
As you can see, I've given this some thought. This kinda ties everything
together so people can get things done. It provides some best practices (use the
SNAP_ variables), teaches a bit about the FHS, a little about debugging, how
services and binaries work, a bit on the security sandbox and how to iterate on
one's snap. In each step we can link to the appropriate guide for more
information. I certainly wouldn't mind writing it, though time being what it is,
I haven't yet. We do have a tech writer for this and I'd be happy to review
something that he writes. Something similar for Go, nodejs, whatever would also
be helpful.This is just a dump of an email thread that could form the basis of a debugging tutorial. From snappy-devel@:
> When I "start.freedomotic" this is the output:
>
> (RaspberryPi2)ubuntu@localhost:~$ start.freedomotic
> aa-exec: ERROR: profile 'freedomotic_start_5.6.0' does not exist
> The file freedomotic.apparmor is present. Infact if I launch "start" in
> bin folder with sudo it works.
> After a random time the app is killed.
>
'start.freedomotic' is not the correct name for the file on either the stable or
the rolling snappy releases-- you should be using 'freedomotic.start'. If
'start.freedomotic' is working for you it indicates either you have some old
files laying around or your system is out of date. Also, running under sudo from
your app's install directory will certainly work-- you are bypassing systemd and
the app launcher (it is useful to know that it works on its own though).
I downloaded the snap from dropbox:
$ click-review /tmp/freedomotic_5.6.0_armhf.snap
/tmp/freedomotic_5.6.0_armhf.snap: pass
and installed on my beaglebone.
$ sudo snappy install --allow-unauthenticated /tmp/freedomotic_5.6.0_armhf.snap
Installing /tmp/freedomotic_5.6.0_armhf.snap
2015/05/11 21:29:36 Signature check failed, but installing anyway as requested
Name Date Version Developer
ubuntu-core 2015-05-08 55 ubuntu
docker 2015-05-08 1.6.1.002
freedomotic 2015-05-11 5.6.0 sideload
beagleblack 2015-05-08 1.7.1
$ start.freedomotic # <----- your command which correctly doesn't work
-bash: start.freedomotic: command not found
$ freedomotic.start # <----- what should work based on your comments,
# but doesn't
-bash: freedomotic.start: command not found
In looking at your snappy packaging, you are not specifying any 'binaries', only
a service so you won't get a binary you can use from the command line-- only a
systemd service.
Looking at your package.yaml, there are several issues:
$ cat ./package.yaml
name: freedomotic
architecture: armhf
version: 5.6.0
vendor: Freedomotic Team <info@freedomotic.com>
icon: meta/freedomotic.svg
services:
- name: freedomotic
description: "freedomotic runtime"
start: bin/start
integration:
freedomotic:
apparmor: meta/freedomotic.apparmor
First, as mentioned, you don't have a 'binaries' entry[1]. Second, you are using
the obsoleted 'integration' hook for specifying security policy[2] (the review
tools should've caught this, and I've added a todo to fix this). Looking at
freedomotic.apparmor, you aren't doing anything out of the ordinary, so I
suggest you change your yaml to simply:
name: freedomotic
architecture: armhf
version: 5.6.0
vendor: Freedomotic Team <info@freedomotic.com>
icon: meta/freedomotic.svg
services:
- name: freedomotic
description: "freedomotic runtime"
start: bin/start
binaries:
- name: start
exec: bin/start
description: "freedomotic runtime cli"
With the above you will get a systemd service (that calls 'bin/start') *and* a
CLI binary in /apps/bin/freedomotic.start (which also happens to call
'bin/start'-- I'm not sure you actually want to do that in your package, but it
illustrates the point I'm trying to make) and you don't have to do anything
special for security.
After making the above change, make sure your target system is up to date. On my
beaglebone black I have:
$ system-image-cli -i
current build number: 55
device name: generic_armhf
channel: ubuntu-core/15.04/edge
last update: 2015-05-08 08:10:00
version version: 55
version ubuntu: 20150508
version raw-device: 20150508
If you are on the stable channel, you will have a different build number (I'm
not sure about rasp pi2). If I were you, I'd reflash to stable and make sure you
have a clean environment. Once you've done that, install the new package with
the packaging changes I suggested.
If you are iterating and installing the same version over and over again, you
will want to do on your target:
$ sudo snappy remove freedomotic
Removing freedomotic
Waiting for freedomotic_freedomotic_5.6.0.service to stop.
$ sudo snappy purge freedomotic
Purging freedomotic
$ sudo snappy install --allow-unauthenticated /tmp/freedomotic_5.6.0_armhf.snap
FYI, I built a package with the above changes and now I can use
'freedomotic.start', and it indicates a problem. Eg:
$ freedomotic.start
/apps/freedomotic.sideload/5.6.0/bin/start: 4: cd: can't cd to
/apps/freedomotic/current
Launching Freedomotic runtime...
/apps/freedomotic.sideload/5.6.0/bin/start: 9:
/apps/freedomotic.sideload/5.6.0/bin/start:
/apps/freedomotic/current/jre/bin/java: not found
This is because the app was sideloaded and your 'start' script doesn't handle
that well. You should use the SNAP_* variables in your 'start' script so you
aren't hardcoding paths.
Eg:
$ sudo snappy install hello-world
...
$ hello-world.env|grep SNAP_
SNAP_APP_PATH=/apps/hello-world.canonical/1.0.15
SNAP_ORIGIN=canonical
SNAP_APP_USER_DATA_PATH=/home/ubuntu//apps/hello-world.canonical/1.0.15
SNAP_FULLNAME=hello-world.canonical
SNAP_NAME=hello-world
SNAP_APP_TMPDIR=/tmp/snaps/hello-world.canonical/1.0.15/tmp
SNAP_OLD_PWD=/tmp
SNAP_APP_DATA_PATH=/var/lib//apps/hello-world.canonical/1.0.15
Do be aware of this bug though:
https://bugs.launchpad.net/snappy-ubuntu/+bug/1449625
So, looking at 'start', if I change this:
appdir=/apps/freedomotic/current
cd $appdir
export JAVA_HOME="/apps/freedomotic/current/jre"
to:
appdir=$SNAP_APP_PATH
cd $appdir
export JAVA_HOME="$appdir/jre"
then "freedomatic.start" tries to do something:
$ freedomotic.start
Launching Freedomotic runtime...
log4j:ERROR Could not find value for key log4j.appender.default.file
log4j:ERROR Could not instantiate appender named "default.file".
INFO [main] - Freedomotic instance ID: c020cc66-8aba-4274-9ccf-2595037d16d6
INFO [main] - Creating new messaging broker
INFO [main] - Configuring messaging broker
INFO [main] - /apps/freedomotic.sideload/5.6.0/freedomotic
The systemd service also tries to do something too:
$ sudo systemctl stop freedomotic_freedomotic_5.6.0.service
...
$ sudo systemctl start freedomotic_freedomotic_5.6.0.service
However, there is a seccomp denial[3][4]:
$ sudo sc-logresolve /var/log/syslog
May 11 22:00:17 localhost kernel: [264654.298530] audit: type=1326
audit(1431381617.920:34): auid=1000 uid=1000 gid=1000 ses=83 pid=5889
comm="java" exe="/apps/freedomotic.sideload/5.6.0/jre/bin/java" sig=31
arch=40000028 syscall=288(socketpair) compat=0 ip=0xb6e9ab86 code=0x0
'socketpair' is part of the 'network-service' cap (apps by default only get
client networking), so you can change your yaml to:
name: freedomotic
architecture: armhf
version: 5.6.0
vendor: Freedomotic Team <info@freedomotic.com>
icon: meta/freedomotic.svg
services:
- name: freedomotic
description: "freedomotic runtime"
start: bin/start
caps:
- network-service
binaries:
- name: start
exec: bin/start
description: "freedomotic runtime cli"
caps:
- network-service
After making these changes, the app gets farther along, but has another security
denial:
apparmor="DENIED" operation="mknod"
profile="freedomotic.sideload_freedomotic_5.6.0"
name="/apps/freedomotic.sideload/5.6.0/freedomotic/plugins/objects/base-things/data/cmd/index.txt"
pid=6557 comm="java" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
the app is incorrectly trying to write to the read-only install directory--
security policy enforces the snappy FHS[5]. It should instead be modified to
write out to SNAP_APP_DATA_PATH.
Attached is a diff of the changes I made.
Hope this helps
References:
[1]https://developer.ubuntu.com/en/snappy/guides/packaging-format-apps/
[2]https://developer.ubuntu.com/en/snappy/guides/package-metadata/
[3]https://developer.ubuntu.com/en/snappy/guides/security-policy/
[4]https://wiki.ubuntu.com/SecurityTeam/Specifications/SnappyConfinement#Debugging
[5]https://developer.ubuntu.com/en/snappy/guides/filesystem-layout/
https://developer.ubuntu.com/en/snappy/guides/