bcc
BCC: a powerful front end to extended Berkeley Packet Filters
The BPF Compiler Collection (BCC) is a toolkit for building kernel tracing tools that leverage the functionality provided by the Linux extended Berkeley Packet Filters (BPF).
BCC allows one to write BPF programs with front-ends in Python or Lua with kernel instrumentation written in C. The instrumentation code is built into sandboxed eBPF byte code and is executed in the kernel.
The BCC github project README file provides an excellent overview and description of BCC and the various available BCC tools. Building BCC from scratch can be a bit time consuming, however, the good news is that the BCC tools are now available as a snap and so BCC can be quickly and easily installed just using:
sudo snap install --devmode bcc
There are currently over 50 BCC tools in the snap:
bcc.argdist bcc.cpuunclaimed bcc.gethostlatency bcc.slabratetop bcc.tcpretrans bcc.bashreadline bcc.dcsnoop bcc.hardirqs bcc.softirqs bcc.tcptop bcc.biolatency bcc.dcstat bcc.killsnoop bcc.sslsniff bcc.trace bcc.biosnoop bcc.execsnoop bcc.mdflush bcc.stackcount bcc.ttysnoop bcc.biotop bcc.ext4dist bcc.memleak bcc.stacksnoop bcc.vfscount bcc.bitesize bcc.ext4slower bcc.offcputime bcc.statsnoop bcc.vfsstat bcc.btrfsdist bcc.filelife bcc.offwaketime bcc.syncsnoop bcc.wakeuptime bcc.btrfsslower bcc.fileslower bcc.oomkill bcc.tcpaccept bcc.xfsdist bcc.cachestat bcc.filetop bcc.opensnoop bcc.tcpconnect bcc.xfsslower bcc.cachetop bcc.funccount bcc.pidpersec bcc.tcpconnlat bcc.zfsdist bcc.cpudist bcc.funclatency bcc.runqlat bcc.tcplife bcc.zfsslower
Let's have a quick look at a few:
cachetop
cachetop allows one to view the top page cache hit/miss statistics. To run this use:
sudo bcc.cachetop
23:11:26 Buffers MB: 123 / Cached MB: 1025 / Sort: HITS / Order: ascending
PID UID CMD HITS MISSES DIRTIES READ_HIT% WRITE_HIT%
10865 _apt gpgv 2 0 0 100.0% 0.0%
373 root jbd2/vda1-8 3 1 1 50.0% 25.0%
10856 root python 4 0 0 100.0% 0.0%
2190 messageb dbus-daemon 6 0 0 100.0% 0.0%
421 root systemd-journal 6 0 0 100.0% 0.0%
2261 syslog rs:main Q:Reg 7 0 2 71.4% 0.0%
10857 root sudo 13 0 0 100.0% 0.0%
10857 king sudo 14 0 0 100.0% 0.0%
10862 _apt http 34 0 0 100.0% 0.0%
10861 root http 48 0 0 100.0% 0.0%
10862 root http 48 0 0 100.0% 0.0%
10863 root http 48 0 0 100.0% 0.0%
10865 root gpgv 48 0 0 100.0% 0.0%
11092 root store 48 0 0 100.0% 0.0%
10975 _apt apt-key 150 0 0 100.0% 0.0%
10977 _apt apt-key 150 0 0 100.0% 0.0%
10978 _apt apt-key 150 0 0 100.0% 0.0%
10987 _apt apt-key 150 0 0 100.0% 0.0%
10989 _apt apt-key 150 0 0 100.0% 0.0%
10990 _apt apt-key 150 0 0 100.0% 0.0%
10997 _apt apt-key 150 0 0 100.0% 0.0%
funccount
The funccount tool allows one to count the number of times specific functions get called. For example, to see how many kernel functions with the name starting with "do_" get called per second one can use:
sudo bcc.funccount "do_*" -i 1 FUNC COUNT do_task_dead 1 do_futex 1 do_exit 1 do_wp_page 1 do_writepages 1 do_truncate 1 do_nanosleep 2 do_flush_tlb_all 3 do_get_write_access 3 do_output_char 3 do_select 5 do_sys_poll 6 do_sys_ftruncate.constprop.15 10 do_timerfd_settime 15 do_IRQ 32 do_check 146 do_timer 150 do_mmap 201 do_dentry_open 332 do_filp_open 522 do_sys_open 636 do_vfs_ioctl 803 do_kernel_range_flush 1004 do_syscall_64 4236
To see how to use all the options in this tool, use the -h option:
sudo bcc.funccount -h
The funccount tool can be useful to check on kernel activity by checking on hits on specific function names.
slabratetop
The slabratetop tool is useful to see the active kernel SLAB/SLUB memory allocation rates:
sudo bcc.slabratetop 14:22:01 loadavg: 2.10 2.47 1.63 3/183 9955 CACHE ALLOCS BYTES TCP 4986 9892224 sock_inode_cache 4986 3191040 kmalloc-256 7549 1932544 dentry 4999 959808 request_sock_TCP 2493 817704 tw_sock_TCP 2493 678096 kmalloc-4096 74 303104 kmalloc-64 2493 159552 vm_area_struct 120 24000 anon_vma_chain 126 8064 shmem_inode_cache 11 7656 anon_vma 65 5200 mm_struct 2 3968 cred_jar 12 2304 sighand_cache 1 2112 signal_cache 1 1088 files_cache 1 704 proc_inode_cache 1 648 inode_cache 1 592 buffer_head 2 208
opensnoop
If you want to see which process is opening specific files, one can snoop on open system calls use the opensnoop tool:
sudo bcc.opensnoop -T PID COMM FD ERR PATH 2433 irqbalance 3 0 /proc/interrupts 2433 irqbalance 3 0 /proc/stat 2433 irqbalance 3 0 /proc/interrupts 2433 irqbalance 3 0 /proc/stat 2433 irqbalance 3 0 /proc/interrupts 2433 irqbalance 3 0 /proc/stat 2433 irqbalance 3 0 /proc/interrupts 2433 irqbalance 3 0 /proc/stat 1 systemd 15 0 /proc/2223/cgroup 1 systemd 15 0 /proc/562/cgroup 1 systemd 15 0 /proc/2290/cgroup 2433 irqbalance 3 0 /proc/interrupts 2433 irqbalance 3 0 /proc/stat
Hopefully this will give you a taste of some of the useful tools that are available in BCC. The BCC snap is regularly updated to be in-sync with the bcc project, so it is worth regularly checking for new or updated bcc tools.
Kernel/Reference/bcc (last edited 2017-01-06 14:57:07 by colin-king)