ply

/* a dynamic tracer for Linux */

$_ flexible

Type inference in combination with a compact C-like syntax means scripts can be written and edited in seconds, directly in your favorite shell. The language is heavily inspired by, though not nearly as powerful as, awk(1) and dtrace(1).

>< small

Designed with embedded systems in mind. Written in C, all ply needs to run is libc and a modern kernel with Linux BPF support. No external kernel modules, no LLVM, no python. Works on aarch64, arm, powerpc, and x86_64 today — adding an ISA is easy.

<3 efficient

Data gathering and aggregation is done in the kernel using Linux BPF programs that are JIT compiled to native instructions on most common architectures. Thus, ply runs with very low overhead, allowing it to probe even the hottest code paths.

 "yeah yeah whatever, so what can I actually do with it?!"

- well I'm glad you asked, dear reader, here are a few examples

count syscalls, system-wide

#!/usr/bin/env ply

kprobe:SyS_*
{
    @syscalls[caller] = count();
}

This probe will be attached to all functions whose name starts with SyS_, i.e. all syscalls. On each syscall, the probe will fire and index into the user-defined aggregation @syscalls using the program counter caller as the key and bump a counter.

ply will compile the script, attach it to the matching probes and start collecting data. On exit, ply will dump the values of all user-defined maps and aggregations.

~$ ply count-syscalls.ply
ply: active
^Cply: deactivating

@syscalls:
{ sys_sigaltstack+1 }: 1
{ sys_set_tid_address+1 }: 2
{ sys_sysinfo+1 }: 2
{ sys_alarm+1 }: 2
{ sys_mincore+1 }: 2
{ sys_pread64+1 }: 2
{ sys_newlstat+1 }: 2
{ sys_unlink+1 }: 2
[REDACTED LINES]
{ sys_newfstat+1 }: 586
{ sys_mmap_pgoff+1 }: 599
{ sys_mmap+1 }: 600
{ sys_rt_sigprocmask+1 }: 859
{ sys_close+1 }: 863
{ sys_ppoll+1 }: 1315
{ sys_epoll_wait+1 }: 1731
{ sys_poll+1 }: 2644
{ sys_write+1 }: 3262
{ sys_recvmsg+1 }: 3624
{ sys_ioctl+1 }: 4423
{ sys_read+1 }: 5059
{ sys_futex+1 }: 7918

read(2) return distribution

#!/usr/bin/env ply

kretprobe:SyS_read
{
    @["retsize"] = quantize(retval);
}

This example shows a very simple script that instruments the return of the read(2) syscall and records the distribution of the return argument.

User-defined aggregations in ply always start with @. When writing small scripts, only using one aggregation, it is common and convenient to simply call it @.

~$ sudo ./read-dist.ply
ply: active
^Cply: deactivating

@:
{ retsize }: 
	         < 0	     861 ┤████████▏                       │
	[   0,    1]	     865 ┤████████▏                       │
	[   2,    3]	     372 ┤███▌                            │
	[   4,    7]	       1 ┤                                │
	[   8,   15]	    1090 ┤██████████▎                     │
	[  16,   31]	     122 ┤█▏                              │
	[  32,   63]	      22 ┤▎                               │
	[  64,  127]	      25 ┤▎                               │
	[ 128,  255]	       1 ┤                                │
	...
	[ 512,   1k)	      23 ┤▎                               │
	[  1k,   2k)	       5 ┤                                │
	...
	[  4k,   8k)	       2 ┤                                │
	...
	[ 16k,  32k)	      11 ┤▏                               │

capture stack traces

#!/usr/bin/env ply

kprobe:i2c_transfer
{
    print(stack);
}

Sometimes it can be useful to know how a particular location is reached. kprobes can get the current stack trace via the stack variable.

In this example, the stack trace is simply printed to stdout, but it can also be used as a map key in an aggregation. I.e. it is possible to do frequency counting based on how a function was reached.

root@chaos:~ $ ply ./i2c-stack.ply &
root@chaos:~ $ ply: active

root@chaos:~ $ hwclock -r

    i2c_transfer
    i2c_smbus_read_i2c_block_data+0x58
    ds1307_native_smbus_read_block_data+0x88
    ds1307_get_time+0x38
    __rtc_read_time+0x54
    rtc_read_time+0x3c
    rtc_dev_ioctl+0x318
    do_vfs_ioctl+0xa0
    sys_ioctl+0x44
    __sys_trace_return
Mon Feb 20 18:33:33 2017  0.000000 seconds
root@chaos:~ $ fg
ply ./i2c_stack.ply
^Cply: deactivating
root@chaos:~ $

opensnoop

#!/usr/bin/env ply

kprobe:do_sys_open
{
    printf("%v(%v): %s\n",
           comm, pid, str(arg1));
}

Every time a process calls open(2) print the calling process's comm, i.e. executable name, PID and the filename by extracting a 128-byte string from the address of the first argument using the str() function.

~$ sudo ./opensoop.ply
ply: active
ply             (28836): /sys/kernel/debug/tracing/events/enable
SimpleCacheWork ( 5818): /home/wkz/.cache/google-chrome/Default/Cache/37586f4b9464a393_0
irqbalance      ( 1083): /proc/interrupts
irqbalance      ( 1083): /proc/stat
irqbalance      ( 1083): /proc/irq/18/smp_affinity
irqbalance      ( 1083): /proc/irq/126/smp_affinity
irqbalance      ( 1083): /proc/irq/128/smp_affinity
irqbalance      ( 1083): /proc/irq/122/smp_affinity
irqbalance      ( 1083): /proc/irq/11/smp_affinity
irqbalance      ( 1083): /proc/irq/124/smp_affinity
irqbalance      ( 1083): /proc/irq/16/smp_affinity
irqbalance      ( 1083): /proc/irq/1/smp_affinity
irqbalance      ( 1083): /proc/irq/8/smp_affinity
irqbalance      ( 1083): /proc/irq/9/smp_affinity
irqbalance      ( 1083): /proc/irq/12/smp_affinity
irqbalance      ( 1083): /proc/irq/120/smp_affinity
irqbalance      ( 1083): /proc/irq/121/smp_affinity
Chrome_IOThread ( 5361): /dev/shm/.org.chromium.Chromium.59XkZF
SimpleCacheWork ( 5818): /home/wkz/.cache/google-chrome/Default/Cache/37586f4b9464a393_0
Core Thread     ( 5368): /home/wkz/.config/spotify/Users/wkz-user/pending-messages.tmp
Core Thread     ( 5368): /home/wkz/.config/spotify/Users/wkz-user/pending-messages.tmp
^Cply: deactivating