Tuesday 29 December 2015

pagemon: an ncurses based tool to monitor process memory

While developing stress-ng I wanted to be able to see if the various memory stressors were touching memory in the way I had anticipated.  While digging around in the Linux documentation I discovered the very useful soft/dirty bit on Page Table Entries (PTEs) that get set when a page is written to.  The mechanism to check for the soft/dirty bit is described in Documentation/vm/soft-dirty.txt; one needs to:
  1. Clear the soft-dirty bits on the PTEs on a chosen process by writing "4" to /proc/$PID/clear_refs
  2. Wait a while for some page activity to occur
  3. Read the soft-dirty bits on the PTEs to see which pages got written to.
Not too tricky, so how about using this neat feature? While on rather long and dull flight over the Atlantic back in August I hacked up a very crude ncurses based tool to continually check the PTEs of a given process and display the soft/dirty activity in real time.  During this Christmas break I picked this code up and re-worked into a more polished tool.  One can scroll up/down the memory maps and also select a page and view the contents changing in real time.  The tool identifies the type of memory mapping a page belongs to, so one can easily scan through memory looking at pages of memory belonging data, code, heap, stack, anonymous mappings or even swapped out pages.

Running it on X, compiz, firefox or thunderbird is quite instructive as one can see a lot of page activity on the large heap allocations.  The ability to see pages getting swapped out when memory pressure is high is also rather useful.

Page view of Xorg
Memory view of stack
The code is still early development quality (so expect some buglets!) and I need to work on optimising it in a lot of places, but for now, it works well enough to be a fairly interesting tool. I've currently got a package built for Ubuntu Xenial in ppa:colin-king/pagemon and the source can be cloned from http://kernel.ubuntu.com/git/cking/pagemon.git/

So, to install on Xenial, currently one needs to do:

sudo add-apt-repository ppa:colin-king/pagemon
sudo apt-get update
sudo apt-get install pagemon

I may be adding a few more features in the next few weeks, and then getting the tool into Ubuntu and Debian.

and as an example, running it on Xorg, it is invoked as:

sudo pagemon -p $(pidof Xorg)

Unfortunately sudo is required to allow one to dig so intrusively into a running process. For more details on how to use pagemon consult the pagemon man page, or press "h" or "?" while running pagemon.

Thursday 17 December 2015

Incorporating and accessing binary data into a C program

The other day I needed to incorporate a large blob of binary data in a C program. One simple way is to use xxd, for example, on the binary data in file "blob", one can do:

xxd --include blob 

 unsigned char blob[] = {  
  0xc8, 0xe5, 0x54, 0xee, 0x8f, 0xd7, 0x9f, 0x18, 0x9a, 0x63, 0x87, 0xbb,  
  0x12, 0xe4, 0x04, 0x0f, 0xa7, 0xb6, 0x16, 0xd0, 0x70, 0x06, 0xbc, 0x57,  
  0x4b, 0xaf, 0xae, 0xa2, 0xf2, 0x6b, 0xf4, 0xc6, 0xb1, 0xaa, 0x93, 0xf2,  
  0x12, 0x39, 0x19, 0xee, 0x7c, 0x59, 0x03, 0x81, 0xae, 0xd3, 0x28, 0x89,  
  0x05, 0x7c, 0x4e, 0x8b, 0xe5, 0x98, 0x35, 0xe8, 0xab, 0x2c, 0x7b, 0xd7,  
  0xf9, 0x2e, 0xba, 0x01, 0xd4, 0xd9, 0x2e, 0x86, 0xb8, 0xef, 0x41, 0xf8,  
  0x8e, 0x10, 0x36, 0x46, 0x82, 0xc4, 0x38, 0x17, 0x2e, 0x1c, 0xc9, 0x1f,  
  0x3d, 0x1c, 0x51, 0x0b, 0xc9, 0x5f, 0xa7, 0xa4, 0xdc, 0x95, 0x35, 0xaa,  
  0xdb, 0x51, 0xf6, 0x75, 0x52, 0xc3, 0x4e, 0x92, 0x27, 0x01, 0x69, 0x4c,  
  0xc1, 0xf0, 0x70, 0x32, 0xf2, 0xb1, 0x87, 0x69, 0xb4, 0xf3, 0x7f, 0x3b,  
  0x53, 0xfd, 0xc9, 0xd7, 0x8b, 0xc3, 0x08, 0x8f  
 unsigned int blob_len = 128;  

..and redirecting the output from xxd into a C source and compiling this simple and easy to do.

However, for large binary blobs, the C source can be huge, so an alternative way is to use the linker ld as follows:

ld -s -r -b binary -o blob.o blob  

...and this generates the blob.o object code. To reference the data in a program one needs to determine the symbol names of the start, end and perhaps the length too. One can use objdump to find this as follows:

 objdump -t blob.o  
 blob.o:   file format elf64-x86-64  
 0000000000000000 l  d .data        0000000000000000 .data  
 0000000000000080 g    .data        0000000000000000 _binary_blob_end  
 0000000000000000 g    .data        0000000000000000 _binary_blob_start  
 0000000000000080 g    *ABS*        0000000000000000 _binary_blob_size  

To access the data in C, use something like the following:

 cat test.c  
 #include <stdio.h>  
 int main(void)  
         extern void *_binary_blob_start, *_binary_blob_end;  
         void *start = &_binary_blob_start,  
            *end = &_binary_blob_end;  
         printf("Data: %p..%p (%zu bytes)\n",   
                 start, end, end - start);  
         return 0;  

...and link and run as follows:

 gcc test.c blob.o -o test  
 Data: 0x601038..0x6010b8 (128 bytes)  

So for large blobs, I personally favour using ld to do the hard work for me since I don't need another tool (such as xxd) and it removes the need to convert a blob into C and then compile this.

Firmware Test Suite, 15.12.00

The Canonical Hardware Enablement Team and myself are continuing the work to enhance the Firmware Test Suite (fwts) on a regular monthly cadence.  The latest changes in FWTS 15.12.00 includes the following new features and changes:

  • ACPI: ASPT (System Performance Tuning Table)
  • Update ACPICA to version 20151124 
  • Boot path sync with UEFI specification 2.5 adding:
    • SD device path 
    • Bluetooth device path
    • Wireless device path
    • Ramdisk device path
  • Mixed tests and test category options, e.g. fwts --uefitests klog cpufreq will run all the UEFI tests as well as klog and cpufreq tests
  • A new --log-level option that allows one to log test that fail at specified a level or higher, e.g. fwts --log-level high will just show high and critical test failures.
  • The apcidump table dump pseudo-test is now aligned with the ACPICA table dumping (disassembly) engine.
  • Various bug fixes.
It is also worth mentioning that the UEFI Board of Directors recommends FWTS as the ACPI v5.1 Self-Certification Test (SCT). This is exciting news and we welcome this decision for FWTS to be recognised in this way.

We are also very grateful for the community contributions to FWTS, this buy-in from community is appreciated and makes FWTS a better tool to support different architectures and systems.

As ever, with new releases, please consult the release notes.

Friday 11 December 2015

Another seasonal obfuscated C program

During an idle moment while on vacation I was reading the paper "Reliable Two-Dimensional Graphing Methods for Mathematical Formulae with Two Free Variables" by Jeff Tupper and I stumbled upon rather amusing inequality at the end of section 12.   In tribute to this most excellent graphing formula, I felt inspired to use the same concept in my Christmas 2015 obfuscated C offering.


I cheated a little by also using a Makefile, but I hope this also adds to the magic of the resulting code.  To make the program more fun I thought I'd use a lot of confusion logic operator names in the code and mix in some incorrect Roman numeral constants too.  I could have obfuscated the code more and made it smaller, but life is too short. I will leave that as an exercise to the reader.

The source is available in my Christmas Obfuscated C git repository if you want to try it out:

 git clone https://github.com/ColinIanKing/christmas-obfuscated-C.git  
 cd christmas-obfuscated-C/2015  
 ./tupper | less