After looking at yesterdays transfer logs on the Colo FTP server, I decided to grok the fields in the xferlog and put them to use.

The ftp transfer log is in /var/log/xferlog. It it rotated daily, and compressed with bz. I’m using yesterdays log, so I have to look at xferlog.0.bz, and use bzgrep to find all of the “Jan 16” transfers. After today, we can reliably use bzcat.

Here is a snippet:

Wed Jan 16 23:57:23 2013 19 10.1.1.10 12627038 /path/to/ftp/root/order_XXXXXXX_XXXXXXXXXXXXXXXXXXX_NNNNN b _ i r user ftp 0 * c

• current-time
• transfer-time
• remote-host
• file-size
• filename
• transfer-type
  • a - Ascii
  • b - Binary
• special-action-flag
  • C - Compressed
  • U - Uncompressed
  • T - Tar’d
  • _ - No Action
• direction
  • i - Incoming
  • o - Outgoing
• access-mode
  • a - Anonymous
  • r - Real User
• username
• service-name (ftp)
• authenticated user id
• completion-status
  • c - Completed
  • i - Incompleted

With all of that, we can obtain a lot of information about a days worth of transfers.

Looking at the example above, we know the following:

• Time: Wed Jan 16 23:57:23 2013 19
• Remote host (the proxy is getting in the way, I’ll see if I can get the re-written): 10.1.1.10
• File Size: 12627038
• Filename: /path/to/ftp/root/order_XXXXXXX_XXXXXXXXXXXXXXXXXXX_NNNNN
• Binary Transfer: b
• No Special Actions: _
• Incoming file: i
• Real user: r
• Login Name: user
• Service: ftp 0
• Authenticated ID not available (LDAP settings)
• Completed transfer: c

Using Awk, I was quickly able to add up the total file size for yesterday (since 10:16am when we flipped over):

bzgrep Jan\ 16 /var/log/xferlog.0.bz| awk '{ sum += $8; count++ } END {print (sum/1024/1024/1024),count}'

Our Total was:

• 46.2587GB
• Number of transfers: 907

The number of incomplete transfers:

bzgrep "Jan 16" /var/log/xferlog.0.bz  | \ 
    awk '{ \
      if ($NF == "c" ? complete++ : incomplete++) \
   } END { \
    print "Complete: " complete "\n" "Incomplete: " incomplete \
 }'

Completed vs Incomplete upload:

• Complete: 864
• Incomplete: 43

I can now revise the first sum of transferred orders to drop incomplete orders:

bzgrep Jan\ 16 /var/log/xferlog.0.bz | awk '{ if ($NF == "c") { sum += $8; count++} } END {print (sum/1024/1024/1024),count}'

• Total data transferred: 41.0385
• Number of Completed transfers: 864