published on

ftp stuff

After looking at yesterdays transfer logs on the Colo FTP server, I decided to grok the fields in the xferlog and put them to use.

The ftp transfer log is in /var/log/xferlog. It it rotated daily, and compressed with bz. I’m using yesterdays log, so I have to look at xferlog.0.bz, and use bzgrep to find all of the “Jan 16” transfers. After today, we can reliably use bzcat.

Here is a snippet:

Wed Jan 16 23:57:23 2013 19 10.1.1.10 12627038 /path/to/ftp/root/order_XXXXXXX_XXXXXXXXXXXXXXXXXXX_NNNNN b _ i r user ftp 0 * c
The fields are as followed:

  • current-time
  • transfer-time
  • remote-host
  • file-size
  • filename
  • transfer-type
    • a - Ascii
    • b - Binary
  • special-action-flag
    • C - Compressed
    • U - Uncompressed
    • T - Tar’d
    • _ - No Action
  • direction
    • i - Incoming
    • o - Outgoing
  • access-mode
    • a - Anonymous
    • r - Real User
  • username
  • service-name (ftp)
  • authenticated user id
  • completion-status
    • c - Completed
    • i - Incompleted

With all of that, we can obtain a lot of information about a days worth of transfers.

Looking at the example above, we know the following:

  • Time: Wed Jan 16 23:57:23 2013 19
  • Remote host (the proxy is getting in the way, I’ll see if I can get the re-written): 10.1.1.10
  • File Size: 12627038
  • Filename: /path/to/ftp/root/order_XXXXXXX_XXXXXXXXXXXXXXXXXXX_NNNNN
  • Binary Transfer: b
  • No Special Actions: _
  • Incoming file: i
  • Real user: r
  • Login Name: user
  • Service: ftp 0
  • Authenticated ID not available (LDAP settings)
  • Completed transfer: c

Using Awk, I was quickly able to add up the total file size for yesterday (since 10:16am when we flipped over):

bzgrep Jan\ 16 /var/log/xferlog.0.bz| awk '{ sum += $8; count++ } END {print (sum/1024/1024/1024),count}'

Our Total was:

  • 46.2587GB
  • Number of transfers: 907

The number of incomplete transfers:

bzgrep "Jan 16" /var/log/xferlog.0.bz  | \ 
    awk '{ \
      if ($NF == "c" ? complete++ : incomplete++) \
   } END { \
    print "Complete: " complete "\n" "Incomplete: " incomplete \
 }'

Completed vs Incomplete upload:

  • Complete: 864
  • Incomplete: 43

I can now revise the first sum of transferred orders to drop incomplete orders:

bzgrep Jan\ 16 /var/log/xferlog.0.bz | awk '{ if ($NF == "c") { sum += $8; count++} } END {print (sum/1024/1024/1024),count}'
  • Total data transferred: 41.0385
  • Number of Completed transfers: 864