I was fortunate enough to get some free drives from someone at work. They live in Austin, and offered up a big heavy box of hard drives to anyone who would pay for shipping. I’ve had a very simple ZFS pool for around 5 years now, composed of 3 2TB drives in a raidz1 with a cold spare.
There have been no issues, I scrub them monthly, however ever since I placed my /home volume onto the ZFS pool, I noticed some large delays when working with a git repo. Read More...
Introduction I started a little home infrastructure project a year. It was mostly because I really wanted to check out ISC’s new DHCP server, kea. Primarily because it is API driven, and the configuration files are in JSON.
At work, I am on a team that is in charge of a few thousand servers all colocated in 4 datacenters across Chicago. For the longest time we have managed to keep those servers auto-registering and pxe booting with 2 VM’s running CentOS and ISC’s DHCPD server. Read More...
Gitorious has some documentation on a local install for Ubuntu, RHEL, and Debian, which turned out to be slightly outdated. Now that they have leveraged the Bundler GEM manager, the installation process is much simpler.
FreeBSD is by far my platform of choice when it comes to flexible and complex open source applications, and it was a cinch to reuse the Linux documentation. Aside from having to re-write the init scripts, the entire process was very easy, and I got to try out a few different web servers for the heck of it. Read More...
I’m pretty happy with the Bacula environment I’ve created.
It has gone through a few iterations, and I’ve learned a lot since I started using it a few years ago. I think its only appropriate to share the evolution of my environment with as many people as possible, and I hope it can help save new bacula administrators some time.
Enough of the preamble, here is my github project page: https://github. Read More...
Atlassian has a wonderful set of tools.
I’ve been using Jira with Fisheye for the last few months, but I recently came across Stash.
Stash is a nice git repository and project manager than provides easy access to creating new Git repo’s with User and Group access controls. I’m considering phasing out FishEye in favor of Stash. I only wanted FishEye as a source code browser, that integrated issues tracking. Read More...
LinkedIn’s users database was leaked, and while passwords were not stored in plain text, they were hashed with the MD5 algorithm. Not salted, just hashed.
MD5 is no longer as secure as it once was, not with all of these GPU’s lying around.
Well, the FreeBSD community has not let this go unnoticed. First off, FreeBSD has defaulted to MD5 for a while, but it has had support for Blow-Fish, and (this is new to me) SHA-256 and SHA-512. Read More...
If you have google’d for smb.conf settings to max your 1Gb (or 10Gb) ethernet based network and you still cannot seem to get beyond 30MB/sec, make sure you have this option:
socket options = SO_RCVBUF=8192 SO_SNDBUF=8192 Either commented it out, or set the buffer size to 128K (131072). Or larger.
I was helping a friend of mine troubleshoot his ZFS + Samba environment.
He was running FreeBSD 9.0 and Samba 3. Read More...
I’ve been using ZFS on FreeBSD since it was first made available in 7.0, and it was not until FreeBSD 8.2 when NFSv4 ACL’s were implemented.
At $oldJob, I had a list of “To do’s”, really cool things like:
“Integrate Puppet into CMDB”, “Rewrite the Bacula addClient script”, “Build a MooseFS Cluster”… At the top though, I had:
Implement ACL’s on ZFS/Samba fileserver Anyway my $newJob always has a high demand for storage and sharing data. Read More...
I like Nginx a lot.
Not because I’m some sort of hipster sysadmin either.
I like it because it is small, fast, and as a FreeBSD port, it compiles and updates quickly.
What I also like is the separation of services and processes. For example, if I want to run a PHP script, I don’t load “mod_php” like you would with Apache. Instead, you have a PHP processor, like php-fpm, running (on localhost, or, another server that only processes PHP scripts). Read More...
My previous experience with a large virtual “enterprise” environment was with VMWare’s ESXi and vSphere.
Performance wise, I was always fairly happy with ESXi knowing that it was a virtual machine of course. There were a few issues I recall:
silent data corruption (fsck’ing Linux and FreeBSD volumes would reveal inconsistent filesystem information, but there were never any errors reported to the VM) Live migrations were not always stable. In fact, most of the time it would result in an unexpected shutdown I doubt it was solely VMWares fault, as it could have been a series of mis-configurations and poor implementation descisions. Read More...
On aspect that I was never happy with the Bacula environment I built while at LLNL was the fact that I could no look up certain values for each client. Values like: * Passwords * Storage Devices * Certificates (if you are using Encryption)
Well, over the past few week’s I’ve been able to work around this problem by storing additional information in a CouchDB DB.
It is not the ideal solution, but it is a start and I’m okay with that. Read More...
At work I ran a 5 part FreeBSD Administration training course for the Unix team.
I enjoyed it a lot, because I really like to share information. It is especially rewarding when it is something I have taken a great interest in, like the FreeBSD Operating System.
The design of the course was simple, I did my best to fill in the gap that a Solaris or Linux administrator might have. Read More...
Software As mentioned many times, this is a FreeBSD based environment. Some good sysinfo output below:
HAHAHUGOSHORTCODE-0xc0009c4a00-1-HBHB
Bootloader settings for the Director/Database node:
The /boot/loader.conf has the following contents: HAHAHUGOSHORTCODE-0xc0009c4a00-2-HBHB
All of the storage nodes and the director are running a GENERIC kernel with very few system tweaking. One of the storage nodes has a Chelsio 10Gb controller, but that hasn’t had a high enough load to crack the 1Gb/sec barrier. Read More...
I’ve been using Bacula, the open source backup software, for over a year now. Things have been going well, and I would like to dedicate a post or two to the environment I built.
Background Over a year ago, I took it upon myself to replace a single Legato Networker server with Bacula. One of our collaborators had decided to ship us (for no reason at all really, I think they were cleaning out their data center) a Sun X4200 AMD server, and two StorageTek/Sun NAS servers. Read More...
A while ago, I posted about how I backup my server with Duplicity to Amazon’s S3 storage.
To follow up, here is a little guide I wrote on using Duplicity in the everyday work environment
Overview Duplicity is a backup tool that will create compressed and encrypted (uses gnupg) backup archives. It can use a variety of protocols as the target (file, ftp, webdav, imap, ssh/scp, rsync, hsi, s3 and hsi). Read More...
I can be a glutton for punishment for a nearly trivial amount of gain. So lets bring on the two-factor authentication for my personal FreeBSD server.
I’ve been using Google’s 2-step verification since Jenny told me about it, along with my android powered phone.
What is nice about Google’s Authenticator app is its availability for multiple smartphone platforms:
Android version 1.5 or later BlackBerry OS 4.2 - 4.7 iPhone iOS 4 or later How it works is pretty simple. Read More...
To follow up on two previous posts, the Buckethead concert and making your own YouTube bootlegs, I’m going to add just a little more.
As I mentioned, the concert was great. If I were to pick out one part, it would have been this lengthy untitled (as far as I know) freestyle jam he did with a simple looping beat.
I cannot think of anything worse than falling in love with a song at a concert, and not knowing the name of it. Read More...
I’ve been using Puppet at work for the handful of FreeBSD and, recently, Ubuntu desktops.
Aside from some very simple system configuration management (I’ve not yet dived too deep into puppet. I mostly use it for configuring system authentication and ensuring a particular computer security baseline), I though it would have been great to store the client’s “facts” into a accessable database.
Sometime last year, Puppet added the ability to store facts into a Couch Database: http://www. Read More...
I love the BSD community. How cool is it to have developers and end users from the various BSD projects in one location, there to talk about the various cool projects and technical challenges that face us now.
Cool I tell you!
Chris, Corrigan and I went to Google in 2008 for MeetBSD. We had a great time, so when I saw the announcement earlier this year for MeetBSD 2010, I rallied the troops. Read More...
I didn’t find any blog posts of discussions on FreeBSD and multipath (for storage) that wasn’t a man page.
That means it is up to me to write about it :)
Hardware CPU Machine class: amd64 CPU Model: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz No. of Cores: 16 Memory Total real memory available: 65511 MB Logically used memory: 3945 MB Logically available memory: 61565 MB Storage The storage is a large ~90TB Enterprise class Fibre Channel array, a Data Direct Networks S2A9900. Read More...
I don’t have a central theme with this post, but I wanted to at least do something (it has been a while).
Packet Filter Based on Chris’s “Falling in love with pf(4)” google status, I decided to take the plunge and move off of ipfw(4) to pf(4). I’m not at the point where I could write my one filter, however, I do feel I at least understand what is happening here. Read More...
I recently configured a NFS/Samba server with FreeBSD’s Link Aggregation protocol. Here is how I set it up.
FreeBSD Configuration /boot/loader.conf
I recommend adding the if_lagg_load=YES and kern.hz=“2000” to /boot/loader.conf.
The OS will automatically load the lagg kernel module when your network configuration loads, I prefer to explicitly set it to load.
ispfw_load="YES" kern.hz="2000" aio_load="YES" hw.igb.rxd=4096 hw.igb.txd=4096 if_lagg_load="YES" Since I am using the igb ethernet device (Intel 82575 and 82576 chipsets), I also set the max number of send and receive descriptors from the default from 256 to the maximum 4096. Read More...
I need to share this. When I google for “Samba performance”, I never see real numbers, real configuration files, or real hardware environments. All I read are anecdotal recollections, and that is not good enough. I like numbers, and I’ll let the numbers speak for themselves:
> netstat -I em0 -w 1 input (em0) output packets errs bytes packets errs bytes colls 90166 0 98762637 95363 0 5332847 0 18131 0 24713156 20042 0 1123684 0 4 0 310 1 0 178 0 8 0 518 1 0 178 0 10153 0 10952920 10696 0 598129 0 92990 0 102837002 98476 0 5514994 0 92025 0 102680574 97277 0 5439496 0 92080 0 101799874 97403 0 5448637 0 75348 0 90861608 80972 0 4537737 0 90895 0 100323946 95781 0 5360948 0 89313 0 97371154 94364 0 5278618 0 81363 0 89229738 85861 0 4803589 0 2 0 126 3 0 286 0 I was so shocked that I had to use gstat and zpool iostat to verify the information: Read More...
I recently had a little problem with a new FreeBSD install, and it is one of those times were I sort of appreciate how FreeBSD assigns device handles, yet at the same time hate it :)
The setup is this: The OS was installed on a mirrored hardware raid device (using the mpt(4) driver), and then I had a large RAID6 array attached via a FC controller (using the isp(4) driver). Read More...
I’ve always had a vested interest in the entire nvidia display driver for FreeBSD project, and I’ve pretty attached to the project. So much, that back in 2001 I started a little petition, got enough attention (and more importantly, a large list of people who signed my petition), and ever since 2002 FreeBSD users have been able to use high quality nvidia drivers. It wasn’t all me, whoever ran nvidia.netexplorer.org asked me to combine efforts, and I gave them my list, and they continued to market it and work with some folks at nvidia. Read More...
So, it looks like FreeBSD 8.0 has been pre-released; the official date is going to be 11⁄25, as noted in src/UPDATING:
Updating Information for FreeBSD current users This file is maintained and copyrighted by M. Warner Losh . See end of file for further details. For commonly done items, please see the COMMON ITEMS: section later in the file. Items affecting the ports and packages system can be found in /usr/ports/UPDATING. Read More...
This was a very cool conference. I picked up a lot of useful information on both the open source tool, Puppet, and some ideas on infrastructure.
What also made this conference unique, is how honest the Puppet team and community were about the projects strength and weaknesses. Those that have deployed Puppet on a larger scale (MessageOne and Google) seemed to go through the same iterations in attempting to scale out their Puppetmaster’s. Read More...
PC-BSD KDE Desktop
PC-BSD is a nice mesh between FreeBSD and a ready to use Desktop (which uses about 6GB of disk space). It is based on FreeBSD 7.2, so it has all the cool features of the latest release. Best of all, without ANY additional configuration, I was able to:
Use the official FreeBSD nVidia driver for hardware acceleration
Watch clips on YouTube (with flashplayer)
Play back all sorts of media types like mp3’s, divx, mpegs,wmv,qt… Read More...
FreeBSD 7.x has been using version 6 of ZFS, and originally only 8.0 was going to have the newly updated ZFS version: 13.
Last week the core team MFC’d (Merge From Current) the ZFS updates to 7.2, so I cvsup’d and re-build my server’s kernel and world ( with a simple “make buildworld && make buildkernel && make installworld && make installkernel ), rebooted, and now I have the latest ZFS version running: Read More...
I don’t have a backup system for home (which is where this site, and others are located), and I have generally relied on duplicating enough of my important stuff between friends and other computers. That, and I have a RAID5 setup for my large storage, and then home directories and website stuff is on a RAID1 ZFS volume. This doesn’t prevent accidental “oh-no”s, but it does protect me from some hardware failures. Read More...
!!! UPDATE on 12/29/2009!!! Since this blog post seems to get a good amount or hits from google, if you are reading this, please see my updated post: http://www.mywushublog.com/2009/12/freebsd-8-0-a-great-nas-server/ which has some additional information about FreeBSD 8.0 EOF
Lately at work, I’ve been involved with a very large file system that is being export from Solaris 10/ZFS to windows and OS X users via Samba. Even with a very large Sun server (T5220) a lot of users are complaining about the slow performance of the system. Read More...
I’ve configured this blog to use my OpenID accounts. I have two (which totally goes against the single identity mindset of OpenID :) )
http://mcarlson.myopenid.com/
http://openid.evil-genius-network.com/mike
The second one I just stood up today. I’m always concerned with who has my information, and if I can, I try to keep it all within the realm of my control. Also, the evil genius domain has absolutely no purpose besides a testing ground that I have no problems destroying :) Read More...
Thankfully I didn’t have to get up at the crack of dawn (7:30!) because it was scheduled to start at 11am. I was a little rushed for time since I had to drop Caralyne off at my parents in Knightsen (so I back-tracked a bit), and I left at 9am, so I barely made it in time for the ZFS talk.
A Closer Look at the ZFS File System by Pawel Jukab Dawidek I’ve heard a great talk on ZFS from Bill Moore, one of the primary developers from Sun, but this talk was really cool because it didn’t just say what ZFS is from Sun’s marketting department’s point of view, but the technical details its mail peices and how it integrates in FreeBSD. Read More...
I spent the weekend at Google for MeetBSD in celebration of FreeBSD’s 15 birthday. I drove 70.2x4 miles, on a weekend, and for Saturday I got up slightly earlier than I would have for work. All worth it, this was the coolest mini-conference I’ve been to. Of course, the last conference I went to was BSDCon in 2003, and that was nice as well. The point is, I don’t get out all that much when it comes to conferences, I just make an exception for my favorite UNIX OS, FreeBSD. Read More...
A friend of mine got my the coolest birthday present I think I’ve ever recieved, and that was a bunch of new and super kick ass hardware. This will soon replace my current server, which is in such bad shape it cannot compile java code, or perl from source. Before I replace it, I wanted to play around with ZFS that comes with FreeBSD 7.0.
Here is a quick rundown of it all: Read More...