Freebsd

ELK Stack

published on
Needless Explanation I have been a long time Splunk user, and I think it is great. Still, I have had my eyes on other log aggregation and analytic tools for a while. Price is usually the pain point with Splunk, ask anyone, I also run FreeBSD, which was recently dropped as a supported server OS. There is still a universal forwarder for FreeBSD, and I am still pleased with that. Read More...

running gitorious on freebsd

published on
Gitorious has some documentation on a local install for Ubuntu, RHEL, and Debian, which turned out to be slightly outdated. Now that they have leveraged the Bundler GEM manager, the installation process is much simpler. FreeBSD is by far my platform of choice when it comes to flexible and complex open source applications, and it was a cinch to reuse the Linux documentation. Aside from having to re-write the init scripts, the entire process was very easy, and I got to try out a few different web servers for the heck of it. Read More...

bacula framework on github

published on
I’m pretty happy with the Bacula environment I’ve created. It has gone through a few iterations, and I’ve learned a lot since I started using it a few years ago. I think its only appropriate to share the evolution of my environment with as many people as possible, and I hope it can help save new bacula administrators some time. Enough of the preamble, here is my github project page: https://github. Read More...

atlassian tools and nginx

published on
Atlassian has a wonderful set of tools. I’ve been using Jira with Fisheye for the last few months, but I recently came across Stash. Stash is a nice git repository and project manager than provides easy access to creating new Git repo’s with User and Group access controls. I’m considering phasing out FishEye in favor of Stash. I only wanted FishEye as a source code browser, that integrated issues tracking. Read More...

puppet module to update freebsd passwd has

published on
LinkedIn’s users database was leaked, and while passwords were not stored in plain text, they were hashed with the MD5 algorithm. Not salted, just hashed. MD5 is no longer as secure as it once was, not with all of these GPU’s lying around. Well, the FreeBSD community has not let this go unnoticed. First off, FreeBSD has defaulted to MD5 for a while, but it has had support for Blow-Fish, and (this is new to me) SHA-256 and SHA-512. Read More...

poor samba performance

published on
If you have google’d for smb.conf settings to max your 1Gb (or 10Gb) ethernet based network and you still cannot seem to get beyond 30MB/sec, make sure you have this option: socket options = SO_RCVBUF=8192 SO_SNDBUF=8192 Either commented it out, or set the buffer size to 128K (131072). Or larger. I was helping a friend of mine troubleshoot his ZFS + Samba environment. He was running FreeBSD 9.0 and Samba 3. Read More...

zfs and acls with samba

published on
I’ve been using ZFS on FreeBSD since it was first made available in 7.0, and it was not until FreeBSD 8.2 when NFSv4 ACL’s were implemented. At $oldJob, I had a list of “To do’s”, really cool things like: “Integrate Puppet into CMDB”, “Rewrite the Bacula addClient script”, “Build a MooseFS Cluster”… At the top though, I had: Implement ACL’s on ZFS/Samba fileserver Actually, come to thing of it, I think the top of my list was: Read More...

mailman with nginx on freebsd

published on
I like Nginx a lot. Not because I’m some sort of hipster sysadmin either. I like it because it is small, fast, and as a FreeBSD port, it compiles and updates quickly. What I also like is the separation of services and processes. For example, if I want to run a PHP script, I don’t load “mod_php” like you would with Apache. Instead, you have a PHP processor, like php-fpm, running (on localhost, or, another server that only processes PHP scripts). Read More...

xen and convirture

published on
My previous experience with a large virtual “enterprise” environment was with VMWare’s ESXi and vSphere. Performance wise, I was always fairly happy with ESXi knowing that it was a virtual machine of course. There were a few issues I recall: silent data corruption (fsck’ing Linux and FreeBSD volumes would reveal inconsistent filesystem information, but there were never any errors reported to the VM) Live migrations were not always stable. In fact, most of the time it would result in an unexpected shutdown I doubt it was solely VMWares fault, as it could have been a series of mis-configurations and poor implementation descisions. Read More...

using couchdb with puppet and bacula

published on
On aspect that I was never happy with the Bacula environment I built while at LLNL was the fact that I could no look up certain values for each client. Values like: * Passwords * Storage Devices * Certificates (if you are using Encryption) Well, over the past few week’s I’ve been able to work around this problem by storing additional information in a CouchDB DB. It is not the ideal solution, but it is a start and I’m okay with that. Read More...

freebsd training

published on
At work I ran a 5 part FreeBSD Administration training course for the Unix team. I enjoyed it a lot, because I really like to share information. It is especially rewarding when it is something I have taken a great interest in, like the FreeBSD Operating System. The design of the course was simple, I did my best to fill in the gap that a Solaris or Linux administrator might have. Read More...

bacula in the enterprise part 2

published on
Software As mentioned many times, this is a FreeBSD based environment. Some good sysinfo output below: Operating system release: FreeBSD 8.2-RELEASE OS architecture: amd64 Kernel build dir location: /usr/obj/usr/src/sys/GENERIC Currently booted kernel: /boot/kernel/kernel Currently loaded kernel modules (kldstat(8)): zfs.ko opensolaris.ko Bootloader settings for the Director/Database node: The /boot/loader.conf has the following contents: kern.ipc.semmni=1024 kern.ipc.semmns=2048 kern.ipc.semmnu=1024 All of the storage nodes and the director are running a GENERIC kernel with very few system tweaking. Read More...

bacula in the enterprise part 1

published on
I’ve been using Bacula, the open source backup software, for over a year now. Things have been going well, and I would like to dedicate a post or two to the environment I built. Background Over a year ago, I took it upon myself to replace a single Legato Networker server with Bacula. One of our collaborators had decided to ship us (for no reason at all really, I think they were cleaning out their data center) a Sun X4200 AMD server, and two StorageTek/Sun NAS servers. Read More...

using duplicity

published on
A while ago, I posted about how I backup my server with Duplicity to Amazon’s S3 storage. To follow up, here is a little guide I wrote on using Duplicity in the everyday work environment Overview Duplicity is a backup tool that will create compressed and encrypted (uses gnupg) backup archives. It can use a variety of protocols as the target (file, ftp, webdav, imap, ssh/scp, rsync, hsi, s3 and hsi). Read More...

cheap two factor authentication with google

published on
I can be a glutton for punishment for a nearly trivial amount of gain. So lets bring on the two-factor authentication for my personal FreeBSD server. I’ve been using Google’s 2-step verification since Jenny told me about it, along with my android powered phone. What is nice about Google’s Authenticator app is its availability for multiple smartphone platforms: Android version 1.5 or later BlackBerry OS 4.2 - 4.7 iPhone iOS 4 or later How it works is pretty simple. Read More...

more fun with ffmpeg

published on
To follow up on two previous posts, the Buckethead concert and making your own YouTube bootlegs, I’m going to add just a little more. As I mentioned, the concert was great. If I were to pick out one part, it would have been this lengthy untitled (as far as I know) freestyle jam he did with a simple looping beat. I cannot think of anything worse than falling in love with a song at a concert, and not knowing the name of it. Read More...

couchdb and ruby on freebsd

published on
I’ve been using Puppet at work for the handful of FreeBSD and, recently, Ubuntu desktops. Aside from some very simple system configuration management (I’ve not yet dived too deep into puppet. I mostly use it for configuring system authentication and ensuring a particular computer security baseline), I though it would have been great to store the client’s “facts” into a accessable database. Sometime last year, Puppet added the ability to store facts into a Couch Database: http://www. Read More...

meetbsd 2010

published on
I love the BSD community. How cool is it to have developers and end users from the various BSD projects in one location, there to talk about the various cool projects and technical challenges that face us now. Cool I tell you! Chris, Corrigan and I went to Google in 2008 for MeetBSD. We had a great time, so when I saw the announcement earlier this year for MeetBSD 2010, I rallied the troops. Read More...

freebsd and multipath

published on
I didn’t find any blog posts of discussions on FreeBSD and multipath (for storage) that wasn’t a man page. That means it is up to me to write about it :) Hardware CPU Machine class: amd64 CPU Model: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz No. of Cores: 16 Memory Total real memory available: 65511 MB Logically used memory: 3945 MB Logically available memory: 61565 MB Storage The storage is a large ~90TB Enterprise class Fibre Channel array, a Data Direct Networks S2A9900. Read More...

stuff and things

published on
I don’t have a central theme with this post, but I wanted to at least do something (it has been a while). Packet Filter Based on Chris’s “Falling in love with pf(4)” google status, I decided to take the plunge and move off of ipfw(4) to pf(4). I’m not at the point where I could write my one filter, however, I do feel I at least understand what is happening here. Read More...

lagg freebsd

published on
I recently configured a NFS/Samba server with FreeBSD’s Link Aggregation protocol. Here is how I set it up. FreeBSD Configuration /boot/loader.conf I recommend adding the if_lagg_load=YES and kern.hz=“2000” to /boot/loader.conf. The OS will automatically load the lagg kernel module when your network configuration loads, I prefer to explicitly set it to load. ispfw_load="YES" kern.hz="2000" aio_load="YES" hw.igb.rxd=4096 hw.igb.txd=4096 if_lagg_load="YES" Since I am using the igb ethernet device (Intel 82575 and 82576 chipsets), I also set the max number of send and receive descriptors from the default from 256 to the maximum 4096. Read More...

freebsd 8 0 a great nas server

published on
I need to share this. When I google for “Samba performance”, I never see real numbers, real configuration files, or real hardware environments. All I read are anecdotal recollections, and that is not good enough. I like numbers, and I’ll let the numbers speak for themselves: > netstat -I em0 -w 1 input (em0) output packets errs bytes packets errs bytes colls 90166 0 98762637 95363 0 5332847 0 18131 0 24713156 20042 0 1123684 0 4 0 310 1 0 178 0 8 0 518 1 0 178 0 10153 0 10952920 10696 0 598129 0 92990 0 102837002 98476 0 5514994 0 92025 0 102680574 97277 0 5439496 0 92080 0 101799874 97403 0 5448637 0 75348 0 90861608 80972 0 4537737 0 90895 0 100323946 95781 0 5360948 0 89313 0 97371154 94364 0 5278618 0 81363 0 89229738 85861 0 4803589 0 2 0 126 3 0 286 0 I was so shocked that I had to use gstat and zpool iostat to verify the information: Read More...

why you should use disk labels

published on
I recently had a little problem with a new FreeBSD install, and it is one of those times were I sort of appreciate how FreeBSD assigns device handles, yet at the same time hate it :) The setup is this: The OS was installed on a mirrored hardware raid device (using the mpt(4) driver), and then I had a large RAID6 array attached via a FC controller (using the isp(4) driver). Read More...

64bit nvidia driver for freebsd

published on
I’ve always had a vested interest in the entire nvidia display driver for FreeBSD project, and I’ve pretty attached to the project. So much, that back in 2001 I started a little petition, got enough attention (and more importantly, a large list of people who signed my petition), and ever since 2002 FreeBSD users have been able to use high quality nvidia drivers. It wasn’t all me, whoever ran nvidia.netexplorer.org asked me to combine efforts, and I gave them my list, and they continued to market it and work with some folks at nvidia. Read More...

freebsd 8 0 is available

published on
So, it looks like FreeBSD 8.0 has been pre-released; the official date is going to be 11⁄25, as noted in src/UPDATING: Updating Information for FreeBSD current users This file is maintained and copyrighted by M. Warner Losh . See end of file for further details. For commonly done items, please see the COMMON ITEMS: section later in the file. Items affecting the ports and packages system can be found in /usr/ports/UPDATING. Read More...

puppetcamp09

published on
This was a very cool conference. I picked up a lot of useful information on both the open source tool, Puppet, and some ideas on infrastructure. What also made this conference unique, is how honest the Puppet team and community were about the projects strength and weaknesses. Those that have deployed Puppet on a larger scale (MessageOne and Google) seemed to go through the same iterations in attempting to scale out their Puppetmaster’s. Read More...

pc bsd 7 1 1

published on
PC-BSD KDE Desktop PC-BSD is a nice mesh between FreeBSD and a ready to use Desktop (which uses about 6GB of disk space). It is based on FreeBSD 7.2, so it has all the cool features of the latest release. Best of all, without ANY additional configuration, I was able to: Use the official FreeBSD nVidia driver for hardware acceleration Watch clips on YouTube (with flashplayer) Play back all sorts of media types like mp3’s, divx, mpegs,wmv,qt… Read More...

zfs updated in freebsd 7 2

published on
FreeBSD 7.x has been using version 6 of ZFS, and originally only 8.0 was going to have the newly updated ZFS version: 13. Last week the core team MFC’d (Merge From Current) the ZFS updates to 7.2, so I cvsup’d and re-build my server’s kernel and world ( with a simple “make buildworld && make buildkernel && make installworld && make installkernel ), rebooted, and now I have the latest ZFS version running: Read More...

using amazon s3 for backups

published on
I don’t have a backup system for home (which is where this site, and others are located), and I have generally relied on duplicating enough of my important stuff between friends and other computers. That, and I have a RAID5 setup for my large storage, and then home directories and website stuff is on a RAID1 ZFS volume. This doesn’t prevent accidental “oh-no”s, but it does protect me from some hardware failures. Read More...

samba 3028a vs 333 on freebsd 71

published on
!!! UPDATE on 12/29/2009!!! Since this blog post seems to get a good amount or hits from google, if you are reading this, please see my updated post: http://www.mywushublog.com/2009/12/freebsd-8-0-a-great-nas-server/ which has some additional information about FreeBSD 8.0 EOF Lately at work, I’ve been involved with a very large file system that is being export from Solaris 10/ZFS to windows and OS X users via Samba. Even with a very large Sun server (T5220) a lot of users are complaining about the slow performance of the system. Read More...

setting up my own openid server

published on
I’ve configured this blog to use my OpenID accounts. I have two (which totally goes against the single identity mindset of OpenID :) ) http://mcarlson.myopenid.com/ http://openid.evil-genius-network.com/mike The second one I just stood up today. I’m always concerned with who has my information, and if I can, I try to keep it all within the realm of my control. Also, the evil genius domain has absolutely no purpose besides a testing ground that I have no problems destroying :) Read More...

meetbsd day 2

published on
Thankfully I didn’t have to get up at the crack of dawn (7:30!) because it was scheduled to start at 11am. I was a little rushed for time since I had to drop Caralyne off at my parents in Knightsen (so I back-tracked a bit), and I left at 9am, so I barely made it in time for the ZFS talk. A Closer Look at the ZFS File System by Pawel Jukab Dawidek I’ve heard a great talk on ZFS from Bill Moore, one of the primary developers from Sun, but this talk was really cool because it didn’t just say what ZFS is from Sun’s marketting department’s point of view, but the technical details its mail peices and how it integrates in FreeBSD. Read More...

meetbsd day 1

published on
I spent the weekend at Google for MeetBSD in celebration of FreeBSD’s 15 birthday. I drove 70.2x4 miles, on a weekend, and for Saturday I got up slightly earlier than I would have for work. All worth it, this was the coolest mini-conference I’ve been to. Of course, the last conference I went to was BSDCon in 2003, and that was nice as well. The point is, I don’t get out all that much when it comes to conferences, I just make an exception for my favorite UNIX OS, FreeBSD. Read More...

freebsd 70 zfs and iscsi

published on
A friend of mine got my the coolest birthday present I think I’ve ever recieved, and that was a bunch of new and super kick ass hardware. This will soon replace my current server, which is in such bad shape it cannot compile java code, or perl from source. Before I replace it, I wanted to play around with ZFS that comes with FreeBSD 7.0. Here is a quick rundown of it all: Read More...