Working Remote

2020-03-11

First, here is a link to a blog post that was created based on a talk Kat Dober and I did at our companies product offsite:

How to Manage Remote Engineering Teams

So, if you want a TL;DR there it is.

I have worked at New Relic for 3 1/2 years now. When I first interviewed and joined, the work culture was strongly based on co-locating engineers. Even the office setup was geared towards this environment as it highly encouraged spontaneous collaboration and flexible meeting spaces.

My team initially had everyone at the Portland office. We all collaborated with what we call “mob” programming (think or paired programming with more people). My boss at the time was ahead of the curve when it came to remote workers, so we hired someone who lived in Austin Texas. Once you hire one remote person, you all become remote for the sake of equity. As my current boss likes to say, “Thanks for keeping this team a remote first team!” even though most of us are based in the Portland area.

So, we slowly started to adjust the teams schedule to work from home once a week. This was at my bosses insistence, so we could all figure out how to work remotely and still mob. Then we hired another remote engineer, this one lives in Colorado.

Over time, the team adjusted to accommodate this. Some were easy and cool technically challenges. Others are still difficult to solve, because they are people and culture issues.

Tools

Here is the easy part, SOFTWARE!

First, there was slack calls and we took turns “driving” by stopping after our time was up, pushing code up, and the next person would take over sharing their screen, pull down the code, and continue working.

This added a lot of friction, and time, to what should be short intervals between drivers. Next, we tried a Slack based call and their now defunct desktop control tool. This worked “okay” if two people are sharing a common desktop. As soon as 3 or more engineers joining the call it broke down REALLY fast.

One of the smarter Engineers on the team (not me obviously) came up with the idea of a Docker container that we could ssh to, and it had a tmux session bound to a socket we all had permissions to read/write to.

The Mob Box

We call it the mob box, and it runs as a container in EC2 and it has:

  • Local users with no password but ssh public key auth
  • Users are part of a common group
  • /code/ is a mapped in docker volume, that is where we check out our projects from our github enterprise instance
  • Vim - the common editor between most engineers
  • emacs is aliased to vim… hah just kidding
  • common shell and vim dotfiles

The Dockerfile is this lengthy beast:

FROM ubuntu:artful as sshkeybuilder
RUN sed -i -E -e 's/(archive|security)\.ubuntu\.com/old-releases.ubuntu.com/g' /etc/apt/sources.list
RUN apt-get update && \
	apt-get -y install openssh-server

FROM ubuntu:artful

RUN sed -i -E -e 's/(archive|security)\.ubuntu\.com/old-releases.ubuntu.com/g' /etc/apt/sources.list
# Install prereqs
RUN apt-get update && \
  apt-get -y install \
    bash-completion \
    build-essential \
    g++ \
    gcc-6 \
    git \
    jq \
    libcurl4-openssl-dev \
    libgdbm-dev \
    libpq-dev \
    libreadline-dev \
    libssl-dev \
    make \
    nodejs \
    openssh-server \
    postgresql-client-9.6 \
    ruby-dev \
    wget \
    zlib1g-dev

RUN ssh-keygen -A
RUN apt-get -y install locales
RUN locale-gen "en_US.UTF-8"

# Configure ssh
COPY --from=sshkeybuilder --chown=root:root /etc/ssh/ssh_host* /etc/ssh/
RUN echo "Port 2222" >> /etc/ssh/sshd_config
RUN mkdir /run/sshd && chmod 755 /run/sshd

# Add docker prereqs
RUN apt-get -y install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

# Install docker apt key
RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

# Add docker repo
RUN add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

# Install docker
RUN apt-get update && apt-get -y install docker-ce

# Install docker-compose (update compose version as needed)
RUN curl -L https://github.com/docker/compose/releases/download/1.17.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
RUN chmod +x /usr/local/bin/docker-compose

# Install bash-completion for docker-compose (update compose version as needed)
RUN curl -L https://raw.githubusercontent.com/docker/compose/1.17.0/contrib/completion/bash/docker-compose -o /etc/bash_completion.d/docker-compose

# Install tmux-next and link it up
RUN add-apt-repository --yes --update ppa:pi-rho/dev && apt-get -y install tmux-next && ln -vs /usr/bin/tmux-next /usr/bin/tmux

# Install go, sudo, vim, etc
RUN apt-get update &&  \
	apt-get -y install \
	apt-file \
	golang-1.13-go \
	iputils-ping \
	iputils-tracepath \
	netcat \
	python-pygments \
	rsync \
	socat \
	sudo \
	vim

# Install ripgrep
ARG RG_VER="0.10.0"
RUN echo "Installing ripgrep ${RG_VER}" && \
  wget --progress=dot:mega https://github.com/BurntSushi/ripgrep/releases/download/${RG_VER}/ripgrep-${RG_VER}-x86_64-unknown-linux-musl.tar.gz && \
  tar zvxf ripgrep-${RG_VER}-x86_64-unknown-linux-musl.tar.gz && \
  cp -v ripgrep-${RG_VER}-x86_64-unknown-linux-musl/rg /usr/local/bin && \
  cp -v ripgrep-${RG_VER}-x86_64-unknown-linux-musl/complete/rg.bash /etc/bash_completion.d/rg && \
  rm -rfv ripgrep-${RG_VER}-x86_64-unknown-linux-musl.tar.gz ripgrep-${RG_VER}-x86_64-unknown-linux-musl && \
  echo "Done installing ripgrep ${RG_VER}"

# Install fd
ARG FD_VER="v7.3.0"
RUN  echo "Installing fd ${FD_VER}" && \
  wget --progress=dot:mega https://github.com/sharkdp/fd/releases/download/${FD_VER}/fd-${FD_VER}-x86_64-unknown-linux-musl.tar.gz && \
  tar zvxf fd-${FD_VER}-x86_64-unknown-linux-musl.tar.gz && \
  cp -v fd-${FD_VER}-x86_64-unknown-linux-musl/fd /usr/local/bin && \
  cp -v fd-${FD_VER}-x86_64-unknown-linux-musl/autocomplete/fd.bash-completion /etc/bash_completion.d/fd && \
  rm -rfv fd-${FD_VER}-x86_64-unknown-linux-musl.tar.gz fd-${FD_VER}-x86_64-unknown-linux-musl && \
  echo "Done installing fd ${FD_VER}"

# Install bat
ARG BAT_VER="v0.10.0"
RUN  echo "Installing bat ${BAT_VER}" && \
  wget --progress=dot:mega https://github.com/sharkdp/bat/releases/download/${BAT_VER}/bat-${BAT_VER}-x86_64-unknown-linux-musl.tar.gz && \
  tar zvxf bat-${BAT_VER}-x86_64-unknown-linux-musl.tar.gz && \
  cp -v bat-${BAT_VER}-x86_64-unknown-linux-musl/bat /usr/local/bin && \
  rm -rfv bat-${BAT_VER}-x86_64-unknown-linux-musl.tar.gz bat-${BAT_VER}-x86_64-unknown-linux-musl && \
  echo "Done installing bat ${BAT_VER}"


# Aadd lolcat and symlink
COPY binaries/lolcat /usr/local/bin/lolcat
COPY binaries/censor /usr/local/bin/censor
RUN ln -s /usr/local/bin/lolcat /usr/local/bin/lolcat-c

# Add shared bash profile/settings
COPY conf-files/bash_profile /etc/profile.d/mob_common_settings.sh
COPY conf-files/bash.prompt /etc/profile.d/mob_prompt_settings.sh

# Add tmux config, tmux-next is because sometimes we run tmux-next from a ppa.
COPY conf-files/tmux.conf /etc/tmux.conf
COPY conf-files/tmux.conf /etc/tmux-next.conf

# Copy ssh public keys, gitconfig for each user
COPY user-data/  /home/

RUN chmod 755 /home/*
RUN chmod 700 /home/*/.ssh
RUN chmod 600 /home/*/.ssh/authorized_keys

# Add vimrc to get vimgo
COPY conf-files/vimrc /etc/vim/vimrc.local

# Add users and groups
RUN \
  addgroup team && \
  addgroup --gid 233 hostdocker && \
  adduser --disabled-password --ingroup team --gecos "" --shell /bin/bash alice && passwd -u alice && chown -R alice /home/alice && chmod 600 /home/alice/.ssh/authorized_keys && \
  adduser --disabled-password --ingroup team --gecos "" --shell /bin/bash bob && passwd -u bob && chown -R bob /home/bob && chmod 600 /home/bob/.ssh/authorized_keys &&  \
  adduser --disabled-password --ingroup team --gecos "" --shell /bin/bash charlie && passwd -u charlie && chown -R charlie /home/charlie && chmod 600 /home/charlie/.ssh/authorized_keys && \
  usermod -a -G hostdocker alice && \
  usermod -a -G hostdocker bob && \
  usermod -a -G hostdocker charlie

# Configure sudoers
RUN \
  echo 'alice  ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers && \
  echo 'bob  ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers && \
  echo 'charlie  ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers


# Get vimplug and vimgo
RUN curl -fLo /usr/share/vim/vim80/plug.vim --create-dirs https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim
RUN \
  mkdir -p /home/alice/.vim/autoload && \
  mkdir -p /home/charlie/.vim/autoload && \
  mkdir -p /home/bob/.vim/autoload

RUN \
  ln -s /usr/share/vim/vim80/plug.vim /home/alice/.vim/autoload/. && \
  ln -s /usr/share/vim/vim80/plug.vim /home/charlie/.vim/autoload/. && \
  ln -s /usr/share/vim/vim80/plug.vim /home/bob/.vim/autoload/.

RUN mkdir /etc/vim/plugged
RUN git clone https://github.com/fatih/vim-go.git /etc/vim/plugged/vim-go

# Add the timer
COPY scripts/timer.sh /usr/local/bin/timer.sh
RUN chmod 755 /usr/local/bin/timer.sh

# Mount code volume
VOLUME /code

#Ensure perms are correct on code directory and make some dirs
RUN \
  chmod 2775 /code && \
  chgrp team /code && \
  mkdir /code/tmux

# Run the thing
EXPOSE 2222
CMD /usr/sbin/sshd -D -e

Our docker project has other files that are copied in, and these are really dependent on the teams preferences so I’m not going to list them all out.

I will note that we found a few things helpful to have shared between everyone:

  • Basic vimrc and plugin setup that focuses on readability and simplicity. A lot of people have come and gone on the team; not everyone is fluent with vim or even comfortable. Keeping it simple helps people learn quickly
  • shell aliases that quickly setup a rails server, pry console and run the test suite quickly
  • a colorful prompt that is easy to notice when someone else presses enter (we use rainbow-bash-prompt)

I like this approach, and I found myself ssh-ing to our mob box and developing in there before I develop locally.

The biggest drawback is tmux’s tendency to shrink the window down to the smallest client attaching. For instance, if two of us have our mob ssh window fill screen on a 2k or 4k monitor, its great. However, when someone joins the session on their 13" macbook pro and only giving their terminal session a paltry 40x140, it quickly feels like you’re writing code on a postage stamp.

The second drawback is the learning curve of vim and tmux all at once. If an engineer on the team comes from more of an aws tools and sublime text editor background, this can be very daunting.

VSCode

As a honest to goodness UNIX fan, someone who has always used FreeBSD or Linux as a desktop and most definitely as a server, I never though Microsoft would have written my 2nd favorite developing application…

Visual Studio Code is incredible, and if you haven’t at least tried it by now you really should.

One of its greatest features for remote collaboration is it’s live sharing extension

You can use either your Live account or public github account to sign in, once you do, you can “share” your current project. You can even share your terminal, have a chat session started, and even voice chat now.

This has allowed us to invite people who might not be comfortable with an editor and terminal muxer like vim and tmux, and we can still hack away together.

I would say VSCode is a bit nicer in some ways:

  • Application window size doesn’t matter! I can have it full screen at 3840x2160 and someone else can have it open at 1920x1080, and neither of us are limited
  • You can “follow” another user who has joined the session. This means you can watch and learn what someone else is working, or, work on a different section of your codebase
  • A shared terminal lets us do things like debug with pry in a rails application together

The short comings so far are:

  • I dislike how small the shared terminal window tends to be, and I hate resizing. I want something like tmux’s zoom hotkey
  • Sometimes you need to bump around a bash shell to look at a system or figure something our. Or, sometimes find, awk and sed are really really useful in bulk changes
  • It is not as fully featured as Jetbrains Goland (for go) or Rubymine (for ruby), if I’m doing a solo go project I tend to stick with Goland or just vim itself

We have found ourselves gravitating towards VSCode for certain tasks and I like it a lot.

Culture

A bigger challenge for remote teams is an embedded culture of colocation. It’s really easy to just walk up to someone and ask a question, and you’d be surprised how hard that is once someone is behind slack or a timezone change.

As a remote first team, especially those of us who still go into the Portland office, we strive to do out best to relay water cooler conversations back to the entire team. If two team members find themselves discussing a project idea or change of direction, we’ll stop where we are and start a Slack call for the rest of the team members who are not physically present.

A tricker and worse bit of realizing if your work’s culture is remote friendly is when a local office mixes up division wide communication and not just locale information. I can’t really give specifics, however, one of my co-workers who is in another state finds that division level information is often locked away in a Portland only channel.

You

Quality time with Levi

You’re a big part of working remote. I personally really enjoy working from home for these reasons:

  • I don’t have to interact with people when I don’t want to. This is huge for me
  • For a very long time, I have exercised. I prefer to continue this over eating out. So while I’m at home, I use my home gym and its awesome
  • Break time can include one of the following:
    • Pet the cats
    • play with the dogs
    • play a game on my console or desktop
  • Household chores

These are earned and with a balance though. While I make take a break from work to stretch, play a quick game, or whatever, I communicate with my team when I’m unavailable and then available again. Remote employees have to sometimes over communicate, because it often feels like they are gone longer then they are. I also tend to work earlier (because I am not commuting in) and drop off for the day a little earlier. This is all within our teams understanding and acceptance.

Equipment and Office Space

Me on the left, Caralyne on the right

Avoid using your comfy recliner or guest room couch for an office. Have a desk, a chair you can comfortably sit in for extended periods of time (or none, if a standing desk is your thing).

Have a second (or third monitor) for increased screen space. Some places will let you expense one, but even if it costs $100 for a second 1080p display its really worth it.

Get a decent headset and mic. The times I’ve wanted to drop off a call because someone forgot their headset and used really bad earbuds with a built in mic have been many. Not to mention they’re in a coffee shop as well.

Speaking of coffee shops… if you plan on talking to co-workers, understand that public spaces are loud. If I do work from a public area, it is when I have no plans on hopping on a call with the team. I will communicate by text.

Dedicate an area at home you can call your office. Hold hours. Let your living space partners or family know what is acceptable behavior while you are “at work”.

The End

With a lot of people and jobs forced to work remote at this time, we can all learn a lot about remote working and how to adjust. Please give people time and be patient.