Services Are Hard

It can be tough running software as a service. Apple Maps has had widespread downtime reported today leading to lots of people poking fun at them and their status as second to Google Maps (dozens of people were affected jokes). The other day I was driving with Maps running when the GPS lost where I was and the arrow marker started drifting off-road randomly around town. I snapped a photo at apparently the perfect moment as the marker drifted by Lost Ln. Not sure if that was a precursor to this outage, but the photo and timing seem appropriate. As funny as that is somewhere there was a team of developers and server admins freaking out that things were going […]

OpenVPN Fails to Start on Boot – CentOS 7

This all started one chilly January morning when I upgraded my CentOS 7 virtual machine. My OpenVPN connection would fail to start because the /run/openvpn/ovpn-client.status file was not there nor the /run/openvpn directory. It took a lot of investigative work but what we found was the /usr/lib/tmpfiles.d/openvpn.conf file had this in it: d /run/openvpn-client 0710 root root – d /run/openvpn-server 0710 root root – This temp file was making the directories in the /run folder that were needed on start for OpenVPN to work. As you can see the first line is creating the /run/openvpn-client folder. This would be all well and good but the openvpn.service on startup is looking for /run/openvpn without the “-client” and that is no good. […]

Test and deploy HAProxy Config

We faced a problem that we thought could use an automated strategy; our HAProxy configuration files were getting larger and more complicated and more prone to a small error resulting in the service not restarting appropriately. We currently use Buddy.Works to manage some of our code deployments and we’ve found a relatively simple way to accomplish this. We created a new repository, commited our base HAProxy configuration, setup our pipeline to grab the official haproxy docker instance and run the following: groupadd haproxy && useradd -g haproxy haproxy mkdir -p /etc/ssl/private/ cp dummy.pem /etc/ssl/private/dummy.pem cp haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg This will create and add the appropriate user/group that our HAProxy runs under, copies over a dummy certificate to […]

Stopping Exchange 2016 Same Domain Spam Spoofing

We recently converted over to using Exchange 2016 for our internal email hosting and we were immediately buried in spam. Initially we enabled the exchange built-in spam protection and it just wasn’t cutting it. We expanded out and got ourselves BitDefender Exchange Protection which promptly dropped our spam rate almost overnight. However, we were still getting emails from our own domain The best places we found to combat this involved removing the permission: ms-exch-smtp-accept-authoritative-domain-sender in the active directory for the receive connectors. This, thankfully, allowed BitDefender to at least classify them as spam, but we were still able to be spoofed. A simple telnet to our server like so: >telnet <server_ip_here> 25 >EHLO >MAIL FROM: test@<domain_here> >RCPT TO: someone@<domain_here> […]