What I do....

It occurred to me that I never provided what I do for a living/hobby here.  Basically I'm a Sr. Perl Engineer on the Infrastructure team for Various, Inc (owns Friendfinder and related sites and is owned by Penthouse Media Group, Inc).  So my daily life includes these basic things:

•  trouble shoot production issues
• work on upgrading systems to be more efficient (logging/analysis/test new software)
• help other programmers with what happens in production (will it handle the load basically)
• or cleaning up what didn't handle the load that got pushed out

We have ~2000 servers in production (and growing).  We're working on being more SOA but it's taking time since we've never had a major revision of code. The many years of organic growth of code have left us with some interesting challenges though.

Things I work with every day:

• Perl
• Mod_Perl 2
• Apache2
• MySQL
• Squid
• Memcached
• Netapp

Things I work with on a regular basis:

• PHP
• Flash
• FMS
• Erlang
• JavaScript
• CSS
• C++

Nothing too out of the ordinary for a web engineer.  The main challenges of this job is trying to make things run better.  Historically we've just thrown hardware at our problems (and that does work to a point).

Basically when you work in an environment that hosts Perl, PHP, Java, and ASP.NET, FMS, Erlang all serving traffic and partner with CDNs for most of our static content there's a lot of things that can go wrong.  But even with optimizing video streaming and CDNs we still push 6-8Gb from one data center (peak of 14Gb at one point) and 3-5Gb from our other data center while maintaining 1M page views every 10 minutes on our larger site and averaging 12M banner impressions an hour.

Part of my responsibilities are:

• code release system
• code review
• service connectors
• session management
• making various systems play nice together

Mostly when you get sites to a level like this, you realize that while you can probably optimize your template engine a little more (we've done this afew times), data aggregation to render pages usually hurts you the most.  And that a lot of open source software has no idea how to scale.