Editors Note: Lenny Rachitsky is the Co-Founder and CEO of Localmind and one of 60 Canadian startups invited to participate in Day 1 of the GROW Conference being held Aug 17th in Vancouver, Canada.

If you’ve ever received an email from Dan Martell (founder of the one and only Maple Butter), you’ve probably seen the following quote in his email signature, referring to the importance of gathering metrics around your product:

“In God we trust, all others bring data.” — W. Edwards Deming

Easier said than done. Gathering data is a bitch. You have to figure out all of the metrics you’ll want to look at, build the infrastructure to capture those metrics, present those metrics in an understandable way, and make sure those metrics are accurate at all times. This takes time away from focusing on your core product, your never ending list of features and fixes, and everything else your startup needs to do. In spite of all this, well thought out reliable metrics are pretty much critical to your startup’s success. I want to share how we gather metrics at Localmind, where we try to find that balance between getting the metrics we need while doing as little as possible.

Pick your KPI’s

If everything is important, nothing is important. Pick 3-5 key performance indicators who’s ebb and flow are directly correlated to the success of your company. Don’t worry too much about getting this right early on, you can always adjust later on. We’ve changed our KPI’s a number of times over the past few months as our product evolves. Our KPI’s today are:

1. Daily active users
2. Monthly active users
3. New users per day
4. Questions asked per day
5. Invites sent per day

For inspiration when coming up with your own metrics, Fred Wilson has a recent post about common metrics for mobile apps, Dave McClure has his infamous Startup Metrics for Pirates, Dan Martell has some great stuff on cohort based metrics. My advice is to think which action in your product, if it went 100x higher, would mean the most dramatic increase in value to your company. That should be your #1 KPI.

Now it’s just a matter of gathering those metrics.

Backend metrics (with Google App Engine)

Let’s look at how we can track the last three metrics listed above relatively easily. Since these are very unique to every service, it’s nearly impossible to avoid having to find a way to track these metrics yourself. How do you do this easily and efficiently? Often I’ve seen people run a query on the database, counting the number of new user rows in the past 24 hours, each time we want to know how many new users they’ve seen today (which would be extra painful on a NoSQL database like App Engine’s). Instead, what we do is keep a single running count of total users (and questions asked, and invites sent out, etc.) on the system since day one.

Every newly created user increments this count (and we have some logic that buffers writes to the DB to avoid contention). Then every hour, and every end-of-day, we take a snapshot of every one of our running counts. The snapshot records the value of each of those metrics that that moment in time. This way, we have a single lookup that tells us how many users/questions/invites we have had every day and every hour historically. We do this by creating three tables in the database:

1. One to track the current running count for each metric we are tracking

DB table of some of our metric

2. One where we store hourly snapshots of each metric

DB table showing a row for snaphots of the past few hours

3. One where we store daily snapshots of each metric

DB table showing a row for snahspts of the past few days

In the screenshots above, instead of having to do a row count on how many checkins we’ve seen over the past day, we simply look at the last entry daily snapshot table and grab that number.

Active users

Since our main product is an iPhone app, we can take advantage of existing services that integrate into our app and track all kinds of delicious data with very little effort. We’ve been using Flurry for a few months now, and just rolled in Localytics in our latest build. Flurry give us the two remaining KPI’s, the number of active monthly and daily users:

Flurry is great, but it isn’t that great. It’s missing some key things like cohort based reporting, which is why we’re testing out Localytics and others. If you have any recommendations on this, I’d love to hear them.

Visualizing the data

We’ve got a lot of great data, now how do we make sense of it? We’ve been using a service called Geckoboard with great success. Geckoboard is basically a Dashboard-as-a-Service, which allows you to suck in all kinds of data from various places, and present it in a customizable dashboard. Here is our main dashboard:

Unfortunately I’ve had to redact some of this, but you can get a sense of how much data you can present in a very small space. Each of these graphs does a simple hit to our backend system, asking for the past 30 days of snapshot metrics (which we talked about above), and is refreshed every few minutes automatically. You have to create a custom API which returns the data in the required format, but if you built a generic way return data for a given time period, each new graph becomes trivial. We use Geckoboard to show us both our own custom metrics that we track ourselves, plus our Flurry stats (which we suck in using their API), our web traffic (Geckoboard integrates with Google Analytics automatically), our invite conversation rate, and every single other metric that matters to us.

Using the data

Visualizing the data is one thing, but what do you do with all of these fancy graphs and metrics? You hold yourself and your team accountable, that’s what. Every action you take, every feature you release, every decision you make should be tied to a subset of metrics. More specifically, every work effort should be tied to getting one or more metrics moving in a given direction (e.g. more users, less churn, more daily actives, more invites sent out, etc.). It’s too easy to get lost in the forrest of interesting data, when what really matters is getting through the forrest (to grandmothers house?).

Close

At a certain point, metrics become so critical to the value of your company that you probably want to bring them in-house, and not rely on 3rd parties to tell you how well you are doing. My personal philosophy in tech startups is to do as little work as possible until you’ve hit product/market fit, and to spend all your time on the things that really bring value and set you apart. In the case of gathering metrics, you can’t avoid the need to have metrics in place. The best you can do is find a way to capture and present these metrics with as little work as possible. I hope this post can help you get started along that road.

If you’d like to know more, or have recommendations tools I haven’t mentioned, please leave a comment below!