Bid Management Tools

Know the difference between a Reporting Squirrel and a Analysis Ninja?

One is in the business of providing data.

One is in the business of providing, to use a old fashioned word, information.

This one of the core reasons why most dashboards are “crappy”, i.e. they are data pukes that provide little in terms of context and even less in terms of actionable value.

Here are some examples of sub optimal dashboards, sub optimal in my mind from a actionable perspective. . . .

Perhaps the most common type is above. Lots of data, even drill downs included, but you can’t look at it and go: “Wow we need to do . . . “. No sirrie bob you can’t.

I wanted to point the above out purely because of a common feature of 80% of Web Analytics Dashboards, in excel with a billion tabs to look through. This is not a dashboard, it is the result of a massive sum of money paid to a Consultant who is trying to impress you with his / her excel skills – without actually telling you anything.

You are walking down the street. You look at someone from behind and you think “hmmm she’s / he’s pretty”. So you speed up and overtake them and in the process you sneak a glance at them (yes you are married but looking is still ok , and you are hugely disappointed. Not pretty. That’s the dashboard above. Very sexy and Web 2.0′fied and a ton of data there, but a lot less actionable than you might have hoped.

Why is this so? All the above efforts are well intentioned, took lots of honest work and probably took months to put together. So why?

Here are some hidden (corrosive) reasons why most dashboards tend to stink when it comes to helping the Executive make any decisions:

1. They leave the interpretation to the Executive (/ customer / requestor / other Squirrels). This is a fatal flaw because most dashboards are highly aggregated views of any KPI and are missing all the nuance and analysis (that only you as Ms. Ninja have, and you don’t go with dashboard).
2. Most Executives actually want insights / action recommendations but they don’t trust the Squirrels / Ninjas / VP’s / Data Providers. So they ask for numbers. We dutifully cram as many of them on to a A4 size paper in 3 size font and send it along with a magnifying glass.
3. Most Squirrels / Ninjas live in a silo. Going out to collect enough tribal knowledge to actually know what is going on to then make recommendations from the data is not something that we do, nor are we encouraged by our Executives or our organization structures. This incentivizes data pukeing.
4. Often dashboard creators tend to be “outsiders” (Consultants, Experts etc) and they often don’t have deep practitioner experience that would allow them to understand the human / “below the surface” issues like the above three. That leads non-Practitioners to make the common mistakes like creating the above three dashboards.

If you want your Executives / Customers to take action, you have to give them information and not data. It takes effort to get there, it will take all your charms (though no violation of any HR intimacy policies), and it will take some time.

Step one as always is to become aware of the above three problems.

Step two is to get a possible solution from the Occam’s Razor blog.

My attempt at solving this problem was to try and attack it from a human psychology perspective: How can I create a “dashboard” that will incent the right behavior from the Squirrels / Ninjas while giving Executives the information they need to make decisions (rather than engaging in a bitchfest which is the typical outcome).

Recommendation #1 was to move to a Critical Few philosophy for Executive reporting: Only report the three or five (at most!) metrics that define success for the whole business. Kill all the ancillary metrics that were nice to know (and my kill I mean let lower levels worry about it).

Recommendation #2 was my humble, admittedly ugly, attempt at a “Action Dashboard”:

4Q. (Sorry Jonathan!

Each quadrant representing a solution to a human problem that lead to crappy dashboards.
(Apologies for having to redact some of the data above, to protect the innocent.)

Let me walk you through it.

First very up top a clear identification of what the Critical Few metric was, who was responsible for that metric from a business perspective (translate into “head on the line”) and who was responsible for the analysis.

Also note the little red dot. That here indicated trouble. It can have two other colors, yellow for don’t fire anyone yet but get ready and green for send someone a big hug and a box of chocolates. Next. . . .

The first quadrant (the graphic) shows the trend for the metric. Ideally segmented (as is the case here, cart abandonment is illustrated for four key customer segments).

This quadrant is to satiate Executive curiosity that you know what you are doing, it will be glossed over (and that’s good!).

The second quadrant (Key Trends & Insights) is to add value by interpreting the trends and adding context. It says there that some things are up or down (in english , and it also warns which data might be bad etc. You are starting to do your job here.

This quadrant is the one that Executives will read a lot initially, over time they will gain confidence in you, they will love that you share context (hello Ninja!), over time they will gloss over it (a good thing).

The third quadrant, clockwise, (Actions / Steps To Take) is force the shy Web Analyst to get out and talk to Marketers, Website Owners, VP’s, Whomever it takes to get all the tribal knowledge, identify root cause for the trends in the metric and recommend solid action to take. The Analyst will rarely be able to do this by themselves. It will require human contact with others, it will require conversations, it will mean identifying solutions collaboratively. It is a fantastic opportunity to become smart about the business.

This quadrant is key to driving action. No longer do you leave things to interpretation or let’s blame people etc. You are recommending what actually needs to get done. Your Executives will kiss you and over time this is the only quadrant they’ll read. It will also mean that monthly meetings will move from bitch fests to deciding who does what. Amen!

The fourth quadrant, (Impact on Company/Customer) exists in case it is not clear to the Executives why they need to take action (listen to poor old you the lowly Analyst). I feel it is the key thing missing from any dashboard, they are normally missing the kick in the rear end and this quadrant delivers it. It is the answer to this question: “As a result of this trend (up or down) what was the impact on the company and its customers”. It also forces you, Marketer / Analyst, to do hard work to estimate the impact and put it on paper.

This is the killer quadrant, if nothing else drives action this will, knowing exactly how much money was lost, how many customers were pissed, how much opportunity was wasted. Now when they ignore you they do that at their own peril and with their butt on the line. Trust me action you recommend will be taken.

See how simple it is?

You fix the human problems, you address the flaws in the system today and you actually become much smarter about the whole business (thanks to q3 and q4).

Win – win – win.

Over time you’ll gain a lot more trust from your Executives and all the crappy dashboards can die and be replaced with one that looks like this one. . . .

Now you are asking your Executives to simply layer their own judgment on the recommendations and help the company take action. Who needs to see the numbers? They pay you and I to deliver actionable insights.

I stress that it won’t happen overnight, but shoot for that nirvana state.

May the force be with you.

Ok now your turn. Care to share your own learnings and battle scars? Your success stories? Perhaps critique my “Action Dashboard” (sorry could not think of a better name, do you have suggestions?). Your perspectives are most welcome and would be greatly appreciated. Thank you.

PS:
Couple other related posts you might find interesting:

• Five Rules for High Impact Web Analytics Dashboards
• Web Analysis: In-house or Out-sourced or Something Else?
• How Should Web Analysts Spend Their Day?
• Seven Steps to Creating a Data Driven Decision Making Culture.
• Six Data Visualizations That Rock!

We have more web metrics and data than there are stars in the universe (slight exaggeration!).

Yet we stink at informing decisions. Our reports are ignored. Sites & online marketing continue to suck.

A large part of the reason is that a large part of our job seems to consist of glorified data puking, hoping someone will be impressed. After all there is so much data in those reports!! #fail

This blog post encourages you see the forest, the much hyped big picture, and shares a framework that will help you ensure that every single moment of your day is spent on activity that will be:

1. of value to your organization, hence appreciated and acted upon

2. has a clear line of sight to the one thing that matters: profit

If you don’t want your professional life to be frittered away then please come along this short journey.

First some context…

If you have seen one of my keynotes recently then you have heard my near evangelical fervor when it comes to trying to convince you to compute Economic Value.

If you have Web Analytics 2.0 then you already know who much attention is paid to this concept in the book (jump to page 159 for how to compute it for your website).

The reason for this emphasis is to help fix our miserable failure at at creating data driven organizations.

To steal your energy away from being just in the report / data production business.

To encourage you to do better than spend a lifetime implementing analytics tools, building data warehouses, chasing the next shiny object.

My recommendation has been:

That last one will get any organization to sit up and pay attention.

Why?

Because for the first time in their young and passionate life they’ll see the complete value your website is adding to the business. And because my dear it will be a huge number that no one can ignore! You are going to tie your work to the bottom line!

Revenue = Good. Economic Value = God! [Also slight exaggeration ]

Professor Ken Wong’s Magic Potion

Prof. Wong is the award winning Commerce ‘77 Teaching Fellow in Marketing at Queen’s School of Business (and an awesome speaker, you should hire him for your next event!).

He took the stage after my talk and said, I am paraphrasing here, “Avinash did not go far enough in his keynote. Economic value is important but the only thing that matters is Profit!”

That was awesome!

One of Prof. Wong’s key points was how the success of our work, as Marketers, is measured based on a lot of things but not often enough based on perhaps the most important metric of them all: Net Income.

Prof. Wong covered a lot of key points (as a MBA with a minor in Marketing I wanted to take off my clothes and jump for joy when he said the 4P’s of Marketing are killing Marketing!).

I wanted to share two of his slides that left a lasting impression on me.

They are particularly applicable in the web analytics context. In sharing my interpretation of them my hope is it will change a little bit how you think about your work and success.

The very first slide, “Profit: The Ultimate Client Need”, shares the key elements that need to function for the outcome (ROI) that causes companies to remain in business.

My interpretative points.

Net Income is driven by two important variables:

Unit Margins (how much you make on each X you sell or Y service you provide)

Unit Volumes (how many of X or Y you sell)

Margin times Volume gives you the golden metric Net Income!

[Keep this formula in mind, your life should be revolving around it else you are wasting everyone's time.]

Peel the onion back one more.

Unit Margins is in turn driven by two more variables:

Price (how much you charge for X product or Y service)

Cost (how much it costs you to make X or provide Y)

Price minus Cost equals Unit Margins.

Get it?

So if you want to have very high Margins you have two variables you can control. You can charge lots for your product or service (think of a Vertu phone).

You can also make it at the cheapest possible cost (no phone costs $100k, you make it for $300 and sell it for $100k).

You can of course also charge lots and lots and it costs you a lot to produce (think of a Tesla car). But give some thought to how you’ll stay in business.

Continuing the onion peeling…

Unit Volumes, our other variable to have high Net Income, is driven by two variables:

Market Share (is your share 90% or 5%?)

Market Size (is that share of a market the size of Maldives or China?)

Both share and size are important.

You’ll sell lots of X or Y if you have a high market share and the limit you’ll hit is the size of the market (you can then play in the current size or grow the pie).

Line of Sight.

Having a clear line of sight means that you are able to map every metric you report on (or better still torture with segmented analysis to find insights) every single day directly to the strategic objective of the company.

Prof. Wong is suggesting, rightly so, that that strategic objective is Net Income.

And you have only one of four things that you’ll move through actions your company takes: Price. Cost. Market Share. Market Size.

Here’s my crystallizing question for you. . . .

When you report the metric Page Views Per Visit which of the four are you solving for?

How about with Bounce Rate? Or Time on Site? Or % of New Visits? Or Visitor Loyalty? Or…..

Is there a direct line of sight between what you as a Marketer are being incented on, or you as an Analyst are spending time analyzing?

If not, are you surprised that no one loves you? Sorry… I mean… no one loves your work?

Here is a simple exercise you could go through: Pick out all the metrics you are reporting today (on your dashboards and top reports). Try to put them into one of the four important buckets from Prof. Wong’s slide.

The clear line of sight exercise. . . .

Were you able to cleanly bucket all metrics you currently report? Time on Site and Conversion Rate and Task Completion Rate and % Internal Site Search Exits and Cart Abandonment Rate and % of the Page Scrolled and % of Visitors Refreshing Pages and all the other sweet things.

Some of the metrics in the above paragraph are complete crap, you are wasting your time and everyone else’s time with them. And you’ll now discover that very quickly because you won’t have a place where you can bucket them.

Other metrics will make you think harder. Where do you bucket Conversion Rate? Are you impacting Price or Cost?

What about Customer Satisfaction? Or Page Rank!

Not every metric will map cleanly, and that is ok. I had to think really really hard to bucket each of my metric in the above picture. Some of the metrics were controversial. But bucket I did.

If it turns out your web metric has no line of site then it might be time to kill.

If the work you do can’t be mapped into Price, Cost, Market Share or Market Size then why are you doing it?

Before you dip your hands into Omniture or WebTrends or Surfaid, , answer that question.

I know it seems like a lot of work for a “lowly” Analyst to do. It is. But without it there is little hope for your personal success (promotions / bonuses) or your company’s success (higher Net Income).

“What Matters Most” Fishbone Analysis

As you look at the picture above it is amply clear that the metrics I have chosen in each of the four buckets are perhaps unique to me/my business.

The reason is simple… they are a reflection of the strategy my company is currently executing, i.e. our “world domination via an effective data driven online marketing plan”.

This simple truth, that metrics should reflect current business strategy, is the reason I loved another slide from Prof. Wong’s presentation.

It leveraged the same framework, but added “what matters most”. . .

[Click on the image above for a higher resolution version.]

The focus is still on Net Income driven by, hopefully, improved Margins and Volume which in turn are driven by much beloved 4 levers of Price, Cost, Share and Size.

What is awesome about the “fish bone” above is that it drills down to the 14 specific strategies that most businesses will use to become great (or simply survive).

You Ms. Web Analyst now have a framework you can take to your Marketing Directors and CMO’s to discuss which of the 14 strategies they are currently executing to drive the 4 beloved levers.

Ask any Web Analytics “Guru” or “Professional Speaker” or “I am so important you are paying me $5,000 an hour to give you generic advice Consultant” and they will always tell you that all good journeys in web analytics start with asking your bosses this question: What are the goals of the organization?

The advice is sound (and well worth $5k/hr). The problem is that we never get an answer from the customers of our data / our management. You are $5k x 8 hrs short and still none the wiser.

Get off the slow train to nowhere…. You now have a new BFF: Prof. Wong’s “What Matters Most” slide!

Don’t ask the generic “What are the goals” question. Ask “Of these 14 specific strategies which are we currently executing”.

Once they tell you which ones (be patient, it might shock them that you are giving them something tough and specific to think about), you’ll be in business.

The 5 strategies they pick from the right-most column will help guide you in terms of picking the right Key Performance Indicators / Web Success Metrics for your business.

And you know why a win now is guaranteed?

Because each metric you identify starts with a specific business strategy which has a direct line of sight to the 4 beloved levers which will have a impact on Net Income!!!

Minorly orgasmic right? [Trust me, you do this and you'll agree. ]

Summary:

Recommendation #1: The Web Analytics Maturity Mandate!

For far too long we have been like toddlers… bumping into things, having a limited vision, working just what we know (which is little).

What I love about this approach is that it forces us to grow up. It forces us to understand what we are solving for: Net Income. It forces us to have a line of sight between our work and the ultimate goal: Net Income. It forces us to not live in our dungeon but rather take a well defined framework to enable the discussion that will yield wins all around.

No lip service to how important process is. This blog post shares what you specifically must do to succeed!

Recommendation #2: Win With Web Metrics: Steps

Here are the specific steps I recommend you follow for optimal execution of the recommendations.

A simple but effective 7 step process.

Good luck.

Ok now it’s your turn.

Do you agree that a focus on Net Income and a focus on “what matters most” is key to success in web analytics? Can Web Analytics tie the work they do, the metrics they report, into Price, Volume, Market Share & Market Size? Or is our work simply not that important? In your job today how do you ensure line of site? Will you change anything based on the recommendations from Prof. Wong?

Please share your feedback / critique / ideas.

Thanks.

[UPDATE]

Zach Olsen, who blogs at By Data Be Driven, has taken the Clear Line of Sight framework outlined in this post and applied it to a medium sized eCommerce website. It is so wonderful, take a look:

[Click on the image above for a higher resolution version.]

Zach’s effort is awesome for these key reasons:

• Really clear line of sight from Business Objective to Net Income.
• Clean flow from What Matters Most to 4 beloved levers (Price, Cost, Share, Size).
• (This one I love the most…) Identifying of Targets for each metric! You can’t be serious about Web Analytics without doing this!

I hope you are as impressed by Zach’s effort as I was.

He has also done something sweet for all of us… he has created a excel spreadsheet that you can download and customize for yourself, and hence get a jumpstart! You can download it at this blog, bottom of this post: Web Analytics Framework Example. Please download it!

My thanks to Zach for his effort and for his permission to share it here.

[/UPDATE]

PS:
Couple other related posts you might find interesting:

• Six Web Metrics / Key Performance Indicators To Die For
• Analyze This: 5 Rules For Awesome Impromptu Web Analysis
• Actively Avoid Insights: 4 Useful KPI Measurement Techniques
• The “Action Dashboard” (An Alternative To Crappy Dashboards)
• Barriers To An Effective Web Measurement Strategy [+ Solutions!]

Marketing executives are under increasing pressure from CEOs to show a return on investment for their programs, but many are struggling with complex processes, technological difficulties and internal resistance to measurement systems, according to a report from The Conference Board.

Because measuring marketing return on investment (MROI) is still relatively new, many executives say they lack the technological and institutional tools necessary to measure their programs, the study found.

Lack of resources, lack of connection with performance objectives and inadequate focus are some of the primary sources of frustration. Major barriers to implementing MROI programs – largely related to issues of business infrastructure – include problems with data availability or integrity (47%), technology/infrastructure (41%), resource dedication (39%), and methodology/know how (22%).

Though the report finds that the top driver of success is a strong commitment from leadership, many marketing execs face considerable internal resistance to MROI, while others report their organization lacks sufficient commitment to allocate the budget, time, and resources required to measure MROI.

ROI is a Slow Go

Half of the companies surveyed have been measuring MROI for less than two years, while more than one-third still report making no efforts to measure MROI at all, the study found. Among the companies that haveimplemented programs, none have yet achieved their goals in measuring ROI. Only one-quarter report making “good” progress, the survey found. The majority said they found the process more difficult than anticipated.

Additional survey findings:

• The most frequently used metrics for measuring MROI include customer loyalty/satisfaction, customer retention, market share, marketing spending, revenue, web page views and profits.
• Technology is key to measuring ROI effectively. More than half (55%) said that technological/infrastructural deficiencies have had “a lot” or a “significant” effect on their efforts.
• 73% of companies that have been measuring MROI for more than three years say they are making good progress, compared with 4% of those who have been at it for less than two years.
• 91% of companies who report “good progress,” and 67% of companies who report “some” progress, have incorporated marketing ROI into their performance objectives. More than half of those who report “good” progress have also put in place recognition programs for marketing ROI.
• The amount of time spent on MROI activities is a greater predictor of success than time spent in meetings.
• B2B companies lag behind B2B/B2C combination companies in measuring MROI. B2B companies do not report as much progress, nor have they been working at it as long.
• The most important competencies cited for success with MROI are decision-making, analytical thinking, collaboration, openness to change, and goal orientation.
• Those making the most progress toward measuring MROI say they have the necessary infrastructure/platform, data availability/integrity, and software/tools in place.

“In the past, marketing awareness and brand-building activities were enough to define marketing’s mission and role in a company, and to justify its budget,” said Lorrie Foster, VP of Councils and Research Working Groups at The Conference Board. “But the focus of marketing has evolved toward more strategic, value-added activities that can be quantified and linked to corporate goals. New approaches, methodologies and tools, and technologies are making it possible to link marketing investments directly to revenues and profits, holding marketing executives accountable for achieving expected results.”

Although the inputs and expenses associated with marketing can easily be measured on a monthly or quarterly basis, the results of a successful marketing effort — enhanced brand recognition and reputation, customer loyalty, improved market penetration, expanded networks and cross-selling opportunities -may not be realized in the form of increased revenue within a specific timeframe and may be difficult to forecast, the report noted. External economic forces, such as market movements, business cycles, and competitors’ marketing efforts, make it difficult to evaluate returns on specific marketing efforts in the near term. Internal factors, such as changes in product quality, delivery times or technical support, can also affect “returns,” making it difficult to attribute results to a specific marketing campaign.

“Progress in this area has been difficult,” says Foster. “Many of the expected returns from marketing efforts are intangible or long-term. Hence, measuring the return on investment in marketing is a more complex and less well-developed process than calculating investment returns in other business areas.”

About the research: To compile the report, “Managing and Measuring Return on Marketing Investment Research Approach,” the Conference Board distributed 7,842 surveys via e-mail and direct mail and obtained valid responses from 73 companies. Of these, 24 companies were not yet making any efforts to measure marketing investment, and hence completed only part of the survey. For purposes of analysis, these companies were treated separately as a control group. The discussion in the report relates to the remaining 49 companies that report some to significant efforts in measuring return on marketing investment. The research was conducted between April and July 2007.

source: http://www.marketingcharts.com/interactive/marketing-execs-struggle-to-show-roi-7634/

This post went up yesterday on Search Engine Land and is getting a pretty positive response. That is, everyone’s emailing me to voice their disagreement with my approach! Oh well, it makes for good discussion, and that’s what this topic really needs…..

In last month’s column I talked about buying brand keywords and some of the great (somewhat) new ad products available to advertisers. All that talk about buying brand keywords brought me back to a familiar topic about which, if you read my column with any frequency, you just might be getting a tiny bit tired of hearing me rant. In spite of that, I’m back for one more round of discussion about the relationship between paid and organic search traffic, this time with some real math to show for myself. Bear with me, this one’s worth it.

I mentioned in my previous post about paid and organic search traffic that the goal in any analysis is to determine if the paid ad is a source of lift, cannibalization, or both.  At the time I was looking at isolating the traffic from the organic listing and then quantifying the incremental effect of the paid search ad. Since then, thanks to Matthias Blume, I have seen the light, statistically speaking, and have thus found a much more elegant way to conduct this analysis. Let’s put on our statistical hats and go on a journey to the heart of the issue.

Let’s reset the situation: You rank #1 organically for your brand keyword search, and you’re trying to determine if and how much you are willing to pay for a paid ad on your brand keyword search. The goal then is to understand how the clickthrough rate (CTR) of the organic listing is affected by the presence of the paid ad. You’ll do this by looking at how the CTR of the organic listing changes with and without the presence of the paid ad. The CTR rates I am referring to need to be normalized for search volume. I’ll explain what that means below, but that was the tricky part that I didn’t get before, the elusive reality about which Matthias enlightened me.

In order to calculate this normalized CTR, you’ll first need to try to figure out what the search query volume is for the keyword you’re targeting. This isn’t always explicitly defined, but if you’re clever you can find some ways to approximate it with a good deal of accuracy. Some search engines provide approximate monthly search query volume for keywords, and I believe there are third party providers of approximate search query volume data as well. Ideally you will want to have search volume by day for whichever keyword you’re testing on a particular search engine. If you can’t get to this level of resolution that’s ok, go with weekly or monthly search volume. This will just mean that it will take longer to get your results.

I’m going to explain how we did our analysis, and you can make any variations you need based on your own constraints.  First, we chose a few brand keywords to track. At Yahoo!, we have many brand terms to choose from: yahoo, yahoo mail, yahoo finance, etc. At your company you may have a similar experience, or you may have only one brand keyword that really matters to your business.

Here’s how the analysis works: On Day 1 of our research period, we had the organic listing in #1 position, with no paid search ads (at all) on the page. We began to collect referral data from the organic listing. After a week we started buying paid search ads for our brand terms, and began tracking referrals from the paid ad as well. From then until Day 30 we bought paid search at different budget levels and turned it off for periods of time as well, just to get a variety of data points.

At the end of the research period we compiled our data and began to plot it out on a chart. The metrics that we calculated were fairly simple. On any given day, Organic CTR is [referrals from the organic listing divided by search query volume], and Paid Search CTR is [referrals from the paid search ad divided by search query volume]. We calculated these two metrics for each day 1-30, and plotted them on a scatter graph.  It looked like this:

Here is the way to read this graph: On the x (horizontal) axis is the Paid Search CTR. On the y (vertical) axis is the Organic CTR. Each of the data points represents a day where we gathered data, and each point is plotted where the two CTRs meet on the graph (I’ve stripped the values out of the chart to protect the innocent). The line on the graph is a linear regression, basically a trend line that represents a summary of the scattered data points. Here’s the key: If the slope of the line is positive (the line goes up and to the right) as it is here, there is positive synergy between the organic listing and the paid ad. This means that on days where we bought the paid ad, the CTR of the organic listing actually increased. If the line had a negative slope (went down and to the right), there would be negative synergy between the organic listing and the paid ad. I referred to this as cannibalism in my previous column on this topic, and it means that your paid ad is stealing traffic from your organic listing.

So in my case I ran around the building, loudly declaring victory. I mean, what could be better than buying paid search traffic knowing that you’re driving more clicks to your organic listings?  Believe me, I’m not saying that it will turn out this way in every case. Do your own analysis and come to your own conclusions, because naturally it’s going to work differently for everyone. I’m just here to share my story with you, and mine happens to have a very, very happy ending.

source: http://industrialstrengthsem.com/2010/06/29/paid-and-organic-search-now-with-more-math/