How to Build a Bad Predictive model

 

  1. Don’t pay attention to data types
    1. Treat anything with numbers as a numeric
    2. Don’t remove infrequently occurring factor levels
  2. Ignore seasonality
    1. When modeling retail data use single months or quarters
    2. Holidays don’t matter
  3. Ignore effects of time when generating error metrics
    1. Mix the same time period in your test / train sets
  4. Don’t use a holdout set to determine model parameters
  5. Don’t do counts of rows / columns
    1. As long as you didn’t get an error message when you run your code everything is ok – no rows or columns were silently removed.
  6. Don’t profile your data before building a model
  7. Assume that unknown’s in the target variable of your classification model are negatives (or positives).
  8. Assume that high scoring models don’t need to be examined
  9. Don’t worry about the time-stamping of your data
  10. Don’t set a seed – no need to reproduce the results

How To Hire Analytics Talent

I went on an interview recently and spent a bunch of time discussing what the potential hiring company’s biggest pain points were.  They were in the web commerce space and expanding very quickly.  The biggest theme was around growing pains.  Specifically, how to maintain the company’s successful culture, avoid redundant work (ie two people on different teams doing the same project without knowing about the other) and hire, especially for middle management.  This got me thinking about two things.  How difficult it can be to scope out projects and how to hire people for analytics work.

I am constantly surprised by how difficult it is to predict project timelines, especially early on before getting my hands on the data.  One of the ways that middle managers in analytics can provide value is by being a middle person between the technical folks doing the work and the end users and business sponsor (the person whose budget this is hitting) to help improve these projections.  Having experience building analytics tools (especially if they’re going into a production environment) makes a huge difference in the quality of timeline projections, the ability to source needed talent and explain processes to non-technical folks.

Frequently, technical folks don’t interact well with business folks as the business folks want their problem solved while the technical folks are in the weeds on technical issues without actually understanding why they’re doing it in the first place.  Often, I’ll see too big of a gap in role and seniority that leads to senior business folks making demands of junior technical folks who are too inexperienced to push back or ask for more information before large amounts of time and resources are expended in the wrong direction.

As far as hiring, things I look for are as follows:

  • Don’t be a jerk
  • Do be detail oriented – ask lots of questions
    1. How many rows of data do we have?
    2. Does this make sense?
    3. Which rows can’t we use? Why?
    4. Is this data timestamped?!?
  • Be good at manipulating data (SQL & Excel !!)
  • Understand some stats (ie how can I tell if a variable is important in a regression model?)

Obviously, there are many other layers but these are pretty good to start with.  Another easy filter (I’ve only had to use once) – don’t ask the admin for a selfie with the company logo…

 

On Becoming A Leader – Warren Bennis

I’m going thru some old notes I took on the book – “On Becoming a Leader” by Warren Bennis and decided to load them here so I’d have a more permanent and easily accessible version.   You can get a copy here.

  • Pg 1
  • Intro
    • Master the context
    • Great leaders and followers are always engaged in creative collaboration
    • 4 concepts
      • They are able to engage others by creating shared memories (aka have vision)
      • They have a distinctive voice (Emotional Intelligence)
      • Integrity (able to tell a friend “NO”)
      • Adaptive capacity (Map vs Compass)
    • Get good at finding good mentors!!
    • Discover and cultivate the authentic self
    • Leaders have in common a passion for the promises of life and the ability to express themselves fully and freely
    • Leaders are made, NOT born
    • No leader sets out to be a leader but rather to express themselves fully and freely. They are interested in expressing themselves NOT proving themselves
    • Adult learning is a huge part of leadership
    • Learning is best achieved when the learner takes charge of the process. This is all part of becoming an integrated person.
    • Learning is a process of remembering what’s important
    • Fame and leadership are NOT the same thing and skill at achieving one is NO guarantee of skill at the other
  • Chapter 1
    • Leaders are important because:
      • They are responsible for the effectiveness of organizations
      • Leaders provide much needed anchors or guiding purposes
      • They determine the integrity of institutions
    • Recognize the “context” for what it is – a breaker, NOT a maker, a trap, NOT a launching pad, an end, NOT a beginning – and declare your independence.
    • Thank people and give compliments
    • Be careful who you choose as a role model
    • At some point, vision and character become important
    • 5 areas to look at:
      • Technical competence
      • People skills
      • Conceptual skills (imagination and creativity)
      • Judgement and taste
      • Character
    • Overcome the rules and overcome the context
    • Steps to mastering the context:
      • Becoming self-expressive
      • Listening to the inner voice
      • Learning from the right mentors
      • Giving oneself over to a guiding vision
    • “When I’ve been most effective, I’ve listened to that inner voice”
    • Find out what it is that you’re about and be that. Be what you are and don’t lose it.  It’s very hard to be who we are, because it doesn’t seem to be what anyone wants.
    • “I have little tolerance for institutional constraints. Institutions should serve people but unfortunately it’s often the other way around.  People give their allegiance to an institution and they become prisoners of habits, practices, and rules that ultimately make them ineffectual.
    • The first step toward change is to refuse to be deployed by others and to choose to be deployed by yourself.
  • Chapter 2
    • Leadership traits:
      • Guiding vision – clear idea of professional and personal goals (in spite of setbacks / failures)
      • Passion – leader loves what they do and love doing it ( Tolstoy – “Hopes are the dreams of the waking man”)
      • Integrity = self knowledge + candor + maturity; when you know what you consist of and what you want to make of it, then you can invent yourself. (Trimming your principles or even ideas to please others is a sign of a lack of integrity). Maturity is important to leadership because leading is NOT simply showing the way or issuing orders. Every leader needs to have experienced and grown thru following – learning to be dedicated, observant, capable of working with and learning from others, NEVER SERVILE, always truthful.  Having located these qualities in themselves, leaders can encourage them in others.
      • Integrity is the basis of trust (earned NOT acquired)(more of a product of leadership than an ingredient)
      • Curiosity and daring – leaders wonder about everything, want to learn as much as they can, are willing to take risks, experiment, try new things. They do NOT worry about failure, but embrace errors, knowing they will learn from them.  Learning from adversity.
    • True leaders are NOT born, but made, and usually self made.
    • Developing character and vision is the way leaders invent themselves
    • As presidents, Johnson, Nixon and Carter were all more driven than driving and each seemed trapped in his own shadows. They were haunted men, shaped more by their early deprivations than by their later successes.  They did not, then, invent themselves.  They were made and unmade by their own histories.

Analytics Reporting Best Practices

 

After 7-8 years of management consulting, there are several things that are common among clients.  Most clients have internal political issues.  For instance different teams have different compensation incentives (market share vs gross margin) which leads to internal feuds and disfunction.  Others are in an industry that used to be growing quickly but has slowed or a new powerful competitor has entered (think any retailer competing with Amazon).  However, one problem that seems to be universal is an inability to master, or even achieve in some cases, basic internal reporting.

The conversations often go like this : how many man hours did you spend doing X last quarter?  How much money did you make from this product last month?  Frequently the answer is an explanation about how so and so is working on that and it will be ready in 3, 6, 12, etc months.  This is the less bad version.  At least there is an understanding that things are broken and need work.

The really bad ones are when there is internal reporting and end users don’t trust it or don’t like the output.  In these situations, the company is actually allocating resources to the effort but they are either ineffective or being used by savvy internal politicians to undermine firmwide efforts and feather their own bed.  At one client, a law firm, there were various firmwide initiatives to introduce automation to their business.  Many of the attorneys were concerned about how this would affect their fiefdom and would spend their time picking apart the internal reporting so that they could postpone any deadlines that affected their business area.  This meant that internal meetings were spent questioning the veracity of the reports instead of planning how to take appropriate action.

Sadly, depending on the situation, this is usually not a problem that can be fixed by external consultants.  The conflict of interest is a very strong weapon for the internal questioners, that is frankly not unjustified.  If we’re selling automation work to a company and we’re also responsible for measuring whether it’s working or providing value, that is a bad situation.  I certainly would not want to invest in a company like that.

The best solution would be to define how success is measured before any project is under way and make sure that the decided metrics are available and trustworthy to the satisfaction of all stakeholders.  This saves many wasted hours in the middle of an active project, bickering over performance and how to measure it.  A secondary solution, would be to hire or allocate an internal resource to focus on fixing the reporting issue so that a person with ties and incentives aligned with the company’s success is at the helm.

 

How to Design an Analytics Dashboard

After doing data analytics (most people would call it data science but I hate that term – I’m not sure why – maybe it seems pretentious??) for 6-7 years, there’s a common conversation that I have about the balance between doing cool analytics and running a business.  The conversation starts with asking what current project you’re working on followed by some description about the business problem and what a potential model will look like.  However, as the conversation continues, it becomes apparent that the model is unlikely to ever exist or be used and the bulk of the person’s time will be spent collecting data, doing some visualization and navigating internal political challenges.

This got me thinking about my own journey in terms of what I thought was important work versus what clients need and what I’d frequently heard artists describe as selling out – choosing commercial success over artistic integrity.  Early on all I wanted to do was build models.  I would avoid any tech or project that didn’t involve modeling.  After doing that for several years I realized that that isn’t really what most consulting clients need (consulting clients are usually mature businesses that can afford consulting fees but don’t have the physical location or reputation to easily attract permanent talent – think large retailer in a fly over state).

Supposedly, there is an old interview with Steve Jobs where he talks about the difference between Apple and Microsoft in terms of their approach to technology.  He claimed that Microsoft would find cool pieces of technology and then build products around them which led to a lack of user friendliness.  Apple, however, would find customer needs and then source the necessary technology to solve the problem.  I think there is a similar dynamic in analytics today, especially on the modeling side.  There are lots of smart people streaming out of analytics programs and quantitative studies ready to build models and do Ted talks but what a lot of clients really need is to be able to get all their data in one place and have it organized in a usable format.

I don’t know which one is right (I know I love spending a few hours with my ear plugs in, doing modeling) but I think there is a lot of opportunity if you think about how to help clients fix their problems as opposed to showing up with a product or skill and trying to shoe horn it into an already struggling organization.

Importance of Persistence

One of the big adjustments I’ve had to make in adult life has been learning persistence.  When I was younger I usually did stuff that was easy or fun and for a variety of reasons (being the oldest child, being good at school, etc) people rarely called me on it or forced me to do stuff.  The interesting thing is how much I’ve been hampered by this and constantly need to work on improving.

Probably the most helpful tool has been distance running.  I’m an awful runner but I’ve found amazing value in ‘exercising my persistence muscle’ by running.  I think it’s a bit of a way to learn discomfort management.  The rule I’ve generally learned is that it’s important to have decision points.  This means pre-planned points in time when decisions are made while the rest of my time in between these points should be focused on execution.  I came to this conclusion after several experiences trying to do long distance runs.   I’d get into the middle of the run and feel pain / discomfort and quit and then I’d feel really awful later for not pushing thru the discomfort and finishing.  Over time I’ve learned to quit less and by waiting to make decisions after completing the task, there’s a sense of accomplishment and a better perspective on the full arc of planning and acting.

Today I went and played golf.  I was playing poorly, it was extremely hot and the people I was supposed to play with never showed up.  Thus, I was wandering around sweating profusely, feeling lonely and angry.  Around the 10th hole I started considering quitting.  I started doing all the intellectually smart quitting questions.  Is this sun good for your skin?  Are you wasting your time golfing when you could be doing work stuff? Etc.  I decided to stick it out but not take things seriously.  I started playing weird shots, with clubs I don’t usually use and tried to not take things so seriously.  This helped and the shots got a bit better and I got a bit less bitter.  Then 2 holes later another single player showed up and asked if he could join me.  We ended up playing 5 or 6 holes together and chatting and generally having a nice time.  This isn’t always going to happen.  Sometimes things will get worse.  But I think there is still some kind of psychic value to working thru unpleasant situations even if there’s no magical resolution.

Fun stuff from ‘Open’ by Andre Agassi

A couple fun things I got from reading Andre Agassi’s bio, ‘Open’.  Many of them are about his father.

  • His dad would keep salt and pepper in his pockets so he could throw it in people’s eyes if he got into a fight.
  • Andre once got a best sportsmanship trophy but didn’t win the tournament. His father smashed the trophy in the parking lot to prove winning is more important than sportsmanship
  • His father told him doing is more important than thinking and felt school was overrated
  • Andre and Stephi Graf’s fathers got into a fight the first time they met and almost came to blows.
  • Agassi had many ups and downs over the course of his career. The  frank discussions of the downs was one of the best parts of the book.
  • Agassi discusses a lot about his team (trainer, coach, confidants, significant other, etc.)– how they were assembled, how the relationships evolved etc. This is something I had not thought about as with tennis you usually just see the player.

Overall, I’d highly recommend the book.  It’s well written and interesting and gives a good picture of the difficulties associated with being a top flight pro athlete.