How to Design an Analytics Dashboard

After doing data analytics (most people would call it data science but I hate that term – I’m not sure why – maybe it seems pretentious??) for 6-7 years, there’s a common conversation that I have about the balance between doing cool analytics and running a business.  The conversation starts with asking what current project you’re working on followed by some description about the business problem and what a potential model will look like.  However, as the conversation continues, it becomes apparent that the model is unlikely to ever exist or be used and the bulk of the person’s time will be spent collecting data, doing some visualization and navigating internal political challenges.

This got me thinking about my own journey in terms of what I thought was important work versus what clients need and what I’d frequently heard artists describe as selling out – choosing commercial success over artistic integrity.  Early on all I wanted to do was build models.  I would avoid any tech or project that didn’t involve modeling.  After doing that for several years I realized that that isn’t really what most consulting clients need (consulting clients are usually mature businesses that can afford consulting fees but don’t have the physical location or reputation to easily attract permanent talent – think large retailer in a fly over state).

Supposedly, there is an old interview with Steve Jobs where he talks about the difference between Apple and Microsoft in terms of their approach to technology.  He claimed that Microsoft would find cool pieces of technology and then build products around them which led to a lack of user friendliness.  Apple, however, would find customer needs and then source the necessary technology to solve the problem.  I think there is a similar dynamic in analytics today, especially on the modeling side.  There are lots of smart people streaming out of analytics programs and quantitative studies ready to build models and do Ted talks but what a lot of clients really need is to be able to get all their data in one place and have it organized in a usable format.

I don’t know which one is right (I know I love spending a few hours with my ear plugs in, doing modeling) but I think there is a lot of opportunity if you think about how to help clients fix their problems as opposed to showing up with a product or skill and trying to shoe horn it into an already struggling organization.

Importance of Persistence

One of the big adjustments I’ve had to make in adult life has been learning persistence.  When I was younger I usually did stuff that was easy or fun and for a variety of reasons (being the oldest child, being good at school, etc) people rarely called me on it or forced me to do stuff.  The interesting thing is how much I’ve been hampered by this and constantly need to work on improving.

Probably the most helpful tool has been distance running.  I’m an awful runner but I’ve found amazing value in ‘exercising my persistence muscle’ by running.  I think it’s a bit of a way to learn discomfort management.  The rule I’ve generally learned is that it’s important to have decision points.  This means pre-planned points in time when decisions are made while the rest of my time in between these points should be focused on execution.  I came to this conclusion after several experiences trying to do long distance runs.   I’d get into the middle of the run and feel pain / discomfort and quit and then I’d feel really awful later for not pushing thru the discomfort and finishing.  Over time I’ve learned to quit less and by waiting to make decisions after completing the task, there’s a sense of accomplishment and a better perspective on the full arc of planning and acting.

Today I went and played golf.  I was playing poorly, it was extremely hot and the people I was supposed to play with never showed up.  Thus, I was wandering around sweating profusely, feeling lonely and angry.  Around the 10th hole I started considering quitting.  I started doing all the intellectually smart quitting questions.  Is this sun good for your skin?  Are you wasting your time golfing when you could be doing work stuff? Etc.  I decided to stick it out but not take things seriously.  I started playing weird shots, with clubs I don’t usually use and tried to not take things so seriously.  This helped and the shots got a bit better and I got a bit less bitter.  Then 2 holes later another single player showed up and asked if he could join me.  We ended up playing 5 or 6 holes together and chatting and generally having a nice time.  This isn’t always going to happen.  Sometimes things will get worse.  But I think there is still some kind of psychic value to working thru unpleasant situations even if there’s no magical resolution.

Fun stuff from ‘Open’ by Andre Agassi

A couple fun things I got from reading Andre Agassi’s bio, ‘Open’.  Many of them are about his father.

  • His dad would keep salt and pepper in his pockets so he could throw it in people’s eyes if he got into a fight.
  • Andre once got a best sportsmanship trophy but didn’t win the tournament. His father smashed the trophy in the parking lot to prove winning is more important than sportsmanship
  • His father told him doing is more important than thinking and felt school was overrated
  • Andre and Stephi Graf’s fathers got into a fight the first time they met and almost came to blows.
  • Agassi had many ups and downs over the course of his career. The  frank discussions of the downs was one of the best parts of the book.
  • Agassi discusses a lot about his team (trainer, coach, confidants, significant other, etc.)– how they were assembled, how the relationships evolved etc. This is something I had not thought about as with tennis you usually just see the player.

Overall, I’d highly recommend the book.  It’s well written and interesting and gives a good picture of the difficulties associated with being a top flight pro athlete.

Machine Learning Next Big Thing

Machine Learning Next Big Thing

One thing that I’ve been wrestling with recently is where is applied machine learning going?  Part of this is pure curiosity but there are also parts around whether there are opportunities to take advantage of and also a somewhat defensive angle – as in, is my job at risk?

When I started in consulting, you could make a nice living just going to a client, pulling some data, running some analysis and giving them a deck or a piece of code.  Those days are pretty much over.  Now, most firms will have at least one person that can manage data and do solid analysis in Excel and be on their way in R, Python or Tableau.  The stuff we do now is more relating to managing very large data systems (usually Hadoop or some other distributed computing environment), managing portfolios of models and all of the people issues that come along with these.  There are occasional point solutions but they need to be built quickly and in a standard way. 

Unfortunately, one area I’m behind on is looking at new products in the various parts of a knowledge management system.  For instance one issue that we wrestle with on a regular basis is code standardization.  This is nothing technically complex – just how do you get multiple, geographically dispersed people writing similar code onto the same page?

There are a couple ideas that I’m working thru such as creating R packages to build functions that are reusable or seeing if a GUI-based tool like Alteryx can help, but I need a working platform in terms of data, models and associated viz / monitoring tools to have a baseline.  With that in mind I spent the weekend putting one together. 

I started out by getting a new laptop – a fairly cheap, 11.6 inch, HP one with 4GB of RAM, a Pentium processor and a 500 GB hard drive (link).  This might seem small but I’m not looking to go crazy here just enough to build a reasonable working environment and something light enough to travel with.  Also, considering my last non-work laptop was a Lenovo 100S with 2 GB of RAM and 32 GB of storage, this is a huge step up.

Next, I installed and transferred a bunch of software including Office, R, Git, SQL Server Express and Sublime Text.  This all took several hours but was not as bad as I anticipated.  The next step is to find a data set that changes with a pretty high degree of regularity.  The first one that came to mind was securities prices.  In the past I’ve used EODData.com as it’s fairly cheap and I already have some existing ingestion scripts.  I went and found some old data (you can find a sample here: http://www.eoddata.com/NASDAQ_20101105.zip) and used it to build some tables in SQL server.  The scripts I used will be in a GitHub folder that I’ll share later.

One point of note here is that in my experience, it’s very important to build out the entire system (even if it’s pretty poor quality) just to figure out if everything connects and to discover the potential pain points.  I’d use the analogy of building a car where you can put a rough prototype together to see how all the necessary systems fit together versus building a perfect steering wheel and realizing it doesn’t fit with the steering column.  Andy Ng has a lecture somewhere about a colleague who was building a facial recognition system and spent months on the eye-recognition portion only to discover it was fairly worthless in the overall scheme of things. 

I’ll end here and do another post once I’ve made some progress.

 

 

Why write a blog?

“We read to know we are not alone.”  — C.S. Lewis

 

There are several reasons I’d like to write a blog:

  • Track my thinking process over time.
    • I think it’s easy to lie to yourself about your past performance, especially around investing and professional matters. Having a written record helps prove or disprove this.
  • Having an audience tightens the logic.
    • When I write for an audience, I have to try to anticipate questions or challenges to the logic I’m using.
  • Get feedback, suggestions and criticism.
    • Sadly, I don’t know everything and hopefully putting ideas out in a public space will solicit feedback to make them better or get rid of them if they are truly bad.

Topics I’m considering

  • Investing & trading
    • I’m very interested in various aspects of investing. I spent 8-9 years making markets in equity derivates and trade a bit still so this is an ongoing area of curiosity.
  • Machine Learning and Automation
    • I currently work as a consultant helping businesses implement machine learning and automation in their businesses. I’m very interested in potentials combinations with investing.
  • Careers
    • I’ve had a pretty circuitous career path between finance and consulting with occasional consideration of doing medicine. I really enjoy learning about what people do for a living and how they found their niche.