I’ve been in a strange place. I mean professionally, of course. Most of the time and in most of the jobs I’ve held I’ve been a data analyst. Or something like that. Research Statistician, Statistical Programmer, Data Scientist, and yes Data Analyst. That is what I do right now.
My jobs have all had different titles, but at the end of the day I’ve always worked with messy data. Databases. Data Sources. ERP Systems. APIs. Data Lakes. Data Oceans? Data Warehouses. You get my drift right?
What I do is part engineering and part scientist. Sometime hacker. I have to find out where data exists, use programming to pull it out of whatever dark corner my boss has hidden it, and then use tools like statistics and data visualization to help understand what it all means.
Usually, this process ends with a plot, a chart, a table, and sometimes a sort of article. Occasionally, I might hand someone a “data dashboard”, which is like a tiny app that is designed to let a user explore a dataset on their own terms. Almost none of my work is meant to be shared with the outside world and what I do rarely finds it’s way past the sliding glass doors of the office where I work. Well used to work. Of course these days I work from my man cave remotely.
But, but. For a long time now, I’ve done more than data analysis in my various 9-5 jobs. I’ve also had this ongoing fling with coding apps, blogging, and doing social media. It was always difficult and sometimes literally impossible to combine these two worlds. The private world of corporate data analysis and the public let-it-all-hang-out world of social media and the Internet of 2021 are not always compatible. Scratch that. Never Compatible.
Flexing on the Web
So, what I’m saying really is that most of the tricks and tools and procedures I use every day just don’t translate to the world of the web. When I want to Flex and show off some rad plotting situation here I’m kinda stuck. It seems like the best I can do is lean on what I would do at work. That means publish a Tableau dashboard or even an R Shiny interactive dashboard. Indeed I have done just that when I built my original county level and state level Covid-19 data dashboards. They are in Tableau. And on the web.
These are all well and good and yes you can use Tableau and Shiny and even RMarkdown to put your work on the web. But, none of these are satisfying. Not to me.
The Problem with Most Web Data Viz
Using these tools like Tableau, Shiny, RMarkdown, and the rest gives you like 85% of what you want. But, at the end of the day you always have this thing which looks just like every other Tableau, Shiny, Markdown, and the rest data dashboard/visualization/infographic monstrosity out there.
That’s not a flex. I want to put something out there that makes a bold statement. Something that says: Matt looked into this, analysed the data, make the effort, and wrote the code that ripped the scales from your eyes. This is the visualization that really made the case. And it was pretty. And most importantly, it looked like it belonged on the blog.
Because that is the problem I have with all canned visualization tools. At the end of the day my readers either have to literally go to another website or try to awkwardly play with an embedded dashboard to use something that was built with Tableau or Shiny. All of them look good enough and they get the job done, but they are not quite good enough for my audience.
So, what is the solution? How are you, how am I, going to embed visualizations and data analysis into a blog like the one you are reading right now?
There is a solution. But, it’s complicated. Not launching rockets complicated, but more complicated than pointing and clicking your way through a Tableau dashboard and calling it a day. And that is why you are now reading an article about getting data onto your website and not just looking at a mindblowing graphic. One that belongs here.
This is basically the problem that I’ve just laid out and that I’ve had for awhile. Apparently, I’m not the only one. Here’s the deal though. Web developers already know how to do visualization. It’s been at least 10 years since D3 was created from Protovis and that is the plumbing that web developers would use. If they had the data to visualize. But they don’t always have that data.
R Programmers on the other hand, they have the data. And they can visualize the data, but they can’t always get the data onto your website in just the way that you want.
So, on one hand this book is an interesting project because it bridges two worlds that rarely intersect: data analysis and web development. On the other hand, this book is impossibly difficult to wrap my head around.
Who is reading this book I’m working on? Is it someone like me who has a passing knowledge of web development but who is an excellent data analyst? Or is the audience web developers who want to show data on their website but don’t have the tools to gather data or build data pipelines?
In fact, years ago I found myself in this situation. I knew R programming and I had a website and I thought it would be a breeze to just add analysis and visualization. It was not.
What ended up happening is that I went down a rabbit hole of learning web development. It took years before I had mastered the right toolset, practices, and technologies just to do web development.
In a perfect world, creating visualizations on the web will be a team sport and an opportunity for collaboration. A Full Stack Web Developer or a Front End Developer could implement almost any visualization that you had in mind and a Data Analyst or Scientist will be able to provide data in the format it is needed.
Can One Person Do Web Visualization on Their Own?
I believe it is possible and it could be an interesting project for data analysts who want to dabble in web programming or web developers who want to try data science. Certainly, this is an area where a bored mid-career programmer could use to make a jump to a new career.
How would you solve this problem? Have you ever published a visualization on the web? Please share in the comment section below. Feel free to flex and link to your work!