Skip to main content

Visualizing large scale Uber Movement Data

New York's cab data visualization from Uber's Engineering blog
Last month one of my acquaintances in LinkedIn pointed me to a very interesting dataset. Uber's Movement Dataset. It was fascinating to explore their awesome GUI and to play with the data. However, their UI for exploring the dataset leaves much more to be desired, especially the fact that we always have to specify source and destination to get relevant data and can't play with the whole dataset. Another limitation also was, the dataset doesn't include any time component. Which immediately threw out a lot of things I wanted to explore.
When I started looking out if there is another publicly available dataset, I found one at Kaggle. And then quite a few more at Kaggle. But none of them seemed official, and then I found one released by NYC - TLC which looked pretty official and I was hooked.

To explore the data I wanted to try out OmniSci. I recently saw a video of a talk at jupytercon by Randy Zwitch where he goes through a demo of exploring an NYC Cab dataset using OmniSci. And since my dataset was very similar to that, I thought of giving it a try.

You can find the Jupyter Notebook here: https://github.com/rabimba/uber_analysis_mapd/blob/master/uberFinal1.ipynb

Analysis

Just as a toy experiment I tried to answer and visualize the following.

Can we visualize the number of Uber Trips in a period



Distribution based on per hour, week and month

Estimated Monthly base revenue 

Distribution of traffic between months

Distribution based on weekdays and weekends on short and long trips

Distribution of Trip Duration


Trip Duration vs Trip Distance

And also a bunch of interesting facts we can glean from this dataset.
However, while trying to do this, I realized it's pretty hard to work on a huge dataset in jupyter directly if you load the whole dataset into a dataframe anyway.

I used OmniSci's Cloud interface to load up my data and then connect to that dataset using pymapd to read the sql data.
What I did not do was to be smart and utilize OmniSci's super powerful mapd core and slice and dice the dataset in the cloud itself. Which cloud have saved me a lot of time. For example, the query I was running on one-sixth of the whole dataset was taking 25 minutes.

You can take a look at some of my rough ideas, tries and more graphs in this Jupyter Notebook.

However, it seems OmniSci also has a super helpful visualization web interface as part of OmniSci Cloud called Immerse. And I was able to cook up these dashboards in less than 5 minutes.

And Immerse was able to crunch through the whole dataset (not one-sixth) almost instantly and produce these charts for me. I am pretty impressed with it so far. And it seems with help of pymapd and crafting some sql queries, I should be able to harness this speed as well. 
That would be my next try probably.

What's Next:

Since I realized how powerful OmniSci Immerse can be and starting to play with pymapd. My next pet project is merging Uber Movement's yearly data with ward based time series data. So that we can recreate the whole dataset and analyze some of the interesting aspects of it as we did above. I am mostly interested to see (preferably in Bengaluru data)
  • Uber's growth through time (and specific activity growth in different wards)
  • Figuring from historical time series which wards and routes have most traffic in which hour (this also should let us predict which areas may face surge pricing)
  • See if the growth has saturated in any specific place (should give us upper threshold for that area)
  • If an increase in Uber Demand directly co-relate to travel time (maybe the increased demand is causing traffic?)
  • Can we load it up in kepler.gl (more specifically using this demo as a template) and have a nice timeseries visualization?
Should be a fun project!

Update:

Since I wrote this post I was also playing with what we can do if we visualize this data in VR. And I have a preliminary asnwer :D




Comments

Post a Comment

Popular posts from this blog

Curious case of Cisco AnyConnect and WSL2

One thing Covid has taught me is the importance of VPN. Also one other thing COVID has taught me while I work from home  is that your Windows Machine can be brilliant  as long as you have WSL2 configured in it. So imagine my dismay when I realized I cannot access my University resources while being inside the University provided VPN client. Both of the institutions I have affiliation with, requires me to use VPN software which messes up WSL2 configuration (which of course I realized at 1:30 AM). Don't get me wrong, I have faced this multiple times last two years (when I was stuck in India), and mostly I have been lazy and bypassed the actual problem by side-stepping with my not-so-noble  alternatives, which mostly include one of the following: Connect to a physical machine exposed to the internet and do an ssh tunnel from there (not so reliable since this is my actual box sitting at lab desk, also not secure enough) Create a poor man's socks proxy in that same box to have...
Post a Comment
Read more

My Google I/O 2024 Adventure: A GDE's Front-Row Seat to the Gemini Era

Hey tech enthusiasts! Rabimba Karanjai here, your friendly neighborhood Google Developer Expert (GDE), back from an exhilarating whirlwind tour of Google I/O 2024. Let me tell you, this wasn't just your average tech conference – it was an AI-infused extravaganza that left me utterly mind-blown! And you know what made it even sweeter? I had front-row seats, baby! Huge shoutout to the GDE program for this incredible opportunity. Feeling grateful and a tad spoiled, I must admit. 😉 Gemini: The AI Marvel That's Stealing the Show Now, let's dive into the star of the show: Gemini . This ain't your grandpa's AI model – it's the multimodal powerhouse that's set to redefine how we interact with technology. Imagine an AI that doesn't just understand text, but images, videos, code, and even your wacky doodles. Yep, that's Gemini for you! Google's been cooking up this AI masterpiece, and boy, did they deliver! The keynote demo had us all gawk...
Post a Comment
Read more

MovieBuff: Dive Deeper into Movies with Generative AI

MovieBuff: Dive Deeper into Movies Before You Watch MovieBuff: Dive Deeper into Movies Before You Watch Have you ever spent two hours watching a movie only to be disappointed? MovieBuff is here to help! This Streamlit application leverages the power of Google's Generative AI, specifically the Gemini-Pro model, to provide you with detailed information about movies and TV series before you invest your precious time. Motivation Choosing a movie can be overwhelming. With countless options available, it's hard to know which ones are worth watching. MovieBuff aims to solve this problem by offering a quick and easy way to explore movies based on your interests. How it Works MovieBuff is incredibly user-friendly. You can either: Enter the movie title and year: Simply type the name of the movie you're interested in, and MovieBuff will fetch relevant information like plot summaries, directors, genres, themes, main conflicts, settings, character descriptions, tr...
Post a Comment
Read more