Explorations in Data Science and Health

Joining DataRobot

2021-01-11T00:00:00-05:00

A new year has brought some significant changes to our household. First, I wanted to post that I have joined DataRobot this January as a data scientist. The DataRobot platform is a game-changing piece of technology. Providing a complete end-to-end platform that can take AI from an idea into production. I am joining a great team, and it is so exciting to get these new toys to play with.

Now reading Vitner's Luck by Elizabeth Knox

2020-02-01T00:00:00-05:00

I am not sure how Lisa See does it but her books (i have read two) are amazingly engrossing. This one ran long but still lingers with me weeks later.

Now reading Gideon the ninth by tamryn Muir

2020-01-10T00:00:00-05:00

Amazing and fun.

Summer Updates

2019-09-01T00:00:00-04:00

Summer has been a whirlwind and only now starting to settle down. WE spent two weeks in Colombia for a family wedding. It was an amazing trip and Colombia is a destination I would reccomend to anyone. We covered a good bit of the country from the capital, Bogota, to the coast, Cartegna, to the mountains, Armenia.

After returning in mid-July, we moved and even though we techically only moved down the street it was still a nightmare of packing, heat, unpacking and negotiation before everything finally landed in our new home.

As for this project on the blog, I am looking through the git repository and I have four half-written posts about everything from real-estate tax policy (opportunity zones) to pandas method chaining. Algorex Health has been on a period of rapid growth as well something that has been incredible to be a part of.

Lilith's Brood Trilogy by Octavia Butler

2019-07-15T00:00:00-04:00

Amazing work of science fiction

One Day in December by Josie Silver

2019-06-30T00:00:00-04:00

Will certainly be a NetFlix Rom Com movie.

American Rust by Pillip Meyer

2019-06-20T00:00:00-04:00

Adding Audio Files for my Daily Commute

2019-04-23T00:00:00-04:00

I consume a tremendous amount of "spoken word" audio. This is because every day I have my commute a fixed block of time where I have nothing to do but stare out the window of the bus or train I am riding in. 95% of this is either a podcast or a public radio feed. But recently, I have stumbled on an interesting mini-hack.

Overcast, the best iOS podcast player in my opinion, has a hidden premium feature. You can upload your own audio files and have them streamed directly into the app. This is a perfect way for me to go through the backlog of interesting YouTube videos that I have been meaning to catch. A significant portion of my YouTube consumption doesn't need to be watched. This includes lectures on computer science concepts, policy debates, even ted talks. Although these may be more engaging as videos they work fine as audio tracks.

YouTube-DL

YouTube-DL is a command-line utility that lets you download for offline playback videos from most websites. You have to be pretty familiar with the command line to use it but it makes it really easy to get the audio track to a you tube video and then I can upload it to OverCast and stream it right to my phone for the ride home.

Once you have installed youtube-dl, use the -x --audio-format to save just the audio file. It helps to rename them afterward.

youtube-dl -x --audio-format m4a https://www.youtube.com/watch?v=SFVMASELX

For my ride home today, I have a suggestion from my friend Duncan who recommended a lecture of reinforcement learning from Deepmind.

Now reading the Lies of Locke Lamora by Scott Lynch

2019-04-01T00:00:00-04:00

Swash-buckling in the economy of thieves

Role of Randomness

2019-03-27T00:00:00-04:00

Randomness is a very underrated tool in data visualizations. The reason we respond to charts is because the human mind is so good at spatial pattern recognition. But if we are looking specifically for patterns, what role does randomness play in exposing patterns?

Consider the following use case using data from Star Wars API.

In [1]:

import swapi
import pandas as pd
import seaborn as sns
import re
import numpy as np

if 'people_data' not in globals():
    people_data = swapi.get_all("people")

In [2]:

def swapi_get_species_name(val):
    """
    """
    try:
        name = val.get_species().items[0].name
    except:
        name =  'unknown'
    if name == "Human":
        return name
    elif name == "Droid":
        return name
    else:
        name = "Other"
        return name
    
people_frame = pd.DataFrame([{'name':person.name, 
                    'species':swapi_get_species_name(person),
                   'number_of_films': len(person.films)} for person in people_data.items])

In [3]:

sns.catplot(x="species", y="number_of_films", jitter=False, data=people_frame);

This chart seems pretty straightforward. It summarizes characters by species and how many of the Star Wars movies they have appeared in. It shows us that no non-human Star Wars character has appeared in more than seven Star Wars films.

But if we add some randomness we can expose a deeper meaning to the chart. Seaborn, the charting library used here, does this automatically applying "jitter" which randomly shifts each point to appear near it entry on the x-axis. Furthermore, we can ask use Seaborn's 'swarm' option which ensures no points overlap.

In [4]:

sns.catplot(x="species", y="number_of_films", kind='swarm', data=people_frame);

From this chart our conclusion changes. Because we can see the full distribution of the data, we now know that distribution of human and non-humans appearing in recurring roles is actually relatively even. There may be a few more recurring Human characters but it is not as lopsided as shown in the first chart. No obviously this chart doesn't account for screen time, or number of lines spoken.

Randomness helped add a dimension to this chart. Everyone should read Seaborn's great documentation on swarm plots for better examples. Fundamentally, randomness served to reveal density where there are many attributes of the same value.

Lack of Precision¶

Randomness can also be useful when there is a lack of precision in data. Consider charts showing the density of people and buildings. Density is defined as the number of observations documented in a given area. Rarely do we know the exact location of each point but we know the bounding box of where they occurred. Placing a point at the center of each area would hide the density but if we randomly distribute the points within an area, we can visualize the density in an intuitive way.

Let's look at this through an example using population mapping. This example borrows data from our past post on Layering Maps and Data.

Areas in the case are census tracts. The traditional approach might be to place a single circle in the center of each census tract and have the size of each point vary by the number of households.

In [7]:

import geopandas as gpd
import altair as alt
import pandas as pd
import json
alt.data_transformers.enable('json')

census_tracts = gpd.read_file("data/census_tracts_ma/tl_2018_25_tract.shp")
households_data = pd.read_csv("data/census_tracts_ma/ACS_17_5YR_S1903.csv", dtype={'GEO.id2':str}).rename(columns={
    'HC01_EST_VC02':'number_households', 'GEO.id2':'GEOID',  'HC01_EST_VC04':'white_households',
    'HC01_EST_VC05':'aa_households', 'HC01_MOE_VC07':'asian_households','HC01_EST_VC06':'Native_American',
    'HC01_EST_VC08': 'native_hawaiian', 'HC01_EST_VC09':'some_other'
})
crs = {'init' :'epsg:4326'}
census_tracts = census_tracts.to_crs(crs)

columns = ['number_households','GEOID','white_households','aa_households','asian_households',
          'Native_American','native_hawaiian','some_other']
census_tracts = census_tracts.join(households_data[columns],rsuffix='ich')
filtered_census_tracts = census_tracts[(census_tracts.COUNTYFP == '025') &
                                      (census_tracts.AWATER/census_tracts.ALAND < .55)]

boston_boundary = gpd.read_file("data/towns/TOWNS_POLY.shp").to_crs(crs)
boston_boundary_features = json.loads(boston_boundary[boston_boundary.FIPS_STCO == 25025].to_json())
altair_boston_boundary = alt.Data(values=boston_boundary_features['features'])


base = alt.Chart(altair_boston_boundary).mark_geoshape(fill='lightgray',stroke='white', strokeWidth=1.25).encode(
)

In [8]:

data_to_chart = pd.DataFrame([{'latitude':z[1].y, 'longitude':z[1].x, 'number_households':z[0]} for z in zip(filtered_census_tracts.number_households,  filtered_census_tracts.centroid)])

points = alt.Chart(data_to_chart).mark_point(filled=True, opacity=.8).encode(
latitude='latitude:Q',
longitude='longitude:Q',
size=alt.Size('number_households:Q', bin=True))

base + points

Out[8]:

This simple example does work and broadly does show the density of tracts throughout the boston area. But since we know the area of each census tract we have the opportunity to distribute each household within the area. This requires a bit of pandas magic:

We will create a dummy array of one race value for each household in the census block according the numbers provided by the census then proceed to finding random points.
For each census tract, we will calculate a "bounding box" which is a standard rectangle that encompasses the full polygon of the census tract.
Now that we have nice rectangle, we can calculate a random position within the x,y ranges for that rectangle.
In a loop, we will calculate one random point for each household from the census data.
We test to make sure that point exists within the actual census tract and if it is not we regenerate. This might create a few extra cycles but its worth it.

In [23]:

from random import uniform
from shapely.geometry import Point

def create_random_point(bounds, tract, race):
    """
    Credit to https://gis.stackexchange.com/questions/6412/generate-points-that-lie-inside-polygon
    for inspiration on this. 
    """
    minx, miny, maxx, maxy = bounds
    geometry = tract.geometry
    def create_test_point():
        x = uniform(minx, maxx) 
        y = uniform(miny, maxy)
        return Point(x, y)
    test_point = create_test_point()
    while not test_point.within(geometry):
        test_point = create_test_point()
    return {'latitude':test_point.y, 'longitude':test_point.x, 'race':race}

def get_tract_points(tract):
    bounds = tract.geometry.bounds
    counter_list =  ['white']*tract.white_households + ['african_american']*tract.aa_households + \
                    ['asian']*tract.asian_households + ['Native_American']*tract.Native_American + ['native_hawaiian']*tract.native_hawaiian + \
                    ['other']*tract.some_other
    tract_points = []
    for race in counter_list:
        tract_points.append(create_random_point(bounds, tract, race))
    return tract_points
        
    

all_points = pd.concat([pd.DataFrame(get_tract_points(tract)) for tract in filtered_census_tracts.itertuples()])

In [35]:

domain = ['white', 'african_american', 'other', 'asian', 'Native_American',
       'native_hawaiian']
points = alt.Chart(
    all_points, 
    title="One Point per Households in Suffolk County Boston Massachusetts Colored by Median Income"
).mark_point(
    size=.3, 
    opacity=.5, 
    filled=True
).encode(
    latitude='latitude:Q',
    longitude='longitude:Q',
    color=alt.Color('race:N', scale=alt.Scale(scheme='dark2', domain=domain))
)


(base + points)

Out[35]:

This post was inspired by the fantastic Racial Dot Map, a project by the University Virginia. This fascinating view provides a realistic view of racial pockets throughout the country and it is driven by the exact same approach we have here.

Here is a view from that project of the same region.

Note the improved granularity of the of their result. This again is example of precision. The Racial dot project used census blocks the smallest area of geography for which the US Census provides data while we are relying on census tracts in the image above.

In [ ]:

Scaling Data

2019-03-19T00:00:00-04:00

Scaling data is a constant challenge in analytics projects. Whether to get a visualization to fit on screen or to take a independent variable and remove outliers scaling transforms data from working in one domain or another.

Let's think about this by example, let's say you wanted to analyze the US Government budget. You would find that the US Federal government spends about \$2 Billion on deposit insurance and \\$1 Trillion on Medicare. A huge range. But what if we wanted to know about what share that would be if the US National budget were only \$100.

In that case we would scale the budgets to between \$0.00 and \\$100. There are many approaches and mathematical ways to scale the data. In this cookbook, we will compare all of the approaches in an attempt to pick the best one.

About the data¶

The dataset for this analysis is the US Budget Federal Account balances which can be downloaded from US Spending website.

let's load the data and take a look. Specifically, we are going to focus in on three main columns:

budget_authority_appropriated_amount: The amount congress decided this account should have.

In [13]:

import pandas as pd
import matplotlib.pyplot as plt
pd.options.display.float_format = '${:,.2f}'.format


budget_data_raw = pd.read_csv("data/all_federal_account_account_balances_1.csv")
budget_data_raw.head()

Out[13]:

	federal_account_symbol	federal_account_name	agency_name	budget_function	budget_subfunction	budget_authority_unobligated_balance_brought_forward	adjustments_to_unobligated_balance_brought_forward	budget_authority_appropriated_amount	borrowing_authority_amount	contract_authority_amount	spending_authority_from_offsetting_collections_amount	total_other_budgetary_resources_amount	total_budgetary_resources	obligations_incurred	deobligations_or_recoveries_or_refunds_from_prior_year	unobligated_balance	gross_outlay_amount	status_of_budgetary_resources_total
0	070-5702	9-11 Response and Biometric Exit Account, U.S....	Department of Homeland Security	Administration of Justice	Federal law enforcement activities	$65,767,068.87	$4,895,455.21	$59,911,887.98	$0.00	$0.00	$0.00	$0.00	$130,574,412.06	$81,302,583.39	$4,895,455.21	$49,271,828.67	$69,493,561.19	$130,574,412.06
1	014-5015	Abandoned Mine Reclamation Fund, Office of Sur...	Department of the Interior	Natural Resources and Environment	Conservation and land management	$173,858,680.72	$21,873,656.93	$379,980,611.05	$0.00	$0.00	$952.77	$952.77	$575,713,901.47	$402,411,947.53	$21,873,656.93	$173,301,953.94	$285,538,947.66	$575,713,901.47
2	014-2640	Abandoned Well Remediation Fund, Bureau of Lan...	Department of the Interior	Natural Resources and Environment	Conservation and land management	$447,121.09	$5.27	$0.00	$0.00	$0.00	$0.00	$0.00	$447,126.36	$285,815.33	$5.27	$161,311.03	$9,415,726.69	$447,126.36
3	068-0118	Abatement, Control and Compliance Loan Program...	Environmental Protection Agency	Natural Resources and Environment	Pollution control and abatement	$0.46	$-0.39	$0.00	$0.00	$0.00	$0.00	$0.00	$0.07	$0.00	$0.00	$0.07	$0.00	$0.07
4	069-0503	Access Highways to Public Recreation Areas on ...	Department of Transportation	Transportation	Ground transportation	$352,333.19	$0.00	$0.00	$0.00	$0.00	$0.00	$0.00	$352,333.19	$0.00	$0.00	$352,333.19	$0.00	$352,333.19

Let's group the data by the "budget subfunction" as defined by the Department of the treasury.

In [20]:

budget_function_data = budget_data_raw.groupby(["budget_subfunction"]).sum()['budget_authority_appropriated_amount'].sort_values()

budget_function_data.plot(kind='bar')
ax1 = plt.axes()
x_axis = ax1.axes.get_xaxis().set_visible(False)
ax1.yaxis.set_major_formatter(FormatStrFormatter('$%.0f'))

Sci-Kit Learn contains most of the libraries necessary to support scaling in a wide variety of use cases.

Below we will create a pipeline that will call five of the most common scaling functions.

In [15]:

from sklearn.preprocessing import *
from sklearn import pipeline
from sklearn_pandas import DataFrameMapper


mapper = DataFrameMapper([
    (['budget_authority_appropriated_amount'], MinMaxScaler(feature_range=(0,100))
     , {'alias': 'MinMaxScaler'}),
    (['budget_authority_appropriated_amount'], MaxAbsScaler()
     , {'alias': 'MaxAbsScaler'}),
    (['budget_authority_appropriated_amount'], RobustScaler()
     , {'alias': 'RobustScaler'}),
    (['budget_authority_appropriated_amount'], QuantileTransformer(n_quantiles=10)
     , {'alias': 'QuantileTransformer'}),
    (['budget_authority_appropriated_amount'], PowerTransformer()
     , {'alias': 'PowerTransformer'})
], df_out=True)
  
scaled_data = mapper.fit_transform(budget_function_data.reset_index())

and let's take a look at the results:

In [16]:

import matplotlib.pyplot as plt
from matplotlib.ticker import ScalarFormatter, FormatStrFormatter



fig = plt.figure(figsize=(20,10))
fig.suptitle('SkLearn Scaling Functions', fontsize=16)
# 3 x 1 grid, position at (1st row, 1st col), take two rows

ax0 = plt.subplot2grid((3, 2), (0, 0), rowspan=1) 

budget_function_data.plot(kind='bar', title='Original Data')
ax0.yaxis.set_major_formatter(FormatStrFormatter('$%.0f'))

ax1 = plt.subplot2grid((3, 2), (1, 0), rowspan=1) 

scaled_data['MinMaxScaler'].plot(kind='bar', title='MinMaxScaler')

ax2 = plt.subplot2grid((3, 2), (2, 0), rowspan=1) 
scaled_data['MaxAbsScaler'].plot(kind='bar', title='MaxAbsScaler')


ax3 = plt.subplot2grid((3, 2), (0, 1), rowspan=1) 

scaled_data['RobustScaler'].plot(kind='bar', title='Robust Scaler')

ax4 = plt.subplot2grid((3, 2), (1, 1), rowspan=1) 
scaled_data['QuantileTransformer'].plot(kind='bar', title='QuantileTransformer')
# get the other ax
ax5 = plt.subplot2grid((3, 2), (2, 1), rowspan=1) 
scaled_data['PowerTransformer'].plot(kind='bar', title='PowerTransformer')


for axis in fig.get_axes():
    axis.get_xaxis().set_visible(False)

The effects and drawbacks should be fairly obvious. Both MinMaxScaler and MaxAbsScaler keep the same overall distribution. The relative difference between say "Executive Management" and "Medicare", which have budgets that are about 2000x different, are maintained. RobustScaler appears the same but it subtly different. It is based on centering between the 75th and 25th percentiles. In this case Medicare and some of the smaller accounts are effectively clipped at the 25th and 75th percentile. Thus, outliers are managed the the relative differences are maintained for the whole population.

The last two PowerTransformer and QuantileTransformer are deliberately manipulative. Each force the population to fit a specific distribution. This removes the vast differences in scale inherent in the data and makes things seem more comparable. But beware, looking at these charts it appears that Medicare and the Defense department are just a little bit more expensive than say interest payments on debt; but, they are an order of magnitude different as shown below:

In [18]:

budget_function_data.tail()

Out[18]:

budget_subfunction
Interest on Treasury debt securities (gross)     $521,553,461,024.26
Health care services                             $551,865,968,531.50
Department of Defense-Military                   $679,616,519,190.72
Social security                                $1,025,474,293,425.75
Medicare                                       $1,072,980,747,544.25
Name: budget_authority_appropriated_amount, dtype: float64

In [ ]:

now reading The Golden State by Ben Winters

2019-03-18T00:00:00-04:00

Great dystopian noir.

Layering Maps and Data?

2019-03-13T00:00:00-04:00

At Algorex Health, we make a lot of maps and get a lot of questions about the tools we use to make them. So, I thought I would briefly describe our process and the tools we use. I covered the basics of geographic charting in a past blog post and some of that terminology will be repeated here.

We use to two primary data visualization systems at Algorex Health both of which were chosen for their support withthin the Python/Jupyter stack, ease of use, and/or aesthetic flexibility:

Altair: Altair is a python library that generates schemas that use Vega and Vegalite specification for visualization, those in turn rely on D3 for robust charting in a variety of formats. Altair becuase it relies on Vega-lite has a broad library of chart types including maps however Altair is all about the data. Every element of a map must be expressed in data and there is no base layer or tile layer. The tile layer is the part of a map you probably most recognize. It is the streets, borders, lakes, oceans that are automatically placed behind your map elements from a service like google maps. Altair maps don't support having a tile layer so when that is needed for interactivity we move on to Folium.
Folium: Folium is a python api that generates maps using the Leaflet javascript library which is one of the defacto standards for web based interactive maps. In rare cases, we write leafletJS directly but generally Folium supports everything we need.

For this post, we will utilize Folium. I think the difference between most of our maps and the ones our clients are creating are in the rich use of layers. Tableau and other GUI applications make great maps but their geographies are predefined and it can be difficult to layer data from multiple types of shapes. Folium, which visualizes maps only provides the most flexibility and interactivity.

In the example here we will layer the Median Income by census tract for the Boston area and transit stations on the MBTA. This will layer two types of data: Median income by shape, and transit stations which are specific points.

Loading the data¶

There are three data files we need for this blog post all freely accessible:

Locations and Shapes for the MBTA Mass Transit system which is available from the MassGIS agency.
Census Tract shapes, available from the US Census.
The median income data also available from the US Census.

We will load these data files using Pandas and GeoPandas to look at the data.

In [1]:

import pandas as pd
import geopandas as gpd

mbta_stations = gpd.read_file("data/mbta/MBTA_NODE.shp")
census_tracts = gpd.read_file("data/census_tracts_ma/tl_2018_25_tract.shp")
median_income_data = pd.read_csv("data/census_tracts_ma/ACS_17_5YR_S1903.csv", dtype={'GEO.id2':str}).rename(columns={
    'HC03_EST_VC02':'median_income', 'GEO.id2':'GEOID'
})

The MBTA data has a point geometry for each MBTA station.

In [2]:

mbta_stations.head()

Out[2]:

	STATION	LINE	TERMINUS	ROUTE	geometry
0	Ashmont	RED	Y	A - Ashmont C - Alewife	POINT (236007.5383913815 892693.0225654766)
1	Harvard	RED	N	A - Ashmont B - Braintree C - Alewife	POINT (231387.2739390731 902684.0159827843)
2	Kendall/MIT	RED	N	A - Ashmont B - Braintree C - Alewife	POINT (234087.917269595 901406.5508935936)
3	Capen Street	RED	N	Mattapan Trolley	POINT (234055.4375255629 890869.3748197779)
4	Tufts Medical Center	ORANGE	N	Forest Hills to Oak Grove	POINT (235900.3235432804 899934.3133242205)

Those values don't look like latitude and longitudes. We need these geometries to be standardized if we are going to layer everything together. We fix this by changing the CRS of the file which will convert the geometry to a standard value.

In [3]:

crs = {'init' :'epsg:4326'}
mbta_stations = mbta_stations.to_crs(crs)
mbta_stations.head()

Out[3]:

	STATION	LINE	TERMINUS	ROUTE	geometry
0	Ashmont	RED	Y	A - Ashmont C - Alewife	POINT (-71.06343014439 42.28388354622533)
1	Harvard	RED	N	A - Ashmont B - Braintree C - Alewife	POINT (-71.11890607237824 42.37402923068517)
2	Kendall/MIT	RED	N	A - Ashmont B - Braintree C - Alewife	POINT (-71.08619149399846 42.36241534488428)
3	Capen Street	RED	N	Mattapan Trolley	POINT (-71.08720482389784 42.2675530566276)
4	Tufts Medical Center	ORANGE	N	Forest Hills to Oak Grove	POINT (-71.06428155077408 42.34908000707444)

Much better now let's look at census tracts which we will convert as well just to be safe.

In [4]:

census_tracts = census_tracts.to_crs(crs)
census_tracts.head()

Out[4]:

	STATEFP	COUNTYFP	TRACTCE	GEOID	NAME	NAMELSAD	MTFCC	FUNCSTAT	ALAND	AWATER	INTPTLAT	INTPTLON	geometry
0	25	027	724100	25027724100	7241	Census Tract 7241	G5020	S	53031300	1639215	+42.2566908	-072.1581690	POLYGON ((-72.21781799999999 42.27018, -72.217...
1	25	027	759100	25027759100	7591	Census Tract 7591	G5020	S	25570221	1427830	+42.2096822	-072.0401777	POLYGON ((-72.07887699999999 42.214751, -72.07...
2	25	023	510503	25023510503	5105.03	Census Tract 5105.03	G5020	S	2561883	3203	+42.0942619	-071.0655746	POLYGON ((-71.080483 42.095539, -71.0801759999...
3	25	005	652000	25005652000	6520	Census Tract 6520	G5020	S	439925	0	+41.6266977	-070.9278159	POLYGON ((-70.931105 41.63193, -70.929509 41.6...
4	25	023	510600	25023510600	5106	Census Tract 5106	G5020	S	5303844	49230	+42.0805604	-071.0602559	POLYGON ((-71.07875799999999 42.088274, -71.07...

Lastly, here is the median income data. The median income for each census tract is on column 7 which we named median_income.

In [5]:

median_income_data.head()

Out[5]:

	GEO.id	GEOID	GEO.display-label	HC01_EST_VC02	HC01_MOE_VC02	HC02_EST_VC02	HC02_MOE_VC02	median_income	HC03_MOE_VC02	HC01_EST_VC04	...	HC02_EST_VC52	HC02_MOE_VC52	HC03_EST_VC52	HC03_MOE_VC52	HC01_EST_VC53	HC01_MOE_VC53	HC02_EST_VC53	HC02_MOE_VC53	HC03_EST_VC53	HC03_MOE_VC53
0	1400000US25001010100	25001010100	Census Tract 101, Barnstable County, Massachus...	1772	135	1772	135	47500.0	9839.0	1635	...	41.3	8.6	31595.0	9777.0	113	50	10.3	4.5	NaN	NaN
1	1400000US25001010206	25001010206	Census Tract 102.06, Barnstable County, Massac...	1503	157	1503	157	59042.0	13567.0	1449	...	32.5	16.3	22177.0	17369.0	60	48	9.0	7.2	80882.0	12715.0
2	1400000US25001010208	25001010208	Census Tract 102.08, Barnstable County, Massac...	796	134	796	134	62844.0	12976.0	772	...	38.9	14.4	43676.0	25949.0	19	22	5.8	7.2	NaN	NaN
3	1400000US25001010304	25001010304	Census Tract 103.04, Barnstable County, Massac...	1145	142	1145	142	71250.0	15039.0	1113	...	27.4	12.4	22986.0	18810.0	0	12	0.0	7.4	NaN	NaN
4	1400000US25001010306	25001010306	Census Tract 103.06, Barnstable County, Massac...	1301	150	1301	150	55694.0	9955.0	1208	...	26.9	14.6	NaN	NaN	10	14	1.9	2.6	NaN	NaN

5 rows × 243 columns

We need to merge the median income data with the tract shapes before we proceed which uses a little pandas magic. The GEOID is the common field between them. We will also select out only the columns we need from the median income data.

In [6]:

columns = ['median_income','GEOID']
census_tracts = census_tracts.join(median_income_data[columns],rsuffix='ic')

Make the Map¶

For the first map, we will use Altair. This map won't have base tiles but we will be able layer each element. The first layer will be a choropleth of median income followed by a layer of each MBTA station. Altair expects JSON data so there is some data wrangling before the map is drawn.

Filter Criteria¶

Because we do this all the time, it is helpful to filter out some of the data right now. This comes from having some hard lessons.

Restrict to only the counties you need. The MBTA mass transit lines are only operating in three counties so we can restrict to just those.
Remove census geographies with a lot of water. The field AWATER has the area in square meters of the census geography that is made of water while ALAND is the area of land. Anything with more water than land we are going to filter out.

In [7]:

filtered_census_tracts = census_tracts[(census_tracts.COUNTYFP.isin(('021','025','017'))) &
                                      (census_tracts.AWATER/census_tracts.ALAND < .55)]

To initiate a folium map, we will first set a few parameters.

In [8]:

import folium 
import branca.colormap as cm

folium_parameters = { 'location':[42.3601, -71.0589],
    'zoom_start':11,
    'width':300,
    'height':300}

boston_map = folium.Map(**folium_parameters)
boston_map

Out[8]:

Now we will begin to add layers starting with the census tracts. Because these are colored shapes, we will use the folium GeoJson class. This will draw one shape for every feature in the data. We will also style each feature using a linear color map so that the colors will be assigned based on the median income of the census tract. The colormap functions are supplied by the branca companion package.

In [9]:

color_scale= cm.linear.PuRd_07.scale(filtered_census_tracts.median_income.min(), 175000)
def tract_styles(feature):
    return {
        'fillColor': color_scale(feature['properties']['median_income']),
        'color': 'grey',
        'weight': .5,
    'dashArray': '5, 5'
        ,'fillOpacity':.8
    }
    
choropleth = folium.GeoJson(filtered_census_tracts.dropna().to_json(), style_function=tract_styles)
choropleth.add_to(boston_map)

Out[9]:

<folium.features.GeoJson at 0x7f7a7c52c080>

Adding the stations works a little differently. We could use the same GeoJson class above but the result would be that the stations would be visualized as markers which looks a little funny.

In [10]:

marker_only_map = folium.Map(**folium_parameters)

folium.GeoJson(mbta_stations.to_json()).add_to(marker_only_map)
marker_only_map

Out[10]:

Because we want to control the points drawn and make them circles not map markers, we will instead add each one to the map individually calling a style function that will assign it's color, based on its MBTA line.

In [11]:

def station_style(station):
    def station_colors(line):
        station_colors = {'ORANGE':'orange','BLUE':'blue','RED':'red','SILVER':'silver','GREEN':'green'}
        return station_colors.get(line,'grey')
    return folium.CircleMarker(
        radius=2,
        location=(station.geometry.y, station.geometry.x),
        color=station_colors(station.LINE),
        fill=True,
        fill_opacity=.8)
stations = folium.FeatureGroup()
for station in mbta_stations.itertuples():
    station_style(station).add_to(stations)

stations.add_to(boston_map)

Out[11]:

<folium.map.FeatureGroup at 0x7f7a6b570f98>

The other note is that the base layer, which defaults to OpenStreetMaps, is a little crowded. We don't need to show how someone navigates in Boston, we just need the minimal geography to give the viewer their bearings. To do this, we will change the default map tiles. Folium and Leaflet support a web standard for fetching tiles but you can also use any of the freely available tile styles shown in the leaflet documentation.

For this chart a basic gray style works best.

In [12]:

boston_map.add_tile_layer(**{'tiles':'https://server.arcgisonline.com/ArcGIS/rest/services/Canvas/World_Light_Gray_Base/MapServer/tile/{z}/{y}/{x}',
    'attr':'Tiles &copy; Esri &mdash; Esri, DeLorme, NAVTEQ'})
boston_map.width = 500
boston_map

Out[12]:

This blog bost originally appeared at Algorex Health

In [ ]:

Solving Problems

2019-03-10T00:00:00-05:00

One of my first consulting projects was working on the call center software for a large health insurer. The client was implementing a massive core system replacement and various satellite applications, lie the call center software, needed to retrofitted to call data from both. As this was the early 2000s, the buzzword de-jour was "service oriented architecture (SOA)" a series of web services on an "Enterprise Service Bus (ESB)" that would of the provide data to the affiliated applications from either the old core or the new.

Large Enterprises Forget Large Things

The centerpiece service on the ESB was the router. It's job to to find members and identify which system their data resided in as of which date and provided keys that could be used in subsequent ESB service calls. The router was in many ways amazing. Built with full Oracle Java stack running on the latest IBM blade servers. It easily could churn through a million transactions without issue. There was one problem. . .

The router was invoked with a date and a member ID which it used to identify whether members records were in the new system or the old system. But members don't always call the call center with their ID, about 2/3 of call center calls use name and date of birth in a search to find the member. As designed without an ID (either from the old system or the new system), the router couldn't identify which system the member would be in and couldn't provide the correct directions to other services. This issue was first identified when I couldn't complete a design a document mapping call center data fields to XSLT templates for the ESB web services. The router was in testing but not yet in production. This problem should have been caught in requirements but at the end of day it hadn't escaped to production or even to user testing.

Never Assume Your Use Cases are Known

How was this missed? Well, the most critical transactions for a health insurer are the automated claims submissions from various hospitals arriving by the thousands per hour. These largely have identifiers and are specific dates of service. The router processed these transactions quickly and efficiently and many of the engineers working on the project had extensive past knowledge working in these processing data pipelines. The most obvious customer service use case just wasn't considered.

I tell new people starting on my teams never to assume that I am aware of even the most obvious problems and try to always come with a solution. My design document did propose a solution one I had to present to a risk management review. The call center software could not use the router and instead issue two calls to a related search service with dummy dates and values that would ensure that either got two response (one from the old system and one from the new) and would allow us to decide how to move forward. My proposed solution had a few problems in it and wasn't adopted but thats fine.

Software projects are complicated and chaotic but no problem is truly unsolvable. There are bad solutions that hardcode core system logic in code, or create massive technical debt like mine but there are always solutions. It's just key to be the one participating and moving the discussion forward to the solution not just pointing out problems waiting for them to be fixed.

Self-Reflection

This is one of those posts where I write about behavior I strive to achieve but absolutely fall short of. I’ve worked on projects in miserable situations where honestly you can’t help but laugh when some major issue causes a derailment. But, this is a bad cycle to fall into. Even on the most hopeless projects, there is always a solution a way to break out and put things on solid footing.

OpenElections

2019-03-06T00:00:00-05:00

Newly elected Congresswoman Alexandria Ocasio-Cortez is in the new a lot these last couple of months. She was elected in the November 6, 2018 general election to represent New York's 14th Congressional District. Now let me ask you a question? do you know how many votes Congresswoman Ocasio-Cortez received in Precinct 037/78 in her district? Go ahead, I will wait . . . You probably went looking on CNN or NYTimes and probably came up short. They don't share this kind of detailed data.

I recently wanted to do a visualization on the US House and Senate voting in the most recent election and I found their exists no publicly accessible dataset that contains these results. You read that right. In one of the most established democracies in the world, its actually very difficult to access election results. News organizations and Universities subscribe to one of a number of companies that synthesize this information such the AP Elections API or CQ Elections Results Collection but again these are closed subscription sources.

That is why with my open-source contribution this week I wanted to highlight the work of two organizations who are working to change this, OpenElections and the MIT Election Lab. But there is simply way too much work todo. As shown in their status chart:

MIT is actually even further behind with data covering just 22 states.

Almost everyone in the United States is assigned a voting precinct. These precincts are very small. Our example Precinct 037/78 had just 46 votes out of more than 690,000 people who live in the 14th District. The vast majority of this precinct is made up of the Bronx Zoo. Many precincts are combined into a polling place. In this case almost 20 precincts all vote at Giordano MS. The totals by precinct are then bubbled up through the elections hierarchy until finally a vote total is published. THe official totals aren't published in a standard format, aren't universally available online meaning its extremely difficult to assemble these for the 435 members of the US House let alone for the over 500,000 elected officials nationwide.¹

These two organizations try to wrangle all of this together. Through web-scraping, volunteer data entry, and freedom of information act requests, they somehow get a hold of the spreadsheets and pdfs that contain this information and make it freely available.

The data challenge

Let's return to our sample district 037/78, what is needed to process this data? New York is one of the organizations that at leasts publishes spreadsheets of the results. Many other agencies only publish PDFs or just have images of paper records which cannot be machine processed. For Congresswoman Ocasio-Cortez, the results data file is over 4400 lines long with no headings. For each of the 450 precincts in her district the following ten lines are repeated:


Public Counter	41
Manually Counted Emergency	0
Absentee / Military	7
Federal	0
Affidavit	0
Alexandria Ocasio-Cortez (Democratic)	39
Anthony Pappas (Republican)	4
Elizabeth Perri (Conservative)	0
Joseph Crowley (Working Families)	1
Joseph Crowley (Women's Equality)	2

This is a nightmare to parse but at least it is parseable. My open source contribution this week was to parse the results of 2013 special election for the Tennessee House of Representatives 91st district. Not glamours. The PDF of the results was available on the website for Tennessee Secretary of State but only as a PDF. I ended up copying out by hand and submitting it just 65 datapoints out of the thousands needed.

now reading The Incendiaries by R.O. Kwon

2019-03-05T00:00:00-05:00

Thanks @d_lazarin for the suggestion.

Who is going to Orlando?

2019-02-11T00:00:00-05:00

This week the health world turns to Orlando for the annual conference of the Health Information Management Systems Society annual conference. it is absolutely one of the biggest events in health IT and one of the top 25 trade shows in the entire country.

How many people attend HIMSS each year? We could search one of hundreds of press releases about the event from exhibitors. We could visit the link above from Trade Show news.

Big Data Approach¶

How about we measure how many people fly to Orlando during the quarter when HIMSS takes place. The Bureau of Transportation Statistics releases quarterly data files drawn from a 10% of all airline itineraries sold each quarter from all of the major US airline carriers, with more than 5% market share.

For this analysis, we downloaded the Q1 2017 which corresponds to the last time HIMSS visited Orlando and Q1 2015 which was presumably a non-HIMSS year, though the conference center likely booked other events for the same period.

However, US airports see about 2.6 Million per day so even 10% of tickets for one quarter is a massive file.

here are some basic stats on just one of the files:

In [7]:

import subprocess

lines_in_file = subprocess.run(["wc", "-l", "data/flight_origin_destination/Origin_and_Destination_Survey_DB1BMarket_2017_1.csv"], stdout=subprocess.PIPE)
print("Lines in the file {0}".format(lines_in_file.stdout))

Lines in the file b'5866624 data/flight_origin_destination/Origin_and_Destination_Survey_DB1BMarket_2017_1.csv\n'

The first step in processing data for our visualizations is typically to load the data into a pandas dataframe to manipulate and prepare the data before visualizing it. The flexibility and extensibility of Pandas means it is extremely memory intensive. One of the original authors of the library recommends having 5 to 10 times as much RAM as the size of your dataset. For one quarter of the airline data (1.4gb), this would require up to 10gb more than my laptops.

Enter Dask, a project developed by Anaconda that uses the same API as pandas but underpins it with a architecture that parallelizes the operations leaving the data on disk until called for. Dask's power is immense allowing a dataframe to span an entire cluster. But locally, it's also an easy way to use the same python data science workflows while keeping the data waiting until it is needed or in our case aggregated for our analysis.

Let's take a look at the data now.

In [8]:

import dask.dataframe as dd
from vega_datasets import data
import altair as alt
import pandas as pd
import geopandas as gpd
from shapely.geometry import Point, LineString
import json
import altair_theme
%matplotlib inline

In [9]:

data_Q1_2014 = dd.read_csv("data/flight_origin_destination/Origin_and_Destination_Survey_DB1BMarket_2014_1.csv")
data_Q1_2015 = dd.read_csv("data/flight_origin_destination/Origin_and_Destination_Survey_DB1BMarket_2015_1.csv")
data_Q1_2016 = dd.read_csv("data/flight_origin_destination/Origin_and_Destination_Survey_DB1BMarket_2016_1.csv")
data_Q1_2017 = dd.read_csv("data/flight_origin_destination/Origin_and_Destination_Survey_DB1BMarket_2017_1.csv")
data_Q1_2018 = dd.read_csv("data/flight_origin_destination/Origin_and_Destination_Survey_DB1BMarket_2018_1.csv")

In [10]:

data_Q1_2014.head()

Out[10]:

	ItinID	MktID	MktCoupons	Year	Quarter	OriginAirportID	OriginAirportSeqID	OriginCityMarketID	Origin	OriginCountry	...	Passengers	MktFare	MktDistance	MktDistanceGroup	MktMilesFlown	NonStopMiles	ItinGeoType	MktGeoType	Unnamed: 41
0	201412918419	20141291841902	1	2014	1	11618	1161802	31703	EWR	US	...	1.0	266.5	719.0	2	719.0	719.0	2	2	NaN
1	201412918420	20141291842001	1	2014	1	13930	1393003	30977	ORD	US	...	1.0	266.5	719.0	2	719.0	719.0	2	2	NaN
2	201412918420	20141291842002	1	2014	1	11618	1161802	31703	EWR	US	...	1.0	266.5	719.0	2	719.0	719.0	2	2	NaN
3	201412918421	20141291842101	1	2014	1	13930	1393003	30977	ORD	US	...	1.0	268.0	719.0	2	719.0	719.0	2	2	NaN
4	201412918421	20141291842102	1	2014	1	11618	1161802	31703	EWR	US	...	1.0	268.0	719.0	2	719.0	719.0	2	2	NaN

5 rows × 42 columns

In [11]:

# Set some indicators so we know from when the data came. There is no indicator is the raw files. 
data_Q1_2014['Period'] = 201401
data_Q1_2014['HIMSS'] = 'No HIMSS'
data_Q1_2015['Period'] = 201501
data_Q1_2015['HIMSS'] = 'HIMSS'
data_Q1_2016['Period'] = 201601
data_Q1_2016['HIMSS'] = 'No HIMSS'
data_Q1_2017['Period'] = 201701
data_Q1_2017['HIMSS'] = 'HIMSS'
data_Q1_2018['Period'] = 201801
data_Q1_2018['HIMSS'] = 'No HIMSS'

# Dask uses append because it fits with the model of lazily parallel calculation 
# where as concat would rely on the whole dataset being available 
all_years = data_Q1_2014.append(data_Q1_2015).append(data_Q1_2016).append(data_Q1_2017).append(data_Q1_2018)

mco_totals_by_year_task = all_years[all_years.Dest == 'MCO'].groupby(['Period','HIMSS']).agg({'Passengers':'sum', 'MktFare':'mean'})
mco_totals_by_airport_task = all_years[all_years.Dest == 'MCO'].groupby('Origin').agg({'Passengers':'sum', 'MktFare':'mean'})

Each row is a itinerary for one passenger between an origin and destination with the fare and distance. We are focused only on Orlando.

The above commands essentially create two tasks and won't process until called for. In the next command, Dask will compute the aggregates and return a dataframe of the result.

In [ ]:

mco_totals_by_year = mco_totals_by_year_task.compute()
mco_totals_by_airport = mco_totals_by_airport_task.compute()

In [13]:

mco_totals_by_airport.sort_values('Passengers').tail()

Out[13]:

	Passengers	MktFare
Origin
ORD	66146.0	217.768325
MSP	66435.0	201.077036
DTW	67107.0	195.274883
PHL	81802.0	184.273720
EWR	84991.0	223.100379

Minneapolis, Detroit, Chicago all major airline hubs. For creating the visualization we will need a few other data sources. Geographic figures for US States, and latitude and longitude locations for major airports. These data sources are actually included in the altair/vega example datasets and thus are easily obtained using the input at the top.

In [14]:

us_states =  alt.topo_feature(data.us_10m.url,  'states')

The following chart will provide a base United States map for the visualization.

In [15]:

usa_chart = alt.Chart(us_states).mark_geoshape(fill='#dedede', stroke='white')
usa_chart

Out[15]:

We can see that looks a little funny. Because the standare projection is a world-mercator, we are seeing the map based on the realtive positions of Alaska and Hawaii. This is something we will correct later.

In [16]:

airports = pd.read_csv(data.airports.url)

In [17]:

merged_airport_data = airports.join(mco_totals_by_airport, on='iata',   how='inner')
# data_json = merged_airport_data[merged_airport_data.Passengers > 5000].to_json()
# data_to_chart = alt.Data(values=json.loads(data_json)['features'])

We are also going to filter the data to just focus on large airports (with over 10,000 passengers of traffic) in the 50 states.

In [18]:

key_airports = merged_airport_data[(merged_airport_data.Passengers > 10000) & 
                                   (merged_airport_data.state != 'PR')]
key_airports.head()

Out[18]:

	iata	name	city	state	country	latitude	longitude	Passengers	MktFare
771	ACY	Atlantic City International	Atlantic City	NJ	USA	39.457583	-74.577167	16133.0	88.635223
825	ALB	Albany Cty	Albany	NY	USA	42.748119	-73.802979	20567.0	204.663398
880	ATL	William B Hartsfield-Atlanta Intl	Atlanta	GA	USA	33.640444	-84.426944	55434.0	176.386318
890	AUS	Austin-Bergstrom International	Austin	TX	USA	30.194533	-97.669872	17309.0	193.862941
933	BDL	Bradley International	Windsor Locks	CT	USA	41.938874	-72.683228	34742.0	188.418242

In [19]:

airport_chart = alt.Chart(key_airports[['name', 'latitude','Passengers','longitude']]).mark_point(
    fill='steelblue',stroke='white', filled=True).encode(
tooltip='name:N', size='Passengers:Q', latitude='latitude:Q', longitude='longitude:Q')

In [20]:

airport_chart

Out[20]:

Great, this shows us the airports sized by their traffic to Orlando. Like with our us map it looks a little funny but we will fix it at the end. We can add one more element which is lines representing the routes between the cities. The code below creates columns for the destination (which is Orlando in each case). So we set them to a constant.

In [24]:

key_airports['dest_latitude'] = 28.428889
key_airports['dest_longitude'] = -81.316028

We are also going to give each route a name so that we can instruct the visualization to draw one line per route.

In [25]:

key_airports['route_name'] = merged_airport_data.iata + '_' + 'MCO'

Still, the data is formatted incorrectly for a line chart. Line charts work two ways, one connects multiple points into continuous line segments. The other way draws multiple points but draws multiple lines in a series which is what we want. So we want to arrange the data so that the origin and destination points are in the same column vertically with 'route_name' identifying which line is whice. The code below creates that structure by concatanting the origin points with the destination points in the same column

In [26]:

line_airport_points = pd.concat((key_airports[['route_name','latitude','longitude','Passengers']]
          , key_airports[['route_name','dest_latitude','dest_longitude','Passengers']].rename(
              columns={'dest_latitude':'latitude', 'dest_longitude':'longitude'})
          )
         )
line_airport_points.head()

Out[26]:

	route_name	latitude	longitude	Passengers
771	ACY_MCO	39.457583	-74.577167	16133.0
825	ALB_MCO	42.748119	-73.802979	20567.0
880	ATL_MCO	33.640444	-84.426944	55434.0
890	AUS_MCO	30.194533	-97.669872	17309.0
933	BDL_MCO	41.938874	-72.683228	34742.0

In [27]:

routes = alt.Chart(line_airport_points).mark_trail(strokeOpacity=.5).encode(
    latitude='latitude', 
    longitude='longitude', 
    color=alt.Color('Passengers', legend=None, scale=alt.Scale(scheme='purples')), 
    size=alt.Size('Passengers:Q', bin=True, scale=alt.Scale(range=(1,2)), legend=None),
    tooltip='route_name')

In [28]:

routes

Out[28]:

The use of mark_trail allowed us to size each line. Now all that's left is to put everything together and add a change to the projection which allows us see everything clearly in a US Centric map.

In [29]:

usa_map_combined = alt.layer(usa_chart, airport_chart,  routes, 
                            width=600, height=600).project(type='albersUsa')
usa_map_combined

Out[29]:

Comparing Multiple Years¶

There we have the legions of HIT vendors streaming through American airport to descend on Orlando. But how does it compare to non-HIMSS years. For that, we will use the 2014 2015 and 2017 data.

We will set up a Dask pipeline that will load the data for each year and give the totals arriving in Orlando for each. This where dask really shines as its essentially chunking through over 5gb of data rows across 5 files.

In [31]:

from importlib import reload
reload(altair_theme)

base = alt.Chart(mco_totals_by_year.reset_index()).encode(
x='Period:O')

passengers = base.mark_bar(size=20).encode(
y=alt.Y('Passengers:Q', axis=alt.Axis(format=",.0f", title='Number of Passengers'), scale=alt.Scale(domain=(0,600000))),
color=alt.Color('HIMSS:N', scale=alt.Scale(range=['#7e78d2', '#fb62f6'], domain=['No HIMSS','HIMSS']))
)

fares = base.mark_line().encode(
y=alt.Y('MktFare:Q', axis=alt.Axis(format='$.0f', title='Average Fare'),scale=alt.Scale(domain=(150,300)))
)

stats_chart = alt.layer(
    passengers,
    fares,
    width=600,
    height=100
).resolve_scale(
    y='independent'
)

In [32]:

final_map = alt.vconcat(usa_map_combined, stats_chart, title="Passenger Traffic Headed to Orlando 2014-2018").configure_title(
fontSize=20, anchor='middle')
final_map

Out[32]:

In summary, HIMSS' 50,000+ attendees don't even appear as a blip for the greater Orlando area. At least each year the avg fare does seem to be getting a little lower.

In [33]:

usa_chart.save('static/flights/usa-chart.html')
airport_chart.save('static/flights/airport_chart.html')
routes.save('static/flights/routes.html')
usa_map_combined.save('static/flights/usa_map_combined.html')
final_map.save('static/flights/final_map.html')

In [ ]:

Using Applescript

2019-02-02T00:00:00-05:00

I am not the type of person who is constantly looking to optimize everything. Profiles of designers and product managers often start with descriptions about their eye being caught by the most minor of inefficiency that they seek to smooth out. I have come late to that skill. Such is my dabbling with Applescript recently.

For those of you who are not aware, their exists a hidden programming language in macOS that purports to allow to extend the operating system to do any of a number things automatically. Applescript has a very cute but often confounding syntax that can occasionally resemble natural english but often in a way that is unexpected. Here is the classic "Hello World" example for Applescript:

say “hello world”

This will have your Mac say "Hello World" outloud. See it's cute. I encourage you to read the other Hello World examples at the blog post linked above. So why am I digging up this very old part of macOS? Because I recently used it to solve one of my more frustrating workflows. We use Box.Com to store and collaborate on documents at work. But, one of the frustrations of working with it are the constantly sharing of non specific links:

Clicking this link will open the document in the Box online viewer, which creates remarkably good renderings of most file types, but won't allow to edit or access the document. Even more frustrating, I generally already have this document on my machine downloaded automatically to a folder on my machine through Box's Sync application. Lastly, I have no idea about the surrounding context of this document: is it a folder called "drafts"? or a folder called "CRITICAL"?

With Applescript, I created a "service" which takes in any file selected in the finder and copies path to that file on box. The script looks like this:

on run {input, parameters}

    set Txt1 to POSIX path of first item of input

    set output to {}

    set AppleScript's text item delimiters to {"/"}

    set delimited to every text item of Txt1

    set output to items 5 through length of delimited
    return output as string

end run

Apple's services which executes this script always begins with the same on run . . . end run sequence with input and parameters. Here is what the service does line by line:

Create a variable Txt1 and set it to the path corresponding to the first element in the input array. Since the service expects a file as input that file will have a known path which we will manipulate.
Create an output variable, blank for now
Tell applescript to use / as a delimiter. Meaning when it looks at Txt1 it will break it up into parts at each / which is the delimiter in a POSIX path.
We then create a new list of all the elements using this delimiter.
We take items 5 through the end of the list and return it.

This creates the following really easy link to use:

Which I then paste into e-mail or slack. I've also created the reverse script:

on run {input, parameters}

    set theUnixPath to input
    set theUnixPath to (POSIX path of (path to home folder)) & "Box Sync/" & theUnixPath
    set theMacPath to (POSIX file theUnixPath)
    tell application "Finder"
        set theItem to item theMacPath
        if (class of theItem) is folder then activate
        reveal theItem
    end tell
    activate application "Finder"
end run

This tells finder to open the folder where the document resides. I have rolled out these scripts across the company and they were a great first foray into Applescript.

If you want to run these scripts, you use the Automator app to create the service. Services can then be registered to shortcuts in Settings. For me, ⌘+⌥+b copies the Box path to the item, ⌘+⌥+o opens the item.

Now reading Lilith's Brood by Octavia Butler

2019-02-01T00:00:00-05:00

This one book weaves together so much about groupthink, knowledge, and destiny. Has been great so far.

Calculating Obesity from FHIR Data

2019-01-30T00:00:00-05:00

When parents bring their kids to the pediatrician, one of the most important checks is to have the childs heigh, weight, and BMI plotted on growth charts to understand their development. The charts, a screen of which appears below, are ubiquitious.

Sure enough, many many EHRs automatically show the graph and the patient's position on the chart right there making it easy for physicians and patients to review. They included automated features to measure growth velocity etc. However while the EHR stores the BMI, height, and weight measurements, Most systems do not store the comparative measure of those values for the child's age. This leaves analysts and data scientists in a tricky position when it comes to estimating obesity. How do we identify children who may be obese absent this information?

In meetings just last week, a reporting team was lamenting the lack of obesity related ICD codes for pediatrics in their population and lamenting that they could not pull BMI for age percentile from their EPIC EHR system. This is no slight against the EHRs. These calculations are calculated based on outside references (provided by the WHO or CDC) they are intrepretations and aren't on their own a vital sign or part of the record.

In this post, we are going to calculate the BMI percentiles for 2000 sample patients. The patient population is a synthetic population of 2,026 pediatric patients (aged 0-20 years) created using the amazing Synthea tool. So we will be reading these patient's charts in the form of FHIR documents directly. In the process we will do the following:

Identify each patient's most recent BMI and their age in months on the date that BMI was taken. We will do this using a synthetic population of FHIR files.
Use the CDC's reference tables to identify the nearest percentile for that patient's measurement from the previous step.
Create a population level chart that shows

Let's start by setting up libraries we will need.

In [21]:

import json
import fhirclient.models.bundle as b
import fhirclient.models.patient as p
import fhirclient.models.observation as o
from pathlib import Path
from decimal import *
import pandas as pd
import matplotlib as plt
from datetime import date, timedelta, datetime
import csv
import altair as alt
import numpy as np
import seaborn as sns
alt.renderers.enable('notebook')

%matplotlib inline

We have posted a zip file of the 2,026 FHIR patient bundles that you can use to run this notebook. Unzip the file to the data/directory to continue.

Below, we will verify that you have some FHIR files to run.

In [6]:

p = Path('data/fhir')
files_to_load = [f for f in p.glob('*.json')]

if len(files_to_load) == 0:
    raise StopIteration("You don't have any FHIR files")
else:
    print("Loading {0} FHIR Files".format(len(files_to_load)))

Loading 2026 FHIR Files

We will also load as constants the CDC data tables that contain the percentile values for BMI at various ages.

In [7]:

def getBMITable(sex):
    if sex == 'M':
        file = 'data/male-2-20.csv'
    elif sex == 'F':
        file = 'data/female-2-20.csv'
    with open(file, 'r', encoding='utf-8-sig') as csvfile:
        reader = csv.DictReader(csvfile, quoting=csv.QUOTE_NONNUMERIC, fieldnames=['AgeMonths',3,5,10,25,50,75,85,90,95,97])
        bmi_table = [row for row in reader]
    return bmi_table

MALE_BMI_TABLE = getBMITable('M')
FEMALE_BMI_TABLE = getBMITable('F')

The functions below uses the SMART Python FHIR Client to parse the FHIR bundles and obtain the elements we will need:

the patients date of birth
their sex
all of their BMI observations and the values

In [8]:

def getBMIObeservation(entry):
    if entry.resource.resource_type == 'Observation' and entry.resource.code.coding[0].code == '39156-5':
        return (entry.resource.effectiveDateTime.date, entry.resource.valueQuantity.value)
    else:
        return None

def getDateOfBirth(entry):
    if entry.resource.resource_type == 'Patient':
        return entry.resource.birthDate.date
    
def getSex(entry):
    if entry.resource.resource_type == 'Patient':
        return entry.resource.gender
    
def getAgeMonths(dob, visitdate):
    age_in_months = (visitdate.date()-dob)/timedelta(days=30.0)
    return age_in_months

Now we will tie all of this together as follows:

In [9]:

def getLatestBMI(bundle):
    # Find any obersvations with a BMI
    possible_visits_with_bmis = [getBMIObeservation(entry) for entry  in bundle.entry if getBMIObeservation(entry) ]
    if possible_visits_with_bmis:
        visit_date, bmi = max(possible_visits_with_bmis, key=lambda x:x[0]) # Get us the latest BMI
    else:
        return None #no possible BMIs 
    #Everyone's birthdate should be same but there is a possibility that this is notated several times in a FHIR bundle
    birthDate = max([getDateOfBirth(entry) for entry in bundle.entry if getDateOfBirth(entry) ])
    age = getAgeMonths(birthDate, visit_date) #uses the visit date from above
    if 24.0 < age > 240.0:
        return None #patients older than 20 and below 2 don't need  this measurementt
    sex = max([getSex(entry) for entry in bundle.entry if getSex(entry) ])
    if sex == 'male':
        bmi_table = MALE_BMI_TABLE
    else:
        bmi_table = FEMALE_BMI_TABLE
    
    age_row = min([row["AgeMonths"] for row in bmi_table if row["AgeMonths"] > age ])
    #Get their BMI from the CDC tables
    bmi_pct_row_values = ([row for row in bmi_table if row['AgeMonths'] == age_row][0]).copy()
    bmi_pct_row_values.pop('AgeMonths')
    bmi_pct_row_values[100] = 100 #this is the maximum case 
    pct = min([percentile for (percentile, bmi_value) in bmi_pct_row_values.items() if bmi <= bmi_value])
    
    
    return {'ageMonths':int(age),'BMI':bmi, 'BMIpct':pct, 'sex':sex}

Now for each file we have, we will run the functions we defined and save the results. This can take up to 2 minutes for all the files

In [10]:

bmi_results = []

for f_json in files_to_load:

    try:
        with open(f_json, 'r') as jsonfile:
            json_results = json.load(jsonfile)
    except:
        continue
    bundle = b.Bundle(json_results) 
    result = getLatestBMI(bundle)
    if result:
        bmi_results.append(result)

In [11]:

len(bmi_results)

Out[11]:

We can now use a Pandas dataframe and seaborn to see how our population looks.

In [12]:

df = pd.DataFrame(bmi_results)
df.dtypes
df['ageYears'] = (df.ageMonths/12).astype(int)

In [13]:

# data_to_plot = df.sort_values('ageMonths').groupby(['ageMonths', 'sex']).agg({'BMI':np.mean, 'BMIpct':np.median})
data_to_plot = df.sort_values('ageMonths').reset_index()

The seaborn charts below plot the observed BMIs at the 10th, 25th, 50th, 75th, and 90th percentile. You can see the fitted polynomial regressions (which are smooth lines) start to appear a lot like the CDC growth chart.

In [24]:

lm = sns.lmplot(data=data_to_plot[(data_to_plot.BMIpct.isin((10,25,50,75,90)) )], 
           x='ageMonths', y='BMI', hue='BMIpct', col='sex', order=3 )
axes = lm.axes
axes[0,0].set_ylim(12,25)
axes[0,1].set_ylim(12,25)

axes[0,0].set_xlim(20,250)
axes[0,1].set_xlim(20,250)
lm.fig.subplots_adjust(top=0.9)
t = lm.fig.suptitle("BMI Growth Charts for Males and Females from FHIR Messages")

Lastly, we can view our population averages across both sexes.

In [15]:

agg_data_to_plot = df.groupby('ageYears').mean().reset_index()
sns.lmplot(data=agg_data_to_plot, x='ageYears', y='BMI', order=3)

Out[15]:

<seaborn.axisgrid.FacetGrid at 0x11d565cf8>

Other Resources¶

This is great fun but not something that should be used in production systems. There is a great python package PyGrowUp that more accurately calculates Z-Scores which are the better statistical method for tracking a patient's deviation from the population. In addition, the CDC has published a very nice SAS program that helps get the official formulation for calculating both he Z-score and percentile. These packages also support the height, and weight measurements for children under than two.

Still, we hope you find this helpful especially if you are trying to measure obesity in pediatric populations.

Now reading Snow Flower and the Secret Fan by Lisa See

2019-01-21T00:00:00-05:00

I am not sure how Lisa See does it but her books (i have read two) are amazingly engrossing. This one ran long but still lingers with me weeks later.

Now reading Bring up the Bodies by Hilary Mantel

2019-01-10T00:00:00-05:00

After about 10 months on the waiting list, I finally got Hillary Mantel's amazing sequel in the Thomas Cromwell series.

Welcome

2019-01-01T00:00:00-05:00

In 2019, it can seem antiquated to have a blog. A running stream of articles, authored by yourself that to achieve volume needs to cover everything from the mundane to the less mundane. Still if this serves as anything more than simply allowing me to organize my thoughts and to introduce myself to you then it will be well worthwhile.

Starting three years ago, I needed a way to interactively demonstrate an interface with the data warehouse for the product suite I helped to manage. That need led me Jupyter and my first interactions with interactive programming, literally weaving code and data together to tell a story. That got me hooked so much so, I started a company to evangelize that through out healthcare.

Over time, I learned a few tricks developed into what could be termed a data scientist. I had to relearn statistics from the ground up. Learn programming across R, Python, Javascript (limited) and also grew out a focus on helping introduce unexpected data to the people who need it.

If that sounds like something you might like, keep on reading. I will keep them coming.

Now reading the Necessary Death of Lewis Winter by Malcom MacKay

2018-12-15T00:00:00-05:00

Really cool crime book taking place in Scotland. Each character instantly recognizable excited for sequel.