# Nearest Neighbors Lab

### Introduction

In this lab, you apply nearest neighbors technique to help a taxi company predict the length of their rides. Imagine that we are hired to consult for LiftOff, a limo and taxi service that is just opening up in NYC. Liftoff wants it's taxi drivers to target longer rides, as the longer the ride the more money it makes. LiftOff has the following theory:

• The pickup location of a taxi ride can help predict the length of the ride.

LiftOff asks us to do some analysis to write a function that will allow it to predict the length of a taxi ride for any given location.

Our technique will be the following:

• Collect Obtain the data containing all of the taxi information, and only select the attributes of taxi trips that we need
• Explore Examine the attributes of our data, and plot some of our data on a map
• Train Write our nearest neighbors formula, and change the number of nearby trips to predict the length of a new trip
• Predict Use our function to predict trip lengths of new locations

### Collect and Explore the data

#### Collect the Data

Luckily for us, NYC Open Data collects information about NYC taxi trips and provides this data on its website. For your reading pleasure, the data has already been downloaded into the trips.json file in this lab which you can find here. We'll use Python's `json` library to take the data from the `trips.json` file and store it as a variable in our notebook.

``````import json
# First, read the file
trips_file = open('trips.json')
# Then, convert contents to list of dictionaries
trips = json.load(trips_file)
``````

Press shift + enter

#### Explore the data

The next step is to explore the data. First, let's see how many trips we have.

``````len(trips)
``````

Not bad at all. Now let's see what each individual trip looks like. Each trip is a dictionary, so we can see the attributes of each trip with the `keys` function.

``````trips.keys()
``````

#### Limit our data

Ok, now that we have explored some of our data, let's begin to think through what data is relevant for our task.

Remember that our task is to use the trip location to predict the length of a trip. So let's select the `pickup_latitude`, `pickup_longitude`, and `trip_distance` from each trip. That will give us the trip location and related `trip_distance` for each trip. Then based on these actual trip distances we can use nearest neighbors to predict an expected trip distance for a trip, provided an actual location.

Add in about trip distance

Write a function called `parse_trips(trips)` that returns a list of the trips with only the following attributes:

• `trip_distance`
• `pickup_latitude`
• `pickup_longitude`
``````def parse_trips(trips):
pass
``````
``````parsed_trips = parse_trips(trips)
parsed_trips and parsed_trips

# {'trip_distance': '18.379999999999999',
# 'pickup_latitude': '40.64499',
# 'pickup_longitude': '-73.781149999999997'}
``````

Now, there's just one change to make. If you look at one of the trips, all of the values are strings. Let's change them to be floats.

``````def float_values(trips):
pass
``````
``````cleaned_trips = float_values(parsed_trips)
``````
``````cleaned_trips

# {'trip_distance': 18.38,
# 'pickup_latitude': 40.64499,
# 'pickup_longitude': -73.78115}
``````

### Exploring the Data

Now that we have paired down our data, let's get a sense of our trip data. We can use the `folium` Python library to plot a map of Manhattan, and our data. First we must import `folium`, and then use the `Map` function to pass through a `location`, and `zoom_start`. If a map isn't showing up below, copy and paste the command `pip install folium` into your terminal to install `folium` then try again.

``````import folium
manhattan_map = folium.Map(location=[40.7589, -73.9851], zoom_start=11)
``````
``````manhattan_map
``````

Ok, now let's see how we could add a dot to mark a specific location. We'll start with Times Square.

``````marker = folium.CircleMarker(location = [40.7589, -73.9851], radius=10)
marker.add_to(manhattan_map)
``````

Above, we first create a marker. Then we add that circle marker to the `manhattan_map` we created earlier.

``````manhattan_map
``````

Do you see that blue dot near Time's Square? That is our marker.

So now that we can plot one marker on a map, we should have a sense of how we can plot many markers on a map to display our taxi ride data. We simply plot a map, and then we add a marker for each location of a taxi trip.

Now let's write some functions to allow us to plot maps and add markers a little more easily.

#### Writing some map plotting functions

As a first step towards this, note that the functions to create both a marker and map each take in a location as two element list, representing the latitude and longitude values. Take another look:

``````marker = folium.CircleMarker(location = [40.7589, -73.9851])
manhattan_map = folium.Map(location=[40.7589, -73.9851])
``````

So let's write a function called to create this two element list from a trip. Write a function called `location` that takes in a trip as an argument and returns a list where the first element is the latitude and the second is the longitude. Remember that a location looks like the following:

``````first_trip = {'pickup_latitude': 40.64499, 'pickup_longitude': -73.78115,  'trip_distance': 18.38}
first_trip
``````
``````def location(trip):
pass
``````
``````first_location = location(first_trip) # [40.64499, -73.78115]
first_location # [40.64499, -73.78115]
``````

Ok, now that we can turn a trip into a location, let's turn a location into a marker. Write a function called `to_marker` that takes in a location (in the form of a list) as an argument, and returns a folium `circleMarker` for that location. The radius of the marker should always equal 6.

``````def to_marker(location):
pass
``````
``````import json
times_square_marker = to_marker([40.7589, -73.9851])

times_square_marker and times_square_marker.location # [40.7589, -73.9851]
``````
``````times_square_marker and json.loads(times_square_marker.options)['radius'] # 6
``````

Ok, now that we know how to produce a single marker, let's write a function to produce lots. We can write a function called `markers_from_trips` that takes in a list of trips, and returns a marker object for each trip.

``````def markers_from_trips(trips):
pass
``````
``````trip_markers = markers_from_trips(cleaned_trips)
``````
``````cleaned_trips[0:4]
``````
``````trip_markers and len(trip_markers) # 1000
``````
``````list(map(lambda marker: marker.location, trip_markers[0:4]))
# [[40.64499, -73.78115],
#  [40.766931, -73.982098],
#  [40.77773, -73.951902],
#  [40.795678, -73.971049]]
``````

Ok, now that we have a function that creates locations, and a function that creates markers, it is time to write a function to plot a map.

Write a function called `map_from` that, provided the first argument of a list location and second argument an integer representing the `zoom_start`, returns a `folium` map the corresponding location and `zoom_start` attributes.

Hint: The following is how to write a map with folium:

``````    folium.Map(location=location, zoom_start=zoom_amount)
``````
``````def map_from(location, zoom_amount):
pass
``````
``````times_square_map = map_from([40.7589, -73.9851], 15)
times_square_map and times_square_map.location # [40.7589, -73.9851]
``````
``````times_square_marker and times_square_marker.add_to(times_square_map)
times_square_map
``````

Now that we have a marker and a map, now let's write a function that adds a lot of markers to a map. This function should add each marker in the list to the map object then return the updated map object.

``````manhattan_map = map_from([40.7589, -73.9851], 13)
``````
``````def add_markers(markers, map_obj):
pass
``````
``````map_with_markers = add_markers(trip_markers, manhattan_map)
``````
``````map_with_markers
``````

### Using Nearest Neighbors

Ok, let's write a function that given a latitude and longitude will predict the distance for us. We'll do this by first finding the nearest trips given a latitude and longitude.

Here we once again apply the nearest neighbors formula. As a first step, write a function named `distance_location` that calculates the distance in pickup location between two trips.

``````import math

def distance_location(selected_trip, neighbor_trip):
pass
``````
``````first_trip = {'pickup_latitude': 40.64499, 'pickup_longitude': -73.78115, 'trip_distance': 18.38}
second_trip = {'pickup_latitude': 40.766931, 'pickup_longitude': -73.982098, 'trip_distance': 1.3}
distance_first_and_second = distance_location(first_trip, second_trip)

distance_first_and_second and round(distance_first_and_second, 3) # 0.235
``````

Ok, next write a function called `distance_between_neighbors` that adds a new key-value pair, called `distance_from_selected`, that indicates the distance of the `neighbor_trip` from the `selected_trip`.

``````def distance_between_neighbors(selected_trip, neighbor_trip):
pass
``````
``````distance_between_neighbors(first_trip, second_trip)

# {'pickup_latitude': 40.766931,
# 'pickup_longitude': -73.982098,
# 'trip_distance': 1.3,
# 'distance_from_selected': 0.23505256047318146}
``````

Ok, now our `neighbor_trip` has another attribute called `distance_from_selected`, that indicates the distance from the `neighbor_trip`'s pickup location from the `selected_trip`.

Understand the data: Our dictionary now has a few attributes, two of which say distance. Let's make sure we understand the difference.

• `distance_from_selected`: This is our calculation of the distance of the neighbor's pickup location from the selected trip.
• `trip_distance`: This is the attribute we were provided initially. It tells us the length of the neighbor's taxi trip from pickup to drop-off.

Next, write a function called `distance_all` that provided a list of neighbors, returns each of those neighbors with their respective `distance_from_selected` numbers.

``````def distance_all(selected_individual, neighbors):
pass
``````
``````cleaned_trips and distance_all(first_trip, cleaned_trips[0:4])
``````

Now write the nearest neighbors formula to calculate the distance of the `selected_trip` from all of the `cleaned_trips` in our dataset. If no number is provided, it should return the top 3 neighbors.

``````def nearest_neighbors(selected_trip, trips, number = 3):
pass
``````
``````new_trip = {'pickup_latitude': 40.64499,
'pickup_longitude': -73.78115,
'trip_distance': 18.38}

nearest_three_neighbors = nearest_neighbors(new_trip, cleaned_trips or [], number = 3)
nearest_three_neighbors
# [{'trip_distance': 7.78,
#  'pickup_latitude': 40.64483,
#  'pickup_longitude': -73.781578,
#  'distance_from_selected': 0.0004569288784918792},
# {'trip_distance': 12.7,
#  'pickup_latitude': 40.644657,
#  'pickup_longitude': -73.782229,
#  'distance_from_selected': 0.0011292165425673159},
# {'trip_distance': 17.3,
#  'pickup_latitude': 40.648509,
#  'pickup_longitude': -73.783508,
#  'distance_from_selected': 0.0042359798158141185}]
``````

Ok great! Now that we can provide a new trip location, and find the distances of the three nearest trips, we can take calculate an estimate of the trip distance for that new trip location.

We do so simply by calculating the average of it's nearest neighbors.

``````import statistics
def mean_distance(neighbors):
nearest_distances = list(map(lambda neighbor: neighbor['trip_distance'], neighbors))
return round(statistics.mean(nearest_distances), 3)

nearest_three_neighbors = nearest_neighbors(new_trip, cleaned_trips or [], number = 3)
distance_estimate_of_selected_trip = mean_distance(nearest_three_neighbors) # 12.7
distance_estimate_of_selected_trip
``````

### Choosing the correct number of neighbors

Now, as we know from the last lesson, one tricky element is to determine how many neighbors to choose, our \$k\$ value, before calculating the average. We want to choose our value of \$k\$ such that it properly matches actual data, and so that it applies to new data. There are fancy formulas to ensure that we train our algorithm so that our formula is optimized for all data, but here let's see different \$k\$ values manually. This is the gist of choosing our \$k\$ value:

• If we choose a \$k\$ value too low, our formula will be too heavily influenced by a single neighbor, whereas if our \$k\$ value is too high, we will be choosing so many neighbors that our nearest neighbors formula will not be adjust enough according to locations.

Ok, let's experiment with this.

First, let's choose a midtown location, to see what the trip distance would be. A Google search reveals the coordinates of 51st and 7th avenue to be the following.

``````midtown_trip = dict(pickup_latitude=40.761710, pickup_longitude=-73.982760)
``````
``````seven_closest = nearest_neighbors(midtown_trip, cleaned_trips, number = 7)
seven_closest
# [{'trip_distance': 0.58,
#   'pickup_latitude': 40.761372,
#   'pickup_longitude': -73.982602,
#   'distance_from_selected': 0.00037310588309379025},
#  {'trip_distance': 0.8,
#   'pickup_latitude': 40.762444,
#   'pickup_longitude': -73.98244,
#   'distance_from_selected': 0.00080072217404248},
#  {'trip_distance': 1.4,
#   'pickup_latitude': 40.762767,
#   'pickup_longitude': -73.982293,
#   'distance_from_selected': 0.0011555682584735844},
#  {'trip_distance': 8.3,
#   'pickup_latitude': 40.762868,
#   'pickup_longitude': -73.983233,
#   'distance_from_selected': 0.0012508768924205918},
#  {'trip_distance': 1.26,
#   'pickup_latitude': 40.760057,
#   'pickup_longitude': -73.983502,
#   'distance_from_selected': 0.0018118976240381972},
#  {'trip_distance': 0.0,
#   'pickup_latitude': 40.760644,
#   'pickup_longitude': -73.984531,
#   'distance_from_selected': 0.002067074502774709},
#  {'trip_distance': 1.72,
#   'pickup_latitude': 40.762107,
#   'pickup_longitude': -73.98479,
#   'distance_from_selected': 0.0020684557041472677}]
``````

Looking at the `distance_from_selected` it appears that our our trips are still fairly close to our selected trip. Notice that most of the data is within a distance of .002 away, so going to the top 7 nearest neighbors didn't seem to give us neighbors too far from each other, which is a good sign.

Still, it's hard to know what distance in latitude and longitude really look like, so let's map the data.

``````midtown_location = location(midtown_trip) # [40.76171, -73.98276]
midtown_map = map_from(midtown_location, 16)
closest_markers = markers_from_trips(seven_closest)

add_markers(closest_markers, midtown_map)
``````

Ok. These locations stay fairly close to our estimated location of 51st street and 7th Avenue. So they could be a good estimate of a trip distance.

``````mean_distance(seven_closest) # 1.26
``````

Ok, now let's try a different location

``````charging_bull_closest = nearest_neighbors({'pickup_latitude': 40.7049, 'pickup_longitude': -74.0137}, cleaned_trips, number = 12)
``````
``````mean_distance(charging_bull_closest) # 3.515
``````

Ok, so there appears to be a significant difference between choosing a location near Times Square versus choosing a location at Wall Street.

### Summary

In this lab, we used the nearest neighbors function to predict the length of a taxi ride. To do so, we selected a location, then found a number of taxi rides closest to that location, and finally took the average trip lengths of the nearest taxi rides to find an estimate of the new ride's trip length. You can see that even with just a little bit of math and programming we can begin to make meaningful predictions with data.

Unlock your future in tech
Learn to code.

## Learn about Flatiron School's Mission ## With a new take on education that falls somewhere between self-taught prodigy and four-year computer science degree, the Flatiron School promises to turn students with little programming experience into developers. ## In the six months since the Manhattan coding school was acquired by WeWork, it has spawned locations in Washington, D.C., Brooklyn, and London. Now, WeWork is opening a fourth Flatiron School location, this time in Houston. 