What is it

It is a custom Django command to execute the load_latlngJSON.py script.

Main purpose of this script is to create a dictionary in the form a JSON object to proivde O(1) lookup for latitudinal and longitudinal information of northeast and southwest corners of a country. This information is needed to dynamically adjust viewport of map as multiple countries are selected.

This command is used to execute a scripts that parses all countries listed on our project‘s cameras webpage, uses geocode API to obtain viewport of each country, and then creates a JSON file of country codes mapped to country viewports.

Why it was needed

See this issue for an in-depth explanation.

A brief summary: this script is a necessary solution to

  1. Limit calls to geocode API: there‘s a limit to number of free requests to geocode API which would have been undoubtedly exceeded in times of relatively heavy traffic to our website.
  2. Increase speed performace of our website‘s cameras page: geocoder API requests may take several hundered milliseconds and this results in a subpar user experience when multiple countries are selected, in fact, dynaymically adjusting viewport when more than 3 countries are selected becomes infeasbile

Executing the script

python manage.py load_latlngJSON

This will print relevant message to to the console for each country as it is successfully, or not, geocoded. If this is not desired then the output can be piped to a dummy file where it can be examined.

python manage.py load_latlngJSON > dummy.txt

The script can run for potentially more than 2 minutes, due to reasons explained here and here.


The script was setup as per the official tutorial on how to create a custom Django command.

The path to the script: /app/management/commands/load_latlngJSON.py


Each time the script is executed, the handle member funtion of the Command class is called

    def handle(self, *args, **options):
        countries = self.get_countries_from_webpage()
        countries = self.geocode_data(countries)

        with open('app/static/app/js/countries_viewport.json', "w") as writeJSON:
            json.dump(countries, writeJSON)

Using Beautiful Soup to parse webpage

As explained in the CAM2 Training docs, Beautiful Soup was used to parse the list of countries for our website‘s cameras page.

This webpage from stanford.edu is also an excellent quick guide to Beautiful Soup

import urllib.request, urllib.parse, urllib.error
import ssl
from bs4 import BeautifulSoup
    def get_countries_from_webpage(self):
        #ignore SSL certificate errors
        ctx = ssl.create_default_context()
        ctx.check_hostname = False
        ctx.verify_mode = ssl.CERT_NONE

        #get country select tag
        url = "http://www.cam2project.net/cameras/"
        html = urllib.request.urlopen(url, context=ctx).read()
        soup = BeautifulSoup(html, 'html.parser')
        country_menu = soup.find('select', id="country").find_all('option')

        countries = {}
        for country in country_menu:
            if country['value']:
                countries[country['value']] =  str(country.text)

        return countries

Using geocode API

Google‘s geocode API was used to obtain viewports of countries listed on our website‘s drop down menu.

These viewports are nested dictionaries containing information about the latitudinal and longitudinal positions of the northwest and southwest corners of a country.

This tutorial on Github was referred to as a guide to use geocode API in python.

The geocoding function takes a dictionary of country codes mapped to country names as inputs and makes requests using country names to obtain viewports. Each country‘s code is then mapped to country‘s viewport. The resulting dictionary is then returned.

import requests
from time import sleep
    def geocode_data(self, countries):
        GOOGLE_MAPS_API_URL = 'http://maps.googleapis.com/maps/api/geocode/json'

        for country_code, country_name in countries.items():
            #to enfore geocode 50 queries per second limit

            #make the request and get the response data
            req = requests.get(GOOGLE_MAPS_API_URL, params= {
                'address': country_name
            res = req.json()

            if(res['status'] != 'OK'):
                print(country_code, country_name)
                req = requests.get(GOOGLE_MAPS_API_URL, params={
                    'address': country_name
                res = req.json()
                if (res['status'] != 'OK'):
                    print("Failed 2nd attempt: ", country_code, country_name)
                print("OK: ", country_code, country_name)

            #Use the first result
            result = res['results'][0]

            countries[country_code] = result['geometry']['viewport']

        return countries

Creating JSON file

Once the nested dictionary of country codes mapped to country viewports is obtained, ‘json‘ library is used to create a JSON file of this dictionary in the appropriate directory:

import json
with open('app/static/app/js/countries_viewport.json', "w") as writeJSON:
    json.dump(countries, writeJSON)