Online: | |
Visits: | |
Stories: |
Story Views | |
Now: | |
Last Hour: | |
Last 24 Hours: | |
Total: |
About a year ago, Sunlight, EFF, and over 150 civic hackers reverse-engineered Congress’ contact forms to create Contact-Congress – an open-source project in the @unitedstates organization on GitHub. This project, as previously highlighted, aims to codify (in an open data format) the steps necessary for submitting electronic messages to any member of Congress to open the door for the development of diverse tools and applications for facilitating dialog with representatives. Indeed, if you’ve emailed your representatives on OpenCongress then you’ve already benefited from the project. Its inception and continued upkeep play an important role in minimizing barriers to constituent communication while promoting a more accessible and open government.
Over the past year we’ve received feedback for our implementation of Email Congress and have been working to identify and address the issues and concerns of our users. In the coming weeks we will be deploying a revamp of the project that will significantly improve the entire experience of sending emails to members of congress. This post will detail some of the technical highlights of our implementation so others may learn more about our development process and what it takes to deploy an email service for Congress. For reference, you can view the open source code here.
Overview
As the GitHub description reads, the project is a “Lightweight Flask web app to courier email messages to phantom-of-the-capitol using postmark.” It works by stitching together a number of APIs and user input for the ultimate objective of transforming an email message into an automated submission of a congressional webform. There are a number of steps and challenges along this road so perhaps tracing the course of a typical user email would best illustrated how each one is overcome.
Step One: User sends an email
The user sends an email to a particular member of congress at an @opencongress address such as [email protected] or [email protected]. Sunlight’s email accounts are managed by Google so we set up a filter to forward messages to our private inbound postmark address as shown below.
The postmark account is then configured with an inbound webhook to make a post request to our web app each time an inbound event occurs (when gmail forwards the email). The url we provide here is simply the endpoint /email/postmark/inbound for our app which is set up to process the postmark JSON data containing the email’s subject, body, and metadata. This request is secured by basic access authentication to prevent unauthorized data from reaching our app.
Step Two: Reaching our app
If all goes well then the email data should reach our app. At this point if you’re not familiar with postmark then you can skim over their Email API documentation to see an example of how the email data is structured as JSON. We first make use of postmark-inbound-python project to convert the JSON into a Python object, which allows us to interact with field data more conveniently. Next, a number of things may occur depending on whether the user has previously registered, is exceeding rate limits, or is a first time user. For completeness, let’s continue with the pathway of first time users.
We can’t immediately submit messages to congress from first time users because congressional webforms require address and contact information to verify that the message is indeed coming from a constituent. Thus first time users will receive a reply to their original email welcoming them and asking them to click a link to complete a registration process where we gather this information. We make use of the python-postmark project to handle routing emails through postmark. In order to thread the emails properly, we make use of the email headers ‘In-Reply-To’ and ‘References’ as well as the traditional subject ‘Re: ‘ prefix as email clients can vary in level of scrutiny necessary for threading. You can see a code example for how we accomplish constructing such a postmark email below.
return PMMail(api_key=settings.POSTMARK_API_KEY, sender=cls.SENDER_EMAIL, to=user.email, subject='Re: ' + msg.subject, html_body=render_without_request("emails/validate_user.html", context={'verification_link': veri_link, 'user': user} ), track_opens=True, custom_headers={ 'In-Reply-To': msg.email_uid, 'References': msg.email_uid, } )
Step Three: Collect data and determine district
The user clicks the link in the reply email and are taken to a page where we collect necessary contact and address information including: prefix title, first name, last name, address, city, state, zip, and phone number. With this information we are able to use Sunlight’s Congress API to determine the user’s congressional district. If a zip code overlaps multiple districts then we make use of Texas A&M Geoservices to obtain a latitude/longitude via geolocation, which we can then pass to the Congress API to definitively determine a congressional district. The accuracy of this step is imperative as congressional webforms will ultimately reject messages if the provided address information indicates the sender resides outside the corresponding district.
Once we have a state and congressional district then we can match the user with their senators and representative. Sometimes this won’t match with the original intended recipient as congressional district boundaries change fairly frequently or the user may be trying to email a member of congress that doesn’t represent them. We thus provide as many explainers as possible to help the user understand how we’ve matched them and why they can only message certain representatives. The user can then decide if they wish to continue by selecting which representatives they wish to send their message and clicking a send button.
Step Four: Automate form submission
If the user decides to send their message then a job is passed to Celery using Redis as the broker. The job is responsible for making an API call to a phantom-of-the-capitol instance with the email contents, address and contact information, and desired recipients. The phantom-of-the-capitol instance handles submitting the data via simulating user input with a headless browser such as PhantomJS. This is a huge upgrade from our original form-submission implementation, formageddon, because it allows navigation of webforms that require JavaScript, whereas formageddon was built on top of Mechanize and couldn’t process JavaScript. This should allow for higher success rates in form submissions as congressional websites have modernized considerably and many are starting to require JavaScript to submit their forms.
In some cases a message will be unable to submit successfully due to unexpected technical complications such as website maintenance or server downtime. In these cases, the user will be notified by email of the situation and we will provide links to forward them to the contact form of the intended recipient. Otherwise, the user will receive an email notification that their message was successfully sent.
Step Five: Freely message members of congress
Now the user is signed up and can freely send emails to their members of congress without going through the signup process again. If a user moves and wishes to change their address information then there is a link at the bottom of every email notification that will take them to a page that will allow them to change their address. This makes it so the entire registration and authentication system is a lightweight email-based using unique URL tokens. No need to keep track of yet another username and password!
Final Remarks
Hopefully this post sheds light on how our new system of emailing congress will work overall and all the moving parts propelling each step. As the reader should see, transforming an email message into a congressional webform submission isn’t a trivial task and relies on a number of APIs and open source projects along the way. Truly appreciative for all the great work and people who provide these resources.
Perhaps some day Congress will release a more convenient means of contacting representatives, but until then, keep your eyes peeled for the launch of our revamped Email Congress project in the coming weeks!
The Sunlight Foundation is a non-profit, nonpartisan organization that uses the power of the Internet to catalyze greater government openness and transparency, and provides new tools and resources for media and citizens, alike.