User:X-Savitar/IFTTT GSoC Report 2016

GSoC 2016 Project Commits on Github

A list of all the commits I made during GSoC on the project in Github can be seen here: Commits.

GSoC 2016 Project Daily Progress Report

Introduction

My name is Derick Ndimnain Alangi(IRC handle: xSavitar), I joined Wikimedia Foundation on Sep 13 2015. Since then, I have been contributing to the movement and part of my work is available on my mediawiki profile page here: My MediaWiki Page or on Wikidata here: My Wikidata Page. In addition to that, I was selected in the GSoC 2016 program to contribute code to the Wikimedia Foundation (WMF). I was selected on the project: "Automated Testing and Integrating IFTTT (IF This Then That) Support for Wikidata" in which you can find the proposal for the project here: Project Proposal. This project is mentored by:

Stephen Laporte (Primary Mentor)
Beneditk Seidl (Co-Mentor)
Marius Hoch (Co-Mentor)
Lydia Pintscher (Co-Mentor & Product Manager, Wikidata)

In addition, I will be executing this project during the Summer and below will be a daily log with respect to my progress in the project starting from Community Bonding to Final Evaluation and Code Submission to Google. You can also follow my progress on the project in my blog: project progress blogspot on my Google Blog Spot in brief.

Community bonding

Week 1 (23nd April - 29th April)

Created Project progress report page (this page) as a sub-page of my profile page to keep track of my work during the Summer of Code and a brief log of this report in blog spot here: project progress blogspot
Created a phabricator conference to enable a synchronized discussion about the project progress with my mentors and notify them with recent updates on the project's progress and any other updates.
Held meetings with some of my mentors individually, having their weekly schedules for code review, guidance etc... This enabled me to get their available times, taking into consideration time zones.
Extending Wikipedia IFTTT to support RSS feeds. Worked on the Article revisions and User revision feeds in addition to the Article of the Day, Word of the Day and New article feeds I already worked on. Commit on github is here: Commit and the pull request for the work here: PR3.
Created a Wikimedia Tool Labs account and submitted to the mentor (Stephen) so that he could integrate me to the dev edition of the Wikipedia IFTTT application on Tool labs. This will enable me to be running Database Access Layer (DAL) tests that is concerned with database driven feeds.
Requested for access to be added to a group on tools lab so that my account can be given rights under a particular project. In this case, testing on Wikipedia IFTTT and development of Wikidata IFTTT during developing the project.
Tool Lab account request was processed and approved by Tim Landscheidt and at this point, it is just left for me to have access to the IFTTT dev project on tools lab for testing.
Stephen (mentor) added me to Tool Labs on the IFTTT-Dev project so that I can perform testing of the RSS feeds work that I have done. I successfully logged into Tool Labs after adding my SSH key to my tool labs profile. Ran DAL (Database Access Layer) tests for feeds that needed DAL for its functioning and all where successful.

Week 2 (30th April - 7th May)

Created a discussion thread here, where my mentors will suggest and decide on the triggers that will be implemented for Wikidata IFTTT web app. Also, I am personally doing some thinking and research on the triggers which will be implemented and will suggest to mentors and if approved, all the decided triggers will be implemented during the coding period.
Further my leaning and investigation on Wikidata API, installed a local instance of the API on my computer and started running local queries on my_wiki database. At this point, I am understanding how the Wikidata API work and following the API's documentation at Wikidata API Docs.
Sent a formal introductory mail to the Wikimedia dev mailing list, Wikidata and Wikidata dev mailing list informing the community members about my acceptance to the GSoC program and also the project I am working on. The links to the mail in the lists are here: Wikimedia-Dev-List, Wikidata-List and Wikidata-Dev-List
Reading documentations about IFTTT concept and also studying in detail Stephen's implementation of IFTTT web app for Wikipedia. Reading docs on RESTful API's and JSON/XML data transfer mechanisms.
Schedule a meeting and sent and email to my mentors including my academic supervisor on my GSoC project and the meeting agenda can be seen here: Meeting Agenda, meeting shall take place on Google Hangout and will be 1hr 30mins maximum.
Setup a hangout on air call and created an event on my YouTube channel where the call can be streamed live by other audience. Here is the link to the video/audio on YouTube: Live Video/Audio.
The meeting held and was very exciting. In this meeting, we had the following points cleared out:
- Modification of time-line to suite the current demands for the project. Once I modify the time-line, I will put a link to it.
- Agreed that we shall investigates all the following listed triggers for Wikidata and rank them to choose the best that will be implemented.
- I shall no longer use Gerrit for development, so I shall use Github and as a result, I shall set-up Travis CI automated testing bot to help me with some testing aspects of the project.
- We also agreed I will also spend some time writing test cases/suite for the current Wikipedia IFTTT web app and also write Wikidata's tests along side building the triggers. In addition to this, I shall also help write documentation for both the current app and the Wikidata app I shall develop.
Modified my GSoC proposal as agreed in the meeting with the mentors. This modification was done both on the Google docs version of the proposal and on the phabricator (T129016) version of the proposal.

Week 3 (8th May - 15th May)

Researched on how to set up Travis-CI bot with Github repositories and also specific to repositories with python projects (any version of python). Read on how to transfer Travis-CI bot setting from one repository to another.
Learned how to use Query module in the Wikidata API, using it in my local instance to run queries again the database. This enabled me to understand how the query module works. Under the query module, I learnt how to use;
- lists
- meta
- generator
- prop
Researched on Unit testing and Integration testing in Python and most especially related to the IFTTT project. Unit testing will be based on the 8 triggers for Wikipedia IFTTT and the triggers I will develop for Wikidata. Also, in addition to the Unit tests, I shall also write integration tests for all the triggers in concern.
Practicing how to write unit tests in python using the "unittest" package and also reading how to run unit tests in python using "Nose". This will enable me to write unit tests for the IFTTT project.
Pushed a mini project on Github where I am using to practice my unit testing skills in python. The link to the repository can be found here: Math Functions & Unit Tests
Had in total of 3 meetings with the mentor with each greater than 1 hours to discuss on how to go about writing the tests. Which properties of the triggers to be tested and other parameters to be tested in the triggers. At the end, we had put in place all the parameters to be tested in triggers.
Researched about how to dynamically generate test cases in python since this will be needed in the project to automatically generate test cases for all the triggers rather than writing a test for each trigger in concern.
Investing the Wikidata list of proposed triggers and ranking them in order for I and the mentors to select the highest priority triggers to implement during the coding period. There are 6 triggers so far and they are going through a ranking process.

Week 4 (16th May - 22nd May)

Finalizing the ranking process of the Wikidata triggers with the mentors.
Wrapping up with the study of the code base and citing some key important core parts of the code that will be related to my GSoC project.
Reading flask docs on client side testing since this will be highly necessary for my project. This will enable me to test the app just withing the application not involving the servers.
Preparing to select triggers that will be implemented based on the ranking done by I and the mentors, but at this point, some mentors have not ranked the triggers so once they are done, triggers will be selected.
More research on client/server side testing of python application. Used the Flask Docs here.
Published my Community bonding meeting (Hangout on Air call with my mentors) on social media and the links can be found below;
- Google+ Link to Video.
- Twitter Link to Video.
- YouTube Link to video.
- etc...
Wrapping up with Community bonding and finalizing all stuffs preparing to start coding tomorrow :)
Let the coding begin !!!!!

First Coding Period

Week 1 (23rd May - 29th May)

Integration of Travis-CI testing bot to my personal repository at the moment for development. This bot will be used for continuous testing and integration of the app. Once we (I and the mentors) decide on the general repository to use for development, I will just transfer the Travis settings to the repo.
Travis-CI is configured to do the following functions;
- Post messages (build related messages) to the Wikidata feed channel on Freenode (#wikidata-feed) so as to notify others on how the development is going (just for monitoring purpose).
- When the build fails or passes, an email is sent to me also for notification.
Here is a link to the Travis file on Github: Travis Settings on Github
I and the mentors have decided a repository that I will be pushing my codes to and it's the official Wikimedia IFTTT repository that can be found here: Wikimedia IFTTT Github app.
Created a GSoC development branch that will be used to push my codes to on the official Wikimedia IFTTT repo. This will enable mentors to review, comment and merge my codes when its ready to the master branch.
- Main Wikimedia IFTTT repo master branch.
- My GSoC Wikimedia IFTTT development branch.

Week 2 & Week 3 (30th May - June 12th)

Wrote unit test for Wikipedia Article of the Day trigger and ran a client test against the trigger and it was working correctly returning the correct result set.
Wrote unit test for Wikipedia New Article trigger and ran a client test against the trigger and it was working correctly returning the correct result set.
Wrote unit test for Wikipedia Word of the Day trigger and ran a client test against the trigger and it was working correctly returning the correct result set.
Wrote unit test for Wikipedia Picture of the Day trigger and ran a client test against the trigger and it was working correctly returning the correct result set.
At this juncture, 4 tests are working correctly (passing) out of the 6 non-database driven tests I have to write. But 2 tests are still failing. Working on these two test and also fixing some few bugs and doing code clean up.
Wrote a Wikidata trigger for Item Revision for which when a particular item in Wikidata is revised, the trigger fires and performs the respective actions.
Developed an RSS View for Item Revisions trigger and an XML template for it for rendering of the data using the browser. The commit can be seen here on my Github account: RSS View Commit
Fixed bug that was causing the non-DB to fail and now all tests, including the test for my new Item Revision trigger is passing. Also, updated the read me file to contain news triggers that are being developed.
Synchronized my online and local development Github branches with the official GSoC development branch in the Wikimedia IFTTT repo and re-committed the whole work I have done, this is because of conflicts that originated when I made a PR to the official Wikimedia IFTTT code base which was different (setup) from that of Stephen that I was using before. The working PR without any conflicts can be seen here: Pull Request

Week 4 (13th June - June 19th)

As instructed by the mentor, I started deploying me work on Wikimedia Tool Labs for integration testing with IFTTT. The mentor added me to a Wikipedia staging channel for me to do integration testing with IFTTT.
The Wikimedia Tool Labs instance I am using is "ifttt-testing" and can be accessed by using this link: IFTTT Test Instance and it has been configured to run my application for testing within the server and also in the Wikipedia staging channel.
- NOTE: Once in a while, we shall have a 404 - Not Found error while trying to access this link since its still unstable as I am working on the server on a daily basis, its a test instance.
Deployed the application on Tool Labs and Integrated to IFTTT Wikipedia Staging to test the functionalities and also the trigger I have developed.
- Trigger was working properly when configured.
- All RSS Views where functioning properly
- All tests where passing on Tool Labs
Fixed Links in the feeds.html file to point to the correct lab instance and no longer "localhost" as before and fixed all the codes to adapt to the configurations of my Lab Instance.

Second Coding Period

Week 5 & 6 (19th June - July 8th)

Submission of Midterm evaluations by both mentors and students. Students evaluate mentors and organisations activities and mentors evaluate their students and both submit evaluations to Google.
Midterm results announced by Google
My midterm evaluations results: Passed
Started my semester exams
Wrote my semester exams for 2 weeks as on the time line.

Week 7 (9th July - July 15th)

Built the RSS view for Database Driven trigger (New Hashtags), Since I have access to the DB now on Tool labs, I can retrieve data concerning this trigger directly from the Database. Here is an example of the RSS: New Hashtag for test hashtag.
Unit test for New Hashtag passes on Tool labs including other tests because of the presence of the DB but on localhost, this test fails and other database driven test because of the absence of the Wikidata database on my localhost.
Fixed the Issue of RSS views working with GET request without checking channel key. The issue here was "if RSS views are working, the integration tests fails on IFTTT or if Integration tests are passing, the RSS views are not working due to the @app.before_request me that checks the channel key before any request". So I added a check to ignore all these on get request using if flask.request.method == 'GET': # do something. At this point, both RSS views and Integration Endpoint tests on IFTTT are working (but some regression occurs, needs to be looked at).
Integration of newly developed trigger "New edit to Wikidata Item" into IFTTT and integration testing. So far, the trigger is working fine and fires when a particular item is edited.
Added Style to the RSS views to improve on UI/UX of the RSS views of the App. For the bootstrap integration into the app, I used the flask.ext.bootstrap and specifically the Bootstrap Module.

Week 8 (16th July - July 22nd)

Debugged IFTTT integration tests together with Stephen LaPorte and all was passing. Integration of Wikidata Edit Item trigger to IFTTT.
Wrote code for GET request to handle parameters for triggers just as the POST request in the app. The commit can be found here: Fixing Regression Commit.
Re-wrote and Improved on the Unit tests for all the triggers found in the ifttt-testing.py file. This improved on the efficiency and getting default trigger values from the trigger.py file and feeds directly into the tests fields. Here are the commits Working Commit - Commit 1 and Efficient and Working commit - Commit 2
After debugging IFTTT integration / endpoint tests with Stephen, this caused a bug in the unit tests which where all not returning data because app.debug was set to True. I debugged this and constrained app.debug to be True only in the context of the FlaskClient tests instance (app.test_client()) so that in the global context, it will be False and the test in the IFTTT integration will all pass. The result of this made both the unit tests and endpoint tests on IFTTT to pass. Here is the commit: Commit ( Shocked :-) )
Added RSS/XML for two last database driven triggers. New category member and Category member revisions, these triggers where challenging to build the RSS since we had to search on Wikipedia categories that are recently edited. It took me some time but finally got it with the help of my mentor (Slaporte). The category we used is 'All articles lacking sources'.
This two triggers (as mentioned above) was failing the unit tests and had no RSS, but after these commits: Commit I and Commit II, the RSS was available and unit tests where passing (all unit tests in the app).
A sample of the output of the RSS of the triggers can be seen here: New category members, Category member revisions.
Item revisions trigger integration tests on IFTTT was failing because of a bug (configuration) that doesn't match the code in the IFTTT app, but together with Stephen on our weekly meeting (21st July, 2016), we debugged the integration tests and the Wikipedia Staging channel passed. Now all tests on IFTTT passes and also all Unit Tests on Tool Labs also pass. Hurrayyy!!! :)

Week 9 (23rd July - July 29th)

Learning the Wikidata Query Services (WDQS) and SPARQL. Experimentation is done here: Wikidata Query Service (WDQS).
Created a Google doc as instructed by my mentors to draft a blog post for the project. The link to the Doc can be found here: Blog post draft.
Thinking and sampling trigger Ideas with the mentors to find the right one to implement for Wikidata using the WDQS and the flask app. So far, we have a list of these trigger ideas:
- Notify me whose birth days is today.
- Notify me when today is a public holiday (we can decide for particular countries) e.g. Notify me when today is a public holiday in the US
- Notify me when my favorite author releases a new book.
Working on the blog post draft on Google docs and developed on the following sections;
- What is Google Summer of Code?
- Why did I apply for GSoC?
- Why the Wikimedia Foundation?
- How has the development of my project looked like?
- Request for help from community members on my project
- Appreciation and Thanks to my mentors and the Wikimedia Foundation
More work on the blog post and addition of content to the various sections
- My recommendations for anyone to participate in GSoC/Wikimedia and Why?
- What I did not anticipate that will happen during GSoC?

Week 10 (30th July - August 5th)

Studying of Wikidata Items (objects) and Wikidata properties to come up with trigger ideas.
More trigger Ideas sent to mentors for evaluation, below are the trigger ideas
- Notify me about the number of new Wikidata items added today.
- Notify me about the number of new Wikidata item properties added today
- Notify me about how many people have birthdays today (could be a very huge number). But this one will be limited to the amount of data in Wikidata (definitely not the whole world).
Worked on the "Experience gained through this project during GSoC" section of my GSoC blog post document and submitted for review to @Lydia_Pintscher.
Developed the "How my experience has affected my Day-to-Day life" of my blog post and submitted to @Lydia_Pintscher for review.
Developed the "About the project and what I have done so far" section of my blog post and submitted for review.
Finished developing the blog post, completed all the remaining parts of the blog and submitted for review.
Blog post finally reviewed and finishing of the blog post. Blog post divided into 3 parts and prepared for publishing. Done with blog :)

Week 11 (06th August - 12th August)

Started building the Birthday trigger view, for the base case, at this level, I limited all the subjects just to 10 and the trigger returned birthdays of entities on Wikidata.
Birthday trigger build using the Wikidata Query Service and SPARQL queries integrated into the flask app.
Created RSS for the birthday trigger which can be found in the popular_persons_birthday.xml template and also added a Unit tests for birthday trigger which is passing on tool labs.
Debugging of IFTTT endpoint tests for birthday trigger.
Birthday trigger endpoint test now passing on IFTTT and also tested it by creating a sample personal recipe and got an action for birthdays of people in Wikidata.
Improvement on the Birthday trigger so that it can use the lang trigger field to get the birthdays. This trigger field will be set on IFTTT recipe configuration by the user creating the trigger.
Fixed bug which prevented RSS content for the triggers to be translatable, now the RSS views for all the triggers are translatable in various languages provided the localization is available.
Modified the birthday trigger to get information about recent birthdays and ordering it in descending order of date. An example instance of the query can be found here: Recent birthdays for computer scientists and the query is below:

SELECT ?entity ?itemLabel ?propertyLabel ?date (year(?date) as ?year) 
WHERE 
{   
 	?entityS wdt:P569 ?date .   
  	?entityS wdt:P106 wd:Q82594 . 
    values ?property { wd:P106 } . 
    values ?item { wd:Q82594 } . 
              SERVICE wikibase:label {
              	bd:serviceParam wikibase:language "en" . 
                        	?entityS rdfs:label ?entity . 
                			?item rdfs:label ?itemLabel . 
                			?property rdfs:label ?propertyLabel
              } 
            FILTER (datatype(?date) = xsd:dateTime && month(?date) = month(now()) && day(?date) <= day(now()))
} ORDER BY DESC(day(?date)) DESC(?year)

Added more trigger ingredients in the birthday trigger. Ingredients are now 3 instead of 1 which are; <lang>, <property> and <item>. These trigger ingredients optimizes the query to produce letter results and also gives room for the IFTTT users to customize the query to suite their needs.
Added options for property and item ingredients on the IFTTT front-end for users to select various options in creating recipes for birthday triggers.
Cleaned up the RSS view for birthday trigger to accommodate the new results that are returned like; age of the person, date of birth, properties and items etc....

Week 12 (13th August - 19th August)

Created an account on the Medium Website in which I will use to post my GSoC blog as recommended by the mentors.
Pencils down, Code clean up and documentation of various section of my codes. The various commits for the documentation are here: Docstring Fixes in ifttt-tests.py, Code Cleanup and Documentation and Code Cleanup.
Filling up the various occupations that IFTTT users will select as option in the IFTTT front-end in the birthday trigger configuration. Property used is occupation, so I am finding the most interesting occupations on Wikidata that will return results from Wikidata to fill up the IFTTT trigger occupation options.
Merged pull request to Wikimedia IFTTT GSoC branch here: Pull Request 15.
Combination of all my work in GSoC from the first day can be seen here (a list of all the commits): All commits.
Migration of the IFTTT application on Wikimedia Tool Labs from Grid Engine to Kebernetes. This was the suggestion from Stephen (primary mentor) and it sounded good and more over why not use recent tool labs technology. So the application now runs on Kebernetes.