On 6/1/19, scraper was interrupted while getting "offers by category":.Issue could be caused by change in how the URL needs to be structured to move through listing pages. Scraper still runs with no error, however no data is collected. 5/24/19 is the last date with data added to listings.csv.The pages under category headers may have inflated numbers due to counting accounts that are no longer active. After viewing while logged in, my assumption is that the directory page shows accurate numbers of currently active members. directory) do not match up to the numbers that appear if you click on one of the category headers. Noticed that numbers for categories on Timebankers' Talents pages (i.e.Planning to add artifacts here once complete. I have been working an a database design as part of my course at BU.Look at demographic info in relation to timebank locations.Should investigate if areas with more in closer proximity also have more activity. Deeper look at connection with language use, framing, and descriptiveness of mission statements and about pages.Use NLP to take a deeper dive into what requests/offers are exchanged, what goes unmet, and other patterns.Consider use of active / passive voice in mission statement / notes. Hourworld wikipedia series#Do time series analysis once more data is collected.Collect updated directory info regularly to gather new timebank info.Get count of modules displayed on timebank main page.Set up the daily scraper to run automatically via AWS.Add dummy to indicate if mission statement uses mutual aid language.Collect updated Facebook/Twitter data and create new features based on daily/new members or posts within given time period.Create line charts for easy visualization. Calculate change from prior day for number of exchanges, members, offers, and requests.Change how data for listings.csv is collected so that it only saves earliest and most recent observation of a listing.Use Twitter API to get updates on known Twitter accounts on a regular basis.The purpose of this log is to document the process as I go through this project, including challenges, questions, and ideas.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |