This is great for freaking people out. It looks like a ghost is typing in your web browser. Web crawling using html parsers to grab links and navigate to new pages with the requests library is all very well, but when you want to physically submit search terms, or login details, or click buttons (etc.) …
WebCrawling: YouTube Pagination in Python
A while ago I wrote a blog post about how to scrape videos from YouTube. One question I've been asked since is how to navigate between different pages of search results. So here's how. YouTube The pre-amble looks exactly the same: Pagination Then we need to find the piece of html that corresponds to …
Continue reading "WebCrawling: YouTube Pagination in Python"
Moving Pictures
We recently did a survey for comments on a particular program. The survey was done through an anonymous Google form, which allowed us to download the list of responses as a CSV file. The question was: How do we display these responses to their intended audience? Sending out a CSV file, or in fact just …
Newtonian Funding
Research funding in the UK is very transparent. In fact you can access details of every RCUK (Research Councils UK) funded project through their online gateway. Like a lot of UK government data the content is available under the Open Government Licence v3.0, except where otherwise stated. The Newton Fund is part of the UK's …
Web Scraping YouTube Videos in Python
Web crawling and web scraping are two sides of the same coin. Web scraping is simply extracting information from the internet in an automated fashion. Web crawling is about indexing information on webpages and - normally - using it to access other webpages where the thing you actually want to scrape is located. YouTube is …
Spinning Stars II : Célérité
Gaussian Process Modelling is all very well, but when you've got lots of data points - and potentially lots of hyper-parameters - optimizing the values of those hyper-parameters can be very computationally expensive, i.e. slow... In my previous post I was using GPM to extract the rotation period of stars from Kepler data. For that …
Spinning Stars
Stars spin - and the rate at which they spin has a profound affect on their health. So measuring the rotation of different types of stars is an important pass-time for astronomers who like that kind of thing (i.e. stars). Fortunately stellar rotation is something that can even be measured directly from their time dependent …
Predicting the Future
One of the most well known examples of using Gaussian Process Modelling for forward prediction is the application described in Rasmussen & Williams, which shows the prediction for the future of atmospheric CO2 levels. When the book was written, the prediction showed the increase in CO2 concentration continuing at roughly the same rate. However, things …
Furrier Cats
Last week I gave an introductory lecture on imaging with radio interferometers. I started off with a Python demonstration of Fourier filtering using cat pictures, which seemed pretty popular and lead to one of my fellow lecturers coining the term "Furrier Cats" (this is genius and I wish I'd thought of it). Since it was …
Gaussian Process Modelling in Python
Non-linear regression is pretty central to a lot of machine learning applications. However, when you don't know enough/anything about the actual physical parametric dependencies of a function it can be a bit of a show-stopper. But... what if you could predict the value of a function at any point based only on its value at …
ALL YOUR BASE ARE BELONG TO US 