I’ve been reflecting a lot recently on the time I’ve spent helping to run WhatDoTheyKnow, first as a volunteer and then as an employee, as is common during life transitions. To help me contextualis...

Disclosure logs My disclosure log scraping project is now well underway. Interestingly, the percentage of requests made to English local government via WhatDoTheyKnow is much higher than I’d previo...

Public authorities answer hundreds of thousands of FOI and EIR requests every year, but most of the information that is released is not searchable or available for wider use. Some authorities do pu...

It is perhaps a sign of the times we live in that many local councils now run commercial advertising on their websites. I couldn’t quite place why that bothered me until today. I was browsing Wals...

Small Things Claude I was testing different models for NER and needed to add two columns to a table and adjust my pipeline a bit. I decided to use Claude code in the CLI, and in an effort to be he...

I’ve been thinking about outliers a lot this week, after a hypothetical scenario came true. UK FOI law requires that you use an acceptable form of your real name when making a request - John Smith ...

The impact of AI on UK FOI requests For a while now, I’ve noticed a steady increase in FOI requests that appeared to me to have been written almost entirely by AI. This weekend, I thought I’d look...

When releasing information under the Freedom of Information Act, public authorities often attempt to redact documents by obscuring sensitive text using black boxes. When this is done incorrectly, t...

In my last post, I mentioned that I’d finally managed to crack how to extract meaningful data from ~3.2 million emails that I’d scraped from WhatDoTheyKnow. I thought it was worth jotting down. Ov...

FOI data After years of chipping away at it, I finally worked out how to extract the answers from FOI responses in a way that is not susceptible to hallucination, and that, as a welcome byproduct, ...