Do you mind giving us brief on what kind of data you are collecting and highlight any interesting findings so far?

I do have a few questions too:

1. Will this scale? One problem with scrapers is that they break when people update their website. I'm imagining this problem multiplied by 18,000 and compounded by each scraper potentially being written by a different volunteer.

2. Where are the scrapers getting run?

3. How do the documents that the scrapers collect get transformed into usable data?

4. It seems to me like a scalable solution would be a standard to report data, a law to compel police departments to follow that standard, and then a system to collect that data and make it available. Do you work with police departments at all on data reporting?

TL:DR; If you want to write scrapers: go for it! Run your scraper, share the results in Discord and with your friends, and talk about the process. We'll be listening, and it will help us build tools to support this important work.

A few things to clarify:

a. The source of truth for "what are we doing right now" and "how can I contribute" is https://docs.pdap.io/.

b. Empowering people who write scrapers is a part of our broad mission of "police data accessibility", but we have some foundational work to do first! Right now our primary project is creating a database of police agencies and data sources. This will help people understand what kinds of data are available, at which agencies, with which steps to access it. It will also help us create archives of the primary sources, so that if they get taken offline we can still go back and scrape them.

c. What we have realized in the past few years: there are already a ton of people writing and using web scrapers for their day to day work. They are as decentralized as our police system. Our scrapers repo will reflect that. We shouldn't all rely on one library, or even one language. The people who need the data are most motivated to maintain scrapers, and we expect that maintenance will be ad-hoc and as-needed for the immediate future. In most cases, data already published on the internet is useful to local users as-is.

d. If you have a question you'd like to answer about the police, here's the investigation process:

1. Determine whether public data exists to answer your question. Use google to find the appropriate agency, and see what they're publishing. 2. Determine how it can be accessed; do you need to make a FOIA request? Is there a URL? 3. If there's a URL, determine whether you need to write a scraper to access the records. Often, the records can simply be downloaded. 4. Write and run a scraper, if you need one! 5. If there's not a URL, make a records request for the public information. This is a long and complicated process. 6. Share the data with your friends.

This means that scrapers are helpful and necessary some of the time; but not always, and not as the first step. We're trying to help with steps 1, 2, 3, 5, and 6. The theory is that writing scrapers is something people can easily slot in and help with; and that, depending on what question you're trying to answer, two scrapers for the same data source might look wildly different.

Scrapers are an important part of the ecosystem, but they're one piece of the puzzle.

You say in your FAQ "We aren't a watchdog—our activism is data collection and accessibility, not analysis or research."

Can you note any instances of other people using your data for analysis or research?

I'm not here defending the police, or denigrating the project, just playing devils advocate. What happens if the police just ignore you?

It was a lot of work to find data on policing nationwide, because the question really was "Is the sheriff doing a bad job, or do bad things happen sometimes?"

After some hard work trying to identify other cities with similar socioeconomic circumstances and populations, it became clear that our local sheriff was actually better than average, and that much of the outrage was fabricated.

That's also when I learned that many people don't want to listen to statistics unless they agree with their own preconceptions.

Maintaining scrapers for 18k county websites and PDs is no small task and looking through the docs for PDAP, it seems like this is still a very open question.

Here’s the pitfalls I see you falling into:

(1) seriously, what data are you collecting? “Everything” isn’t a great answer (who’s supposed to use ‘everything’, anyway? “Anyone”?). “Apples-to-apples police misconduct statistics” is a good one.

(2) it’s important to clarify 1 because you need to know who you’re serving, and why. Different activists need different data. “Have all data” sounds good until you need to decide how to allocate your resources.

(3) more deeply, data is the land of edge cases. Even just with police misconduct, you need to get DEEP to rigorously compare seemingly-simple stats like “# of unjustified police killings”. If you don’t start narrow, you’ll never show value. If you don’t show value, nobody will ever care you exist.

When I look at the data you’ve collected, it ranges from annual reports, to municipal contact info, to crime stats. What’s important to collect at scale? To whom? What do they need it for?

Again - great, ambitious idea! But $250k goes fast. Show value before it runs out!

Maintaining thousands of scrapers for different formats seems like a nightmare, and it won't take long for departments to learn they can slightly tweak the format of their reporting to cause extra work for you.

On the plus side, working with all this data probably makes you all very qualified to advise on developing standards.

There are other sources regarding Brady lists like https://giglio-bradylist.com/ and http://bradycops.org/, but they are obviously not 100% complete.

Departments that are trying to be transparent (or who just don't want to deal with figuring it all out from scratch) may be happy to adopt something considered a "standard" for tracking and reporting data. In some cases it means it is a checkbox they can check without having to deal with annoying people and their annoying questions... but that hardly matters so long as the data is made available. It would also give companies developing software for police departments a target to aim for.

2. You can run them locally. We're not running the scrapers anywhere, or storing extractions anywhere.

3. This is a big, big question. Right now, the answer is dependent on the use case. Rather than trying to make the world's biggest database, we're going to respond to community needs and build this kind of thing as it comes up.

4. https://measuresforjustice.org/ is doing something like this! We're interested in creating incentives for police departments to make their data more accessible and transparent.

"Making public information public" is a good tagline too.

Do you know what kinds of work people did with the data? It seems to me one of the best ways to address BART crime would be to support the impoverished and desperate people who don't have any recovery or mental health support, but that work is slow...

> What does [dead] mean?

> The post was killed by software, user flags, or moderators. Dead posts aren't displayed by default.

I suspect that it is a false positive. Maybe email the mods for help? hn@ycombinator.com

[0] https://news.ycombinator.com/newsfaq.html

We've come a long way since that post in terms of strategy and focus. Most of that time was spent with between 1 and 3 volunteers, working a couple hours a week.

Transparency is a good goal in itself, I think. People are already using this public data, we're just trying to make it more accessible.

"Policing the police" was the original phrase used on reddit, but if you look at our website (https://pdap.io), that's not a phrase we use.

This has been my experience with bodycam footage, I've found that there's been quite a few heavily protested police involved shootings that when looking over the footage and the facts of the situation, were by the book and completely justified, yet no matter how many times you say to someone "you do know there's footage of the entire event, uncut and unfiltered", it doesn't seem to matter.

EDIT: I just remembered what my throwaway username is.

Listening is not illegal, but recording or redistributing "is illegal", however, it's not clear whether it's actually illegal or it just depends how and when you use it and what you do with it. There was a kid here that brought it up with the police and was harassed over it, I believe he made a website to broadcast it over a web page and was given a stern telling off. Which tbh is fairly valid as it has horrific amounts of PII in it.

The 80000 Hours podcast has an interview with the (non-technical) creator of OWID. I seem to recall some interesting stories about them getting emailed PDFs with COVID data and such.

I had the same question as you, and I was hoping to find ideas in the comments. It seems like the kind of thing that's both inherently messy and scrappy yet if you don't get at least somewhat organized it can't scale.

Update: link to the podcast episode page with quotes, transcripts, etc. https://80000hours.org/podcast/episodes/max-roser-our-world-...

1. Agreed. Our strategy for this isn’t clear on the website, I guess, but we do have one. It’s to focus on depth in geographic areas. This is because context is critical, and because most of the users we talk to are operating locally with municipal or county level data. So it’s more important to have every data source we can possibly find relevant to Pittsburgh than it is to have every arrest record in every municipality. Or at least, it’s more immediately useful to people.

That said, most people seem to contribute data sources from where they live. I think little microcosms will spring up where people take stewardship of maintaining information about their chosen geo or subject areas. Not too far down the roadmap, Milestone 2 for the PDAP heads.

2. I will take it as a next step to make this strategy clear and say why. We want to basically allow the community to make its own to do list: what kind of question are you trying to answer? That creates a “bounty” for data which can be fulfilled by an altruistic volunteer, another member of your team, etc.

3. Yes. We’re not trying to do apples to apples comparisons of departments yet, partly because it’s so absurdly difficult and you don’t know where to start. Why would you undertake a 12 hour research project to compare St. Louis and Minneapolis incident reports if you don’t have a use case? Instead we’re focusing on what we DO know we need: complete local data, town by town / county by county.

The data we collected reflects the nature of our early experiments, which were scattered. This airtable prototype is maybe 2 weeks old, next up is helping people understand where to focus.

The idea for demonstrating value is also local. I’m working with groups in Pittsburgh (where we are based, and where our funding came from) to make ourselves indispensable to them. I’m hoping to turn the $250k into a handful of killer local case studies in this year, rather than marking 0.1% progress toward a national vision.

Thanks again for giving me the practice explaining this stuff. I hope I’m making any kind of sense, and of course happy to hear where I’m still wrong.

Measures for Justice is working on developing standards: https://measuresforjustice.org/

Glad I saw this.

I took a step back from the project for a few months, not having time to volunteer. My understanding is that the board was basically formed out of all the people remaining after some enthusiasm died down.

When I came back, the board had incorporated and applied for 501c3 status. There were four board members, and a few volunteers who mostly just helped talk through the massive problem and plan. Eventually Kristin (OP) stepped down from the board, but was still at some meetings. A rotating cast of 2-3 other people would be hanging around the meetings at any given time.

I became Director of Operations on a volunteer basis for a bit over a year. This mostly just means paying bills, knowing passwords, and updating the website.

We had weekly meetings, where we'd talk for a few minutes or hours about the project, our ideas, and what we could do to move things forward. [0]

We ran a data bounty during this time [1]. One volunteer, Eric, made a bunch of prototypes around metadata for data sources.

Then we got 501c3 status after waiting for almost a year. I quit my day job and started writing grants and set up online donations. I hired two contractors for a bit of grant writing help, but otherwise did not have "coworkers" or "co-volunteers".

We got the grant money [2] about 8 months later. I went looking for a full-time software engineer. I started getting a salary and working full-time on the project as Executive Director, doing all the non-technical design, planning, and product work.

Throughout, I spent a lot of time interviewing and doing design research: investigating the work being done journalists, transparency activists, and local data users in Pittsburgh (and elsewhere). I've also been collecting feedback and experience from everyone in the Discord. Most of our current ideas about what's important and where to start come from that work, and the recent addition of an engineer with excellent journalism and software experience (about 6 weeks ago) has allowed us to start prototyping and developing something together in earnest.

Now: We're excited about our strategies, and it's probably a little early for broad consumption. We didn't coordinate this post; everything you can see is a work in progress. There's lively discussion in Discord about our goals, and I've been typing for about 24 hours straight with a break to nap and a break to eat something.

[0]: https://docs.pdap.io/updates/working-sessions [1]: https://docs.pdap.io/updates/blog/7-14-21-bounty-retro [2]: https://docs.pdap.io/updates/blog/5-17-22-first-grant-awarde...