I wrote a suite of scripts to help you background check your doctor.
The Primary Care Provider Investigator imports a list of health care providers and then scrapes multiple review sites (e.g. Yelp, UCompareHealthCare) to aggregate their reviews: PCPInvestigator.
This project was born of frustration with my health insurer’s “search for a doctor” function, which listed the doctors my insurance covered but didn’t give me any sense of whether a doctor was any good.
Now I have a database of doctors and a snapshot of their reviews across multiple sites, plus a local webpage that displays the results and lets me sort by the categories I care about. (Click any image to view full-size.)
Y’see, doctor reviews became important to me when I visited Dr. Norma C. Salceda back in 2009 and had such a bad experience that I complained to my insurer (then Blue Shield) and was told just go find another doctor.
If I’d had the PCPInvestigator at the time, I would have been able to see this before I ever made an appointment:
That’s 38% positive reviews for Dr. Norma C. Salceda with over 150 reviews to examine. That’s bad.
In the summary report, I would have seen such quotables as “WORST experience that I’ve ever had with a doctor,” “worst medical experience I’ve ever had,” and “LADIES BEWARE SHE IS AN ANIMAL!!!”:
Sure, any doctor can have dissatisfied patients and a few complaints, but the intensity and the sheer number here is pretty special.
If you want to try the PCPInvestigator out yourself, it’s on github. You’ll have to be comfortable with the command line, and it’s useful to have some familiarity with python and mongoDB. You may also have to massage the doctors’ information input parsing to get it to match whatever list you can get from your health insurer. There’s more detail in the README. And you should know…
WARNINGS
- Web scraping violates Google ToS. Irresponsible use may result in your IP getting blocked.
- Pymongo cursor has been set to have no timeout (due to long intervals between scrapes.) Proceed w/ caution.
KNOWN ISSUES
- Health care review website formats change without notice.
- Searches by doctor name and so may suffer false positives (i.e. include reviews for other doctors of the same name) or false negatives (i.e. omit some available reviews)
- Disparities between your input file and the expected input format may require elbow grease to resolve.
- Script developed on OSX. Installation/execution not tested on any other system.
Even with these issues, if you want to find the best, closest primary care provider covered by your insurance, this is the only way I’ve found.
Hope this helps. Enjoy!
