About the workbooks

For agencies that need a quick data-driven solution or a point of comparison, Excel workbooks from BGL contain dozens of targeted flagging rules selected for each state. States can test the rules by pasting internal data into the workbook, turn on and off rules for flagging new cases directly in the workbook, or export selected rules for use in another system. Using the state’s actual data, the workbooks will show what share of cases would be pulled for review under those rules (workload), and what share of cases pulled would have true errors (precision) as well as what share of error dollars in that sample would be caught (recall).

States can easily test out many combinations of rules and compare performance across them, allowing for rapid testing by program and data analysts without new software or integrations.

These workbooks can serve as a stopgap solution for states that do not have an internal data team doing PER modeling or for teams that lack the data access, tooling, or time to build an internal solution. They may also be useful for agencies interested in benchmarking their current operations, or evaluating a vendor’s proposed approach. 

How to use the workbooks

Each workbook has flagging rules that can be used immediately (within the workbook or re-implemented in queries) to prioritize cases for review. These flagging rules are combinations of case characteristics, such as household size, earned income amount, and the ratio of benefits of the max allotment. The rules are extremely targeted. Naively applying the rules at the top of their lists to the caseload, the median state would need to use 39 rules to flag 5% of its caseload. 

We strongly recommend that states not simply use the rules as-is. They ought to validate the rules by pasting in their internal case data, then selecting rules using data that the models we used have not seen. In addition, it is critical to select rules that reflect what each state’s secondary review process can detect and remedy. This requires bringing in expertise on those processes to review the rules characteristics and the cases that the rule flags (both easy to find in the Excel workbook). 

Each workbook begins with a “Start Here” tab, which contains instructions on how to use it. The tab also has overall performance numbers that reflect the much more in-depth QC process, as compared to how the rules will be used, which is for pre-auth or QA reviews. Yet, we have found that pre-auth reviews can find and address a wider range of errors that are flagged by these rules. If your state primarily relies on post-enrollment reviews, then the best starting point is to input your QA data, with original benefit amount and corrected benefit amount, into the spreadsheet to help drive rule selection. This will better reflect which rules translate to QA. 

How these tools were developed and validated

These rules come from national or state-specific predictive models and the rules that are included had the strongest results on a statistically conservative measure of precision (see documentation on our github for how these rules were created, filtered, and ranked).

We have validated with a few states that at least some of the rules perform in line with the numbers in the workbook when they test on internal future data (post FY24, which we do not have access to). You should not expect all the rules to perform well - that's why a testing and refining step is essential.

For more information or assistance

We can help you optimize the rules based on your internal, updated data and help you understand and use the tool. States that have worked with us have seen performance gains of 10-25%. Please reach out to us so that we can send you a custom workbook  and set up a time for a consultation.