Our Work


Measure Thrice, Cut Once

By Teri Ross on September 21, 2015

Product Owner Teri Ross here. Development Director Samantha Kyrkostas suggested the title of this blog after I wrote it. I had to look it up (what did we do without the internet?). It is an old adage used in carpentry. Reduce waste by ensuring your measurement is right. As an organization with primarily anonymous website users, we have a measurement problem at Illinois Legal Aid Online. We can't adequately demonstrate our impact; we are a legal aid organization with unknown outcomes.

We can demonstrate outputs. (If you don't know the difference between outcomes and outputs, consult Rachel Perry.) We got outputs covered. On the quantitative side, we can give you real-time website pageviews, visitors, average session durations and a host of other metrics. We can tell you how many people started online triage and how many of those submitted online applications and, if they didn't, how they exited the system. With a 3-month lag, we can tell you how many automated document interviews were started versus documents assembled.

The problem with outputs is that sometimes they feel like marks without meaning. Or, as T.S. Eliot lamented, "I have measured out my life with coffee spoons." That is why quantity is only part of the equation. We must also look at quality. Or, as William Bruce Cameron wrote, "Not everything that can be counted counts, and not everything that counts can be counted."

On the qualitative side, we collect hoards of feedback from users - various website surveys, contact us/feedback forms, LiveHelp surveys, automated form feedback, social media, training evaluations, and on and on. When we can reach those folks who are generous enough to leave their contact information, we hear the impact we made on their lives, but this is one or two in a million. Literally. We cobble our website numbers and users' stories together to gain some insight, but it is not enough.

Lord Kelvin wrote, "If you cannot measure it, you cannot improve it."

This is not an unfamiliar problem in legal aid. Although Illinois Legal Aid Online is uniquely positioned as a stand-alone statewide website and technology center (versus one that is connected to a program with direct services), measuring the impact of legal information is not all that different from trying to figure out an outcome for legal advice. What happens to that help-desk client after you advise them on their rights and procedural next steps? Occasionally, they follow up to let you know. Sometimes, they resurface with another question or legal issue. Mostly, they disappear back into the crowd of people waiting for help at legal aid's narrow door.

What are we to do? We must invest in discovery. In law, 'discovery' has a particular meaning, often tainted with delay, waste, paper cuts, or even carpal tunnel. Discovery is expensive and time-consuming, but necessary. It is an investment. Arthur Nielsen, a market analyst, said, "The price of light is less than the cost of darkness." Led by our Board of Directors, investment in measurement is the approach we at Illinois Legal Aid Online are taking this year. Stay tuned -