The dredging up of historical data was painful for the head accountant as they had to find out what was posted during the year to a certain account, and why. Sometimes, errors were spotted and reclasses had to be made. Nevertheless, our presence and incessant requests would put them in a somewhat grumpy mood.
When I first started working in financial reporting myself, I realised why the late nights happened. Every month, errors in the ledger would create a break with expectations or budget or forecasting. I did not like this as I did not enjoy going home at 9 or 10 pm a few days every month end, for something so unnecessary. So I started my own investigations in between the reporting days.
I started by teasing out the recurring issues that had to be fixed every single month. Over the next 8 months, I managed to find the roots of the problems and would speak to the owners to eradicate the errors. Eventually, I got the overtime reduced to 0, which resulted in the team getting disbanded. No good deeds go unpunished.
In a later part of my career, I applied the the same logical thinking to data testing. This resulted in seamless upgrades as I managed to catch the bugs during before implementation (which, incidentally, were missed by the vendor's other clients). Sadly, shortly after I left the team, I heard that the governance and accountability on data change management was abandoned, swiftly followed by a severe deterioration in the quality of data. That system fell into disrepute and was never the same again.
To me, data is like the raw materials or components of a supply chain. If any single piece is of poor quality, the final product will be defective. Since data is not life threatening, unlike say a car, the same level of Quality Control is not applied.
Before fretting over data, it is important to understand if the data has gone bad. Things to consider include:
- Manual handling/workarounds
Ideally, this should be close to the 6 sigma (no more than 3.4 defects per million opportunities). But let's be realistic - is 3 sigma even achievable? 67 defects per 1000 opportunities. If not, there is a serious data problem - either at data capture and/or handling - Confusion
Project/ change teams are constantly arguing over where to get good data OR
are finding it hard to get the data they need - Multiple spends
Requests for spend on data sourcing for various projects which are similar in nature - Not reusing data
Data owners consistently having requests for similar data feeds and/or fixes - Data user silos
This happens when Users do not share data and disparate reports are created. A whole new cottage industry is then set up to perform reconciliation activities. See #6 - Reconciliation cottage industries
When the number of operations people working on reconciliations is growing consistently or exponentially, year-on-year - Delivery does not match Data turnaround time
If the data is "available" daily, but direct reports take more than 2 days to create, then have a look to see if it is a data quality or process issues - Upset usersTeams who have to work overtime to produce reports tend to have lower job satisfaction, higher burnout rates and less energy to focus on what is most important: understanding the data
- Regulatory wrath
Poor data will always get discovered as the final products, reports, are unable to reconcile with each other.
I'll write another article on my thoughts on how bad data can be prevented, and on how bad data can be repaired.
