Let’s begin with the concept of basic and most common way to do analytics. The best way to predict something is to have a data rich environment. You get the data, split it up in two groups, the learning data and the test data (normally 80% / 20% when not an A/B test), after that you make your model with the best algorithm for the especific job and run the flow. The algorithm will learn from the data and make its assossiations. Then you use the 20% to test if the prediction match with the know data.
The algorithm does not have a mind of its own, it learns with the data given to them.
Let’s supose that you want to predict where to improve policing based on case statistics, it’s a good idea, but you have to take into account that the data is likely to be bias. It is common knowledge that the police are more active and less lenient in poorer neighborhoods, thus raising the statistics in these places. There is different treatment of the resident of one of these neighborhoods compared to the treatment of a resident of the suburb. So, with this data feeding the algorithm, it will reflect the same bias.
On other case, this one the AI, algorithm or what you want to call it, should decide your credit limit based in your data. The problem started when gender was one of the data considered. That way, cases like David Heinmeier Hansson that got 20x the limit provided for his wife when both share the same assets and his wife makes more money them him. Does the algorithm has anything against women? Of course not. But probably (there is no way of knowing for sure) the data provided has some bias in it self.
But can this be detected in the test phase? If the data is from the same source (split in 2 as explained before), then the data will continue to be biased and the prediction will check with the data.
This teaches us that the technology is not good or bad, the people behind it may be or just do some harm without even knowing or wanting.
We have a long way to mature as a data society. The people has to know the power that their own data have. The analysts and data scientists behind the tech have to look at the data they got and question is validity to the outcome proposed.
Be careful with the data and be responsible